Grafana stopped working after an update to 8.2.1

Grafana Installation
1

Hello,
I run Grafana 8.2.1, Apache 2.4.18 on a VPS with Ubuntu 16.04.7 LTS.
After Grafana was updated to 8.2.1 (I think because I ran “sudo apt-get upgrade” while installing something else), it stopped working, i.e. the site is down. I did about 2-3 hours of googling all the errors but didn’t really find something that helped - maybe you can help? I ran out of ideas.

When I run “service grafana-server status”, the following is returned:

● grafana-server.service - Grafana instance
   Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; vendor preset: enabled)
   Active: failed (Result: start-limit-hit) since Mon 2021-10-18 18:41:40 EDT; 1s ago
     Docs: http://docs.grafana.org
  Process: 954 ExecStart=/usr/sbin/grafana-server --config=${CONF_FILE} --pidfile=${PID_FILE_DIR}/grafana-server.pid --packaging=deb cfg:default.paths.logs=${LOG_DIR} cfg:d
 Main PID: 954 (code=exited, status=232/ADDRESS_FAMILIES)

Oct 18 18:41:39 my-server.com systemd[1]: grafana-server.service: Unit entered failed state.
Oct 18 18:41:39 my-server.com systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Oct 18 18:41:40 my-server.com systemd[1]: grafana-server.service: Service hold-off time over, scheduling restart.
Oct 18 18:41:40 my-server.com systemd[1]: Stopped Grafana instance.
Oct 18 18:41:40 my-server.com systemd[1]: grafana-server.service: Start request repeated too quickly.
Oct 18 18:41:40 my-server.com systemd[1]: Failed to start Grafana instance.
Oct 18 18:41:40 my-server.com systemd[1]: grafana-server.service: Unit entered failed state.
Oct 18 18:41:40 my-server.com systemd[1]: grafana-server.service: Failed with result 'start-limit-hit'.

In the syslog I can see:

Oct 18 18:41:39 101z systemd[1]: Started Grafana instance.
Oct 18 18:41:39 101z systemd[938]: grafana-server.service: Failed at step ADDRESS_FAMILIES spawning /usr/sbin/grafana-server: Operation canceled
Oct 18 18:41:39 101z systemd[1]: grafana-server.service: Main process exited, code=exited, status=232/ADDRESS_FAMILIES
Oct 18 18:41:39 101z systemd[1]: grafana-server.service: Unit entered failed state.
Oct 18 18:41:39 101z systemd[1]: grafana-server.service: Failed with result 'exit-code'.
Oct 18 18:41:39 101z systemd[1]: grafana-server.service: Service hold-off time over, scheduling restart.
Oct 18 18:41:39 101z systemd[1]: Stopped Grafana instance.

I was able to make Grafana running again temporarily when entering:

grafana-server --homepath='/usr/share/grafana' --config='/etc/grafana/grafana.ini'

However, it looks like this will generate a new grafana instance with the default admin password??

When I just run

grafana-server

I get the error message:

Grafana-server Init Failed: Could not find config defaults, make sure homepath command line parameter is set or working directory is homepath

How can I tell Grafana the config defaults, homepath, … so the “service grafana-server status” command works again?

My /etc/grafana/grafana.ini looks like this:

[paths]
[server]
protocol = http
http_port = 3000
domain=my-server.com
root_url = http://my-server.com
[database]
[datasources]
[remote_cache]
[dataproxy]
[analytics]
[security]
[snapshots]
[dashboards]
[users]
[auth]
[auth.anonymous]
[auth.github]
[auth.gitlab]
[auth.google]
[auth.grafana_com]
[auth.azuread]
[auth.okta]
[auth.generic_oauth]
[auth.basic]
[auth.proxy]
[auth.jwt]
[auth.ldap]
[aws]
[azure]
[smtp]
[emails]
[log]
[log.console]
[log.file]
[log.syslog]
[log.frontend]
[quota]
[alerting]
[annotations]
[annotations.dashboard]
[annotations.api]
[explore]
[metrics]
[metrics.environment_info]
[metrics.graphite]
[grafana_com]
[tracing.jaeger]
[external_image_storage]
[external_image_storage.s3]
[external_image_storage.webdav]
[external_image_storage.gcs]
[external_image_storage.azure_blob]
[external_image_storage.local]
[rendering]
[panels]
[plugins]
[live]
allowed_origins = *
[plugin.grafana-image-renderer]
rendering_ignore_https_errors = true
[enterprise]
[feature_toggles]
[date_formats]
[expressions]
[geomap]

And my Apache2 vhost configuration like this:

<VirtualHost *:80>
        ServerName my-server.com
        ServerAdmin webmaster@localhost
        ProxyPreserveHost On
        ProxyPass / http://127.0.0.1:3000/
        ProxyPassReverse / http://127.0.0.1:3000/
</VirtualHost>

<IfModule mod_ssl.c>
        <VirtualHost *:443>
                ServerName my-server.com
                ServerAdmin webmaster@localhost
                Header always set Referrer-Policy "no-referrer"                 
                ProxyPreserveHost On
                ProxyPass / http://127.0.0.1:80/
                ProxyPassReverse / http://127.0.0.1:80/
SSLCertificateFile /etc/letsencrypt/live/my-server.com-0001/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/my-server.com-0001/privkey.pem
        </VirtualHost>
</IfModule>

Any clues how I can make Grafana work again?

2

I finally found a solution!
As ssmoss suggested (Thanks!), try to replace the content of the 8.2.0 service (or in my case 8.2.1) with the 8.1.7 one. (I’d recommend to make a backup copy first.)

su
cd /usr/lib/systemd/system/
cp grafana-server.service grafana-server.service.bckp
vi grafana-server.service

Then, delete all lines (e.g. keep ‘d’ pressed to delete lines (=double ‘d’ input))
And insert new configuration: Copy from this URL, then in the vi editor, press ‘i’, then Shift + Insert.

See original post here:

Never change a running system. :see_no_evil:

P.S.: For some weird reason, I also had to change permissions on the grafana.ini since the syslog ("vi /var/log/syslog") showed:
lvl=crit msg="failed to parse \"/etc/grafana/grafana.ini\": open /etc/grafana/grafana.ini: permission denied"
So I ran:
chmod g+rw /etc/grafana/grafana.ini
Not sure if this was the reason for the error in the first place, but I don’t want to touch anything anymore.

3

Note: This is still an issue in Version 8.2.3 (commit: fb85ed6, branch: HEAD).
See my solution in:

1 Like