I had grafana working just fine. mabe after an update it has stopped working
Oct 10 15:18:51 rockpi systemd[1]: Started Grafana instance.
Oct 10 15:18:51 rockpi kernel: [16047.250717] audit: type=1326 audit(1633875531.393:16): auid=4294967295 uid=110 gid=115 ses=4294967295 pid=22881 comm="grafana-server" exe="/usr/sbin/grafana-server" sig=31 arch=c00000b7 syscall=214 compat=0 ip=0x7fabb45cf4 code=0x0
Oct 10 15:18:51 rockpi systemd[1]: grafana-server.service: Main process exited, code=killed, status=31/SYS
Oct 10 15:18:51 rockpi systemd[1]: grafana-server.service: Failed with result 'signal'.
Oct 10 15:18:51 rockpi systemd[1]: grafana-server.service: Consumed 94ms CPU time
Oct 10 15:18:51 rockpi systemd[1]: grafana-server.service: Service hold-off time over, scheduling restart.
Oct 10 15:18:51 rockpi systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 5.
Oct 10 15:18:51 rockpi systemd[1]: Stopped Grafana instance.
Oct 10 15:18:51 rockpi systemd[1]: grafana-server.service: Consumed 94ms CPU time
Oct 10 15:18:51 rockpi systemd[1]: grafana-server.service: Start request repeated too quickly.
Oct 10 15:18:51 rockpi systemd[1]: grafana-server.service: Failed with result 'signal'.
Oct 10 15:18:51 rockpi systemd[1]: Failed to start Grafana instance.
I am not getting any clue what might have gone wrong,
1 Like
wnasich
October 10, 2021, 11:57pm
2
Hi @niceguysan ,
This comment on github helped me to solve this issue:
opened 07:57AM - 08 Oct 21 UTC
closed 10:37AM - 08 Oct 21 UTC
After upgrade to version 8.2.0 can't start grafana service. Installation via ap… t-get update was fine and at the end, service was restarted but never go up again
I found that service file /usr/lib/systemd/system/grafana-server.service was changed and whole bunch of protection settings was added.
Some of this new settings instruct kernel to kill grafana-server proccess.
When I use old systemd service file, grafana run just fine.
Below are informations about my system, log and "bad" & "good" systemd service file.
$ uname -a
Linux zabbix 5.10.71-sunxi64 #21.11.3 SMP Thu Oct 7 23:52:46 CEST 2021 aarch64 aarch64 aarch64 GNU/Linux
$ cat /etc/issue
Ubuntu 18.04.6 LTS \n \l
Welcome to Ubuntu Bionic with Armbian Linux 5.10.71-sunxi64
$ dpkg -l | grep grafana
ii grafana 8.2.0 arm64 Grafana
syslog:
Oct 8 08:20:33 localhost systemd[1]: Started Grafana instance.
Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Main process exited, code=killed, status=31/SYS
Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Failed with result 'signal'.
Oct 8 08:20:33 localhost kernel: [ 7618.317684] audit: type=1326 audit(1633674033.139:18): auid=4294967295 uid=116 gid=124 ses=4294967295 subj==unconfined pid=21346 comm="grafana-server" exe="/usr/sbin/grafana-server" sig=31 arch=c00000b7 syscall=214 compat=0 ip=0xffffb465ecf4 code=0x0
Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Service hold-off time over, scheduling restart.
Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 1.
Oct 8 08:20:33 localhost systemd[1]: Stopped Grafana instance.
Oct 8 08:20:33 localhost systemd[1]: Started Grafana instance.
Oct 8 08:20:33 localhost kernel: [ 7618.587286] audit: type=1326 audit(1633674033.407:19): auid=4294967295 uid=116 gid=124 ses=4294967295 subj==unconfined pid=21351 comm="grafana-server" exe="/usr/sbin/grafana-server" sig=31 arch=c00000b7 syscall=214 compat=0 ip=0xffff80f02cf4 code=0x0
Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Main process exited, code=killed, status=31/SYS
Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Failed with result 'signal'.
Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Service hold-off time over, scheduling restart.
Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 2.
Oct 8 08:20:33 localhost systemd[1]: Stopped Grafana instance.
Oct 8 08:20:33 localhost systemd[1]: Started Grafana instance.
Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Main process exited, code=killed, status=31/SYS
Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Failed with result 'signal'.
Oct 8 08:20:33 localhost kernel: [ 7618.836798] audit: type=1326 audit(1633674033.659:20): auid=4294967295 uid=116 gid=124 ses=4294967295 subj==unconfined pid=21356 comm="grafana-server" exe="/usr/sbin/grafana-server" sig=31 arch=c00000b7 syscall=214 compat=0 ip=0xffff95418cf4 code=0x0
Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Service hold-off time over, scheduling restart.
Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 3.
Oct 8 08:20:33 localhost systemd[1]: Stopped Grafana instance.
Oct 8 08:20:33 localhost systemd[1]: Started Grafana instance.
Oct 8 08:20:33 localhost kernel: [ 7619.087799] audit: type=1326 audit(1633674033.907:21): auid=4294967295 uid=116 gid=124 ses=4294967295 subj==unconfined pid=21360 comm="grafana-server" exe="/usr/sbin/grafana-server" sig=31 arch=c00000b7 syscall=214 compat=0 ip=0xffffbcb75cf4 code=0x0
Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Main process exited, code=killed, status=31/SYS
Oct 8 08:20:33 localhost systemd[1]: grafana-server.service: Failed with result 'signal'.
Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Service hold-off time over, scheduling restart.
Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 4.
Oct 8 08:20:34 localhost systemd[1]: Stopped Grafana instance.
Oct 8 08:20:34 localhost systemd[1]: Started Grafana instance.
Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Main process exited, code=killed, status=31/SYS
Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Failed with result 'signal'.
Oct 8 08:20:34 localhost kernel: [ 7619.341092] audit: type=1326 audit(1633674034.163:22): auid=4294967295 uid=116 gid=124 ses=4294967295 subj==unconfined pid=21364 comm="grafana-server" exe="/usr/sbin/grafana-server" sig=31 arch=c00000b7 syscall=214 compat=0 ip=0xffff86bbccf4 code=0x0
Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Service hold-off time over, scheduling restart.
Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Scheduled restart job, restart counter is at 5.
Oct 8 08:20:34 localhost systemd[1]: Stopped Grafana instance.
Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Start request repeated too quickly.
Oct 8 08:20:34 localhost systemd[1]: grafana-server.service: Failed with result 'signal'.
Oct 8 08:20:34 localhost systemd[1]: Failed to start Grafana instance.
Service file /usr/lib/systemd/system/grafana-server.service
"Bad systemd service file":
[Unit]
Description=Grafana instance
Documentation=http://docs.grafana.org
Wants=network-online.target
After=network-online.target
After=postgresql.service mariadb.service mysql.service
[Service]
EnvironmentFile=/etc/default/grafana-server
User=grafana
Group=grafana
Type=simple
Restart=on-failure
WorkingDirectory=/usr/share/grafana
RuntimeDirectory=grafana
RuntimeDirectoryMode=0750
ExecStart=/usr/sbin/grafana-server \
--config=${CONF_FILE} \
--pidfile=${PID_FILE_DIR}/grafana-server.pid \
--packaging=deb \
cfg:default.paths.logs=${LOG_DIR} \
cfg:default.paths.data=${DATA_DIR} \
cfg:default.paths.plugins=${PLUGINS_DIR} \
cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}
LimitNOFILE=10000
TimeoutStopSec=20
CapabilityBoundingSet=
DeviceAllow=
LockPersonality=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
PrivateUsers=true
ProcSubset=pid
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectProc=invisible
ProtectSystem=full
RemoveIPC=true
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@privileged
SystemCallFilter=~@resources
UMask=0027
[Install]
WantedBy=multi-user.target
"Good", old systemd service file":
[Unit]
Description=Grafana instance
Documentation=http://docs.grafana.org
Wants=network-online.target
After=network-online.target
After=postgresql.service mariadb.service mysql.service
[Service]
EnvironmentFile=/etc/default/grafana-server
User=grafana
Group=grafana
Type=simple
Restart=on-failure
WorkingDirectory=/usr/share/grafana
RuntimeDirectory=grafana
RuntimeDirectoryMode=0750
ExecStart=/usr/sbin/grafana-server \
--config=${CONF_FILE} \
--pidfile=${PID_FILE_DIR}/grafana-server.pid \
--packaging=deb \
cfg:default.paths.logs=${LOG_DIR} \
cfg:default.paths.data=${DATA_DIR} \
cfg:default.paths.plugins=${PLUGINS_DIR} \
cfg:default.paths.provisioning=${PROVISIONING_CFG_DIR}
LimitNOFILE=10000
TimeoutStopSec=20
UMask=0027
[Install]
WantedBy=multi-user.target
I am running Grafana on Ubuntu 16.04.7 LTS , it seems related to the use of a systemd function not available in old versions
1 Like
