Over the weekend our grafana server was running low on swap, so all the high-swap processes were stopped and restarted. (fwiw We have done this several times in the past couple of years we have used grafana, w/out issue as we always use same two scripts to start and stop, which were adapted from grafana examples + docs)
For whatever reason over the weekend, the grafana server came back up but it’s no longer reading it’s ini paths, database, panels, datasource or even the org name.
It’s strange because there is no particular error message, we can look at the process and see it was passed in with --config to the ini file that is the correct path and which hasn’t changed in 6 months.
we tried to find a config file checker, but didn’t seem to find one. but we’ve run this ini for a long time w/no issues, and we see other forum messages indicating the config should error if it finds a problem parsing (which it doesn’t).
the server starts and works just fine, it just has no settings or data. it’s like a brand new install. when we look at the server->admin->settings section, it has none of the command-line options we can see it’s running with. we tried running command line options with – and - but no effect. (note that our command line script is taken directly from grafana install and has same vars and so forth, just updated to our ini path w/customizations).
Struggling a little bit how to troubleshoot this, since we have no errors and everything seems to be as it should be… it’s just ignoring the command line args that it used to read just fine.
When we look in the log it just loads the server as normal, don’t see errors.
Haven’t been able to find anyone with this issue, and it’s strange as we haven’t changed grafana. Any tips from someone who has seen this before… or any suggestions on how to collect more info for troubleshooting would be appreciated.
/usr/sbin/grafana-server --v
Version 5.0.4 (commit: 7dc36ae)