Ghost alerts from a destroyed organization/dashboard

Grafana Alerting
1

  • What Grafana version and what operating system are you using?
    Grafana v8.1.1 on Debian 10

  • What are you trying to achieve?
    I am trying to delete ghosts alerts that appears in the log with the message: <<lvl=eror msg=“Alert Rule Result Error” logger=alerting.evalContext ruleId=4 name=“Status 104 alert” error=“could not find datasource: data source not found” changing state to=keep_state>> but can’t be find with the API or on any dashboard. I think they was part of a dashboard that was in a deleted organization.
    Moreover migration to “ngalert” failed with the error: t=2021-08-19T14:54:07+0200 lvl=info msg=“Executing migration” logger=migrator id=“move dashboard alerts to unified alerting”
    t=2021-08-19T14:54:07+0200 lvl=eror msg=“Executing migration failed” logger=migrator id=“move dashboard alerts to unified alerting” error=“failed to migrate alert 2: dashboard with UID under organisation 1 not found: %!w()”

  • How are you trying to achieve it?
    I searching the alerts by API or via panel alert list etc… but can’t find them and thus can’t detroy them.

  • What happened?
    I can disable them with alerting parameters execute_alerts = false or disable engine alert. But i want to be able to do other alerts…

  • What did you expect to happen?
    I expect that alert disappear with the organization or to find a way to kill them after that. Or to be able to migrate to ngalert.

2

Hi @thomasbrom

To clarify, have you enabled the feature toggle in the Grafana configuration to enable ngalert? GF_FEATURE_TOGGLES_ENABLE=ngalert

3

Hi @mattabrams ,
I didn’t.
When I try to do it Grafana doesn’t start properly because of the inability to migrate to ngalert system my ghost alert (to summarize, i can find you the exact log if you want, it is about not finding the dashboard of the alert if i remember well).

4

yes I think that log line would be super helpful :+1:

5

With enable = ngalert, after sudo systemctl restart grafana-server.service.
Looking at /var/log/grafana/grafana.log I found that (showing you an entire “cycle” of log but the main point is “Executing migration failed” I guess):

t=2021-08-26T19:19:06+0200 lvl=info msg="Starting Grafana" logger=server version=8.1.1 commit=90c87a52f7 branch=HEAD compiled=2021-08-09T14:53:33+0200
t=2021-08-26T19:19:06+0200 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2021-08-26T19:19:06+0200 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
t=2021-08-26T19:19:06+0200 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"
t=2021-08-26T19:19:06+0200 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.logs=/var/log/grafana"
t=2021-08-26T19:19:06+0200 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.plugins=/var/lib/grafana/plugins"
t=2021-08-26T19:19:06+0200 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.provisioning=/etc/grafana/provisioning"
t=2021-08-26T19:19:06+0200 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana
t=2021-08-26T19:19:06+0200 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana
t=2021-08-26T19:19:06+0200 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana
t=2021-08-26T19:19:06+0200 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins
t=2021-08-26T19:19:06+0200 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning
t=2021-08-26T19:19:06+0200 lvl=info msg="App mode production" logger=settings
t=2021-08-26T19:19:06+0200 lvl=info msg="Writing PID file" logger=server path=/var/run/grafana/grafana-server.pid pid=2426
t=2021-08-26T19:19:06+0200 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
t=2021-08-26T19:19:06+0200 lvl=info msg="Starting DB migrations" logger=migrator
t=2021-08-26T19:19:06+0200 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting"
t=2021-08-26T19:19:06+0200 lvl=eror msg="Executing migration failed" logger=migrator id="move dashboard alerts to unified alerting" error="failed to migrate alert 2: dashboard with UID  under organisation 1 not found: %!w(<nil>)"
t=2021-08-26T19:19:06+0200 lvl=eror msg="Exec failed" logger=migrator error="failed to migrate alert 2: dashboard with UID  under organisation 1 not found: %!w(<nil>)" sql="code migration"
t=2021-08-26T19:19:07+0200 lvl=info msg="Starting Grafana" logger=server version=8.1.1 commit=90c87a52f7 branch=HEAD compiled=2021-08-09T14:53:33+0200

And Grafana don’t start/work…

Just after to get back my working grafana i put “;” befoore enable=ngalert and get (among all the fine log):

t=2021-08-26T19:22:10+0200 lvl=info msg="Starting DB migrations" logger=migrator
t=2021-08-26T19:22:10+0200 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=330 duration=602.226µs

Grafana starting and everything is fine.

6

This topic was automatically closed after 365 days. New replies are no longer allowed.