I have added an alert to my grafana dashboard. I see 2 different issues:
-
The Alert fires even though the conditions are not met. Notice that my metric is consistently above 0 but the alert still triggered.
-
The next alert did not run for 56 minutes. Notice the time difference between the firing alert(noon) and the OK alert(that ran at 12:56 pm). I would have expected the OK state to be evaluated at either at 12:30 pm or 1:00 pm. Grafana probably ran the alert in between and the state was not OK and that is why it didn’t adjust the state. However, my confusion is why is it not running this in increments of 30 minutes.
My DataSource is WhisperDB.
Config
Alerts: