Hello,
I’m using Grafana v9.1.3 and alerting with a PostgreSQL data source. If results are available query will returns for each result the value 1 with two labels. Time range for the query is now-55 minutes to now and rule is evaluated every hour, for 0s. This way I get an alert fired each hour if the query have at least one record within the last 55 minutes.
Issue 1, duplicated records.
Into my DB a new record is created at 08:34 am, rule is evaluated at 09:00 am, fires, I am notified. New record is created at 09:12 am, rule is again evaluated at 10:00 am, fires, receive notification with new record from 09:12 am AND from 08:34 am. This should not be possible as I my time filtered query is based on the last 55 minutes.
Issue 2, notifications skipped.
With the same example as above, I can receive an alert at 9:00 am, then nothing at 10:00 am, then at 11:00 am the record from 09:12 am that should have been in the 10:00 am alert, and a new record from between 10:00 am and 11:00 am.
My notification policy is configured with a parent one (based on a single label) with “Continue matching subsequent sibling nodes” activated, and 2 nested, with a single label for each. For each nested policy :
Group interval: 1h
Repeat interval:1h
I have no mute timings configured.
Question 1: Why is Grafana alerting with events outside query time window ?
Question 2: Why is Grafana sometimes missing alert firing ?
Thank you very much for helping me resolve this.