State change to no_data

Grafana Alerting
1

Hi,
I am using Grafana version 8.3.6 and using timestream as data source. I am receiving no data alerts randomly and do not see a gap in the data at the time of the alert. I have tried changing evaluation interval, ‘for’ interval and alert query to give enough time for no_data to not get triggered.

Example alert query: WHEN avg(A) OF query(A,15m, now) > 75

Is there a different time interval that control no_data alerts? Would appreciate any help in understanding no_data triggers.

Many thanks!

2

@sparkhi Welcome to the forum. If I had to speculate, I would say you are getting the No Data alerts because you have null values in your data, and occasionally this fouls up your query (average of NULL + 7 = ?).

Two suggestions:

  1. Can you change your query to use last() instead of avg() ?
  2. Can you upgrade to Grafana Unified Alerting (you will have to eventually, so might as well do it now) and then follow the various online guides to constructing your alert?

Good luck!

3

Thanks @grant2. Yes, I do have null values in the data sometimes but usually it doesnt cross the threshold for an alert. Unfortunately I may not be able change my query to last in all cases as I am really looking for avg. I tried unified alerting but since I could not find an easy way to generate alert yamls and provision alerts via files, I had to postpone that change. Feel free to correct me if there is an easy way to port alerts in bulk into unified alerting.