I’m using the singlestat panel in grafana to try to count the number of servers that have registered a failure event in the past week. I’m using the latest prometheus as the datasource and have tested against both
v4.6.2
v5.1.3
If I run a query like this:
sum(count({name=“job_fail”} > 0) by (servername))
to get the count of servers that saw a failed job in the last 7 days, the number is significantly lower than it should be. I have the step in the data source configured for 15s. The min step in my singlestat is also set for 15s. In 4.6.2, for the week it sets the step way up to 7200. I get that it’s a min step, but is there a way for me to get the accurate number? With a step so large as 7200, there are too many points that are missing.