I’m trying get a notification whenever I have an error in my logs (from Loki). I’ve set up a rule that seems to be doing exactly what I want, and when I hit the Preview button, it shows a red Firing label (first picture). Promising!
However, when I look at it in the Alert rules view, it’s always at Normal, and I never get notified. (Second picture) I’ve been stumped for a while; what am I doing wrong?
welcome @pozwhwitmegqhjesqslg
in the rule details how do the every and for part look like?
also, could you please increase the the Grafana logs to debug and note any errors ?
Thanks for you reply, @antonio.
The every and for part are 1m each if I understand it correctly, see below. The grafana instance is managed, so I don’t have access to that at the moment, but I’ll see what I can do.
what happens if you change the for to 0s ? Afterwards, check the state history (in Alert list > select your rule > click View rule icon)
It still doesn’t fire and I don’t see anything in the alert history (after having 0s for some time). However, interestingly, I see it shows as “Firing” at the bottom of the view, but “Normal” at the top. Is there some kind of hysteresis that prevents it from triggering? That is, does it need to go to normal before it can trigger? One thing that speaks against that theory is that I do have another alert that goes back and forth between triggers, but still doesn’t show anything.
Hey, sorry I am late to the party. Just wondering if a solution was found to this problem? I am seeing similar behaviour in v10.2
A similarity between my situation and yours is that there are no “matching instances” in the alert - maybe this is something I do not understand yet and will need to read more into
Same Problem I am also facing…can you provide any solution?
Hi Antonio, I have a very similar issue. Was this problem ever solved?
@bobuser @mamonchw you comment under almost 2 years old topic. A lot has changed since then. So, if you want to get some meaningful help, then please provide more information about your problem: Grafana version, What you see in preview and in rule state etc.
In general, this problem could happen because preview and actual evaluation are slightly different. 1. When you click preview, the rule is evaluated at exactly the time you pressed the button. Once it is saved, it can be evaluated at different time interval, which might happen when the last point of the metric is below threshold.
2. Loki and Mimir\Prometheus queries may be “optimized”: in preview you see timeseries but in reality, that query could be evaluated as “instant”. Loki and Mimir have default lookback of 5 minutes for instant queries, so if there is no point within last 5 minutes, it will return NoData. Otherwise, it will take the most recent data point.
