Trying to wrap my head around how to use the new alerting. I’m specifically trying to use labels in the message or details of the alert. For this post, I’ll use prometheus up{} evaluation as an example.
Alert query is using two sections
Defined metric for up{env=“prod”}
Classic Condition expression looking for condition when last() of above metric is below 1
Alert functions, but can’t plug in any labels for Notication as documentation shows ie. {{ $labels.instance }}
I understand from a closed inquiry that this is not possible w/ the classic condition. The fix appears to be to use Math and Reduce expressions.
How to do use these to do the same evaluation as I did with the Classic Condition?
Why can’t the alerting and use of variables in Grafana be a lot more user friendly and/or intuitive to admins?
You can pass template variables into alerting messages, but the variables themselves need to be defined in the dashboard and then used in the query on which you are alerting.
@grimzly the classic condition doesn’t have labels available to it, which it sounds like you’ve noticed. The biggest reason for this is that the other functions can return N unique matches for the query, each with a slightly different label set. Because we can’t know which label set you would need to use for templating, we do not allow classic conditions to attempt to access the labels at all. Are you able to create a multi-dimensional alerting rule for your use case?
I think you @davidparrott get what I’m wanting, but no. I don’t know how to make the multi-dimensional work for what I want.
What I desire is a match on this condition “last metric read is less than 1” since up equals 1 and down equals 0. This would hold for other conditions as well with systems monitoring like disk space threshold >= 95%, for example.
A super simple condition I know, but I only see how to structure it in a classic condition. I don’t want the mean/sum/min/max/etc… I simply want to evaluate the most recent value read in.
I have the same challenge. I’ll play with it to see how it behaves. I think in this particular case I could reduce to min, strict mode. Since it can only be 1 or 0 (or no value), then I’ll get 1 if it was 1 for the time period; 0 if there was any 0; and NaN if the data are missing, in which case something went wrong and I also want to alert.
If I’m getting anywhere with that I may come back here and say how it went.
Since necro’ing this thread yesterday, I have made progress. I’ll describe an up{} but this same pattern will work for most anything, with adjustments.
Challenge:
Get labels into alerts
If there are multiple services, have an alert for each without manual duplication
Classic condition fails on both counts. It cannot handle labels, and it has a single alert for all time series created by the Query.
Multidimensional fixes it. Example for an up on a few instances.
Query: up{instance=~"name1.*|name2.*"}
Expression B: Reduce, Min, Input A, Replace Non-Numeric with 0
Expression C: Math, $B < 1
Alert condition: C
B reduces to 1 while things are up, and 0 when an instance is down because of Min
C will be 0 while an instance is up, and 1 when it is down. Labels can be used in the Description and Summary. This creates individual alerts for each instance that matches the Query, rather than having a single alert across all instances.
Alerting on a math expression uses false/true: 0 means don’t alert, 1 means alert. There might be a nuance here that not 0 means alert, haven’t tested it but believe that’s true.
This could be also done with a Reduce, Last instead of Reduce, Min; then again Math for C as described above. There’s some experimenting to be done.
My recommendation is to work with the tools Grafana gives you, which is using Multidimensional Alerting, vs. waiting for being able to do it with Classic Condition which may never come, or come so late you don’t care any more.
I’m new to Grafana but have been able to fairly easily get most other basic elements of the system up and running nicely (ie setting up grafana dashboards that incorporate data from prometheus and loki). However, setting up alerting (and especially not being able to easily use expected labels in templating alert messages) has been surprisingly difficult. I would love to see some documentation that shows specific examples of how people are using this feature in production. Does anyone know of such documentation, blog posts, tutorials, github gists, etc? Thanks in advance!
As a more specific example, here is an alert email that I received. Does anyone know how can I get the placeholders in the Summary section {{$value.labels.service}}@{{$value.labels.instance}}
to be populated with the actual values that I see in the Value section?
