How To Use Alert Message Templates in Grafana

I am attempting to create a message template for my alerts. My goal is a simplified message using labels. I will paste my template below and the message I receive in slack. The ‘variable’, ‘location’, etc are names of labels.

{{ define "Alerting Template" }}
Alert: {{ .Labels.variable }} on {{ .Labels.location  }} is above threshold of {{ .Labels.threshold }}. For more detail, {{ .PanelURL }}.
{{ end }}

This is the output:

Alert: <no value> on <no value> is above threshold of <no value>. For more detail, <no value>.

I’ve tried to follow the default template and the documentation here. Is there something I am missing?

I am currently using OS 8.5 (and will soon be upgrading to 9).

3 Likes

Edited this solution to add the most common answers in this thread in one place, including template functions and labels with dots in the name - read through to the end for those

welcome to the :grafana: community @baileyheit1!

This is a really common question and I’m still learning how to access the values out of the alert query too. I hope this is a helpful example - I tried to make it very detailed for the benefit of other community users, even though I can tell you already know how to do some of these steps :wink:

With an alert query that looks like this - I created this with our sample TestDataDB datasource and manually added some labels that are similar to the ones you mentioned.

Click on the preview alert button to see the labels and values in the map:

To refer to labels and values from the query add a new annotation to the alert like this. Name the annotation AlertValues

{{ with $values }}
{{ range $k, $v := . }}
   Location: {{$v.Labels.location}}
   Variable: {{$v.Labels.variable}}
   Alerting value: {{ $v }}
{{ end }}
{{ end }}

Here it is in the context of the alert definition:

Then you can add a message template called myalert to create a single alert message and print the annotation. Notice that you will reference the annotation by name (.Annotations.AlertValues) in the sample below:

{{ define "myalert" }}
  [{{.Status}}] {{ .Labels.alertname }} 
  {{ .Annotations.AlertValues }}
{{ end }}

Here it is in context of the UI:

Next, you can add a message template for the full message, I called this one mymessage. Notice how the mymessage template references the myalert template that I created in the last step:


{{ define "mymessage" }}
  {{ if gt (len .Alerts.Firing) 0 }}
    {{ len .Alerts.Firing }} firing:
    {{ range .Alerts.Firing }} {{ template "myalert" .}} {{ end }}
  {{ end }}
  {{ if gt (len .Alerts.Resolved) 0 }}
    {{ len .Alerts.Resolved }} resolved:
    {{ range .Alerts.Resolved }} {{ template "myalert" .}} {{ end }}
  {{ end }}
{{ end }}

Lastly, you can add the mymessage template to the body of your slack contact point under Optional Slack Settings:

Alert summary:
{{ template "mymessage" . }}

In the UI, it looks like this:
Screen Shot 2022-07-20 at 5.22.40 PM

Now you should have an alert to your slack contact point that looks something like this:

A side note related to using Alert.Labels . . .

Something that can get confusing is that .Labels mentioned in the Template Data docs refer to the labels attached to the alert object NOT the query response labels. For those, we need to get into the values map in the exercise above:

Extra helpful reference table from the alerting docs

Examples for specific situations like . . .

When a label has dots in the name

If you have a label like netflow.ipv4_dst_addr with a value of 192.168.1.1, you might find that your alert template just shows the raw template (which is what happens if there’s an error in the evaluation).

If you’re using a classic condition, here’s an example of using the go index function:

{{ index $values.B0.Labels “netflow.ipv4_dst_addr” }}

Screen Shot 2022-08-15 at 6 15 21 PM

if your alert has two or more conditions or an expression, then you can use the $labels variable:
{{ index $labels “netflow.ipv4_dst_addr” }}

Screen Shot 2022-08-15 at 6 11 25 PM

How to use template functions like humanizePercentage

To reference the value in $values.RefIdX.Value (ex: $values.B0.Value) with template functions use .Value:

{{ with $values }}
{{ range $k, $v := . }}
   Location: 
{{ $v.Labels.location }}
   Alerting value: 
{{ humanizePercentage .Value}}
{{ end }}
{{ end }}
15 Likes

Thank you very much! It works like expected but it’s pitty that the documentation is not very clear :confused:

3 Likes

this is awesome! isn’t it odd that templates are under contact points?

4 Likes

Hi, thanks for your detail instructions. those helps me a lot.
However, I would like to send the alert to webhook which did not have any option to pairing with custom message/template that we already created. I have seeking many threads or in config file but still did not get any idea to use custom template into webhook alert (not like optional slack settings).
Have you try with webhook? thanks in advance

3 Likes

@melori.arellano do you know what tweaks are needed when not using a classic query as the current example returns two results with a reduce and use C for results alert:

I have tried $values.C in the alert in place of $values but the results in the email just ignoring the options:

Thanks

@ardianre these instructions are meant to be for slack. The webhook has a different format the json body sent uses the valueString property for the metric values returned by the alert: Webhook notifier | Grafana documentation

@mrdibbley Can you share an example of the reduce condition? I tried an example that uses input C as the alert condition and it still worked for me:

When I preview the alert, the values are in var=B even though it references the query result C:

My alert returns only the results of expression B (based on query C):

Screen Shot 2022-08-01 at 7.20.55 PM

@melori.arellano sure I’ve got it working with the following:



Thanks

@mrdibbley thank you for sharing this! Can you also show me what shows in the box when you click Preview Alerts button? That will show me what the values look like.

@melori.arellano sure here you go:

Thanks

1 Like

@mrdibbley I’m just getting back from some time away. Just to confirm - is this working for you now using $values.B?

Another interesting use case I want to capture in this thread. What to do about metric names that have a dot in the label name!

The template variables for values are now described here: How to template annotations and labels | Grafana documentation

If you have a label like netflow.ipv4_dst_addr with a value of 192.168.1.1, you might find that your alert template just shows the raw template (which is what happens if there’s an error in the evaluation).

If you’re using a classic condition, here’s an example of using the go index function:

{{ index $values.B0.Labels “netflow.ipv4_dst_addr” }}

Screen Shot 2022-08-15 at 6 15 21 PM

if your alert has two or more conditions or an expression, then you can use the $labels variable:
{{ index $labels “netflow.ipv4_dst_addr” }}

Screen Shot 2022-08-15 at 6 11 25 PM

3 Likes

Hi melori,

I have this exact issue and i have been stranded for days now.

This is my alert preview

Can you please assist with how the template will be?

@sijuade644 What values are you trying to display in your alert?

You should be able to do something like:

{{ index $values.B0.Labels “host.hostname” }}

Thanks for this post.
It was very useful for me to recover the data.
On the other hand, I can’t remove the Value part.
do you have to create your own message template?

image

@sch1 yes, you’ll want to use your own message template to customize the message

1 Like

Hi @melori.arellano, I am trying to create a similar template as @sijuade644 but I’m not receiving any data on my Slack channel. Could you please help me? This is my basic configuration to try to make it work, there must be something wrong in one of the steps:

This is my current alert and the annotation I’m using

The basic template

And the basic text body:

{{ template “myalert” .}}

I’m using a classic condition and it’s firing, but my result shows nothing in Slack.

I appreciate the support. Best regards.

I deleted the last dot in the text body and no I’m receiving the message

< no value >

Hey @melori.arellano,
Is there any way to add variables in custom Labels for Azure data source in Alerting.
I am trying to add resourcename as a variable. Could you please help me?