Working configuration example for alerts templating - Telegram and Slack

Grafana Alerting
, , ,
1

Hi Grafana community!

I have spent a lot of time to configure templates of my alerts, a lot of time lost with all recent updates since Grafana 8 so I wanted to share here my templates, hoping it can help members.

Important: this configuration is running on Grafana v9.3.2 and may not work with an earlier version.

Alert rules

The following alert rules apply to a Prometheus datasource where I have added a custom label “node” to each target.
If you are using a different datasource, you have to adapt these.

Alerts are set with latest Grafana defaults:

  • A = query
  • B = Reduce last of A
  • C = Threshold of B - set as Alert condition

Summary and annotations

  • Summary: summary of the alert
    example: CPU consumption is high
  • AlertValues: scraping values from query B to show value in alert
    example: CPU usage: {{ $values.B }} %
  • Node: value coming from a custom label to get the instance concerned
    example from my P: {{ $labels.node }}

Templates

In the following templates, you have to replace fields {grafana_URL} and {dashboard_id/name}.

Slack message templates

slack_title

{{ define "slack_title" }}
  {{ if gt (len .Alerts.Firing) 0 }}
  🔥 {{ len .Alerts.Firing }} alert(s) firing
  {{ end }}
  {{ if gt (len .Alerts.Resolved) 0 }}
  ✅ {{ len .Alerts.Resolved }} alert(s) resolved
  {{ end }}
{{ end }}

slack_message

{{ define "slack_message" }}
  {{ if gt (len .Alerts.Firing) 0 }}
    {{ range .Alerts.Firing }} {{ template "slack_alert_firing" .}} {{ end }} {{ end }}
  {{ if gt (len .Alerts.Resolved) 0 }}
    {{ range .Alerts.Resolved }} {{ template "slack_alert_resolved" .}} {{ end }} {{ end }}
{{ end }}

slack_alert_firing

{{ define "slack_alert_firing" }}
  Severity: *{{ .Labels.severity }}*
  *{{ .Labels.alertname }}*
  {{ .Annotations.summary }}

  Node: *{{ .Annotations.Node }}*
  {{ .Annotations.AlertValues }}
  <https://{grafana_URL}/d/{dashboard_id/name}|Access dashboard> - <https://{grafana_URL}/alerting/silence/new?matcher=node%3D{{ .Annotations.Node }}|Silence all alerts for this node> - <https://{grafana_URL}/alerting/silence/new?matcher=alertname%3D{{ .Labels.alertname }}|Silence this alert for all nodes>{{ end }}

slack_alert_resolved

{{ define "slack_alert_resolved" }}
  *{{ .Labels.alertname }}*
  
  Node: *{{ .Annotations.Node }}*
  {{ .Annotations.AlertValues }}
  <https://{grafana_URL}/d/{dashboard_id/name}|Access dashboard>
{{ end }}

Telegram message templates

telegram_message

{{ define "telegram_message" }}
  {{ if gt (len .Alerts.Firing) 0 }}
  <b>🔥 {{ len .Alerts.Firing }} alert(s) firing:</b>
    {{ range .Alerts.Firing }} {{ template "telegram_alert_firing" .}} {{ end }} {{ end }}
  {{ if gt (len .Alerts.Resolved) 0 }}
  <b>✅ {{ len .Alerts.Resolved }} alert(s) resolved:</b>
    {{ range .Alerts.Resolved }} {{ template "telegram_alert_resolved" .}} {{ end }} {{ end }}
{{ end }}

telegram_alert_firing

{{ define "telegram_alert_firing" }}
  Severity: <b>{{ .Labels.severity }}</b>
  <b>{{ .Labels.alertname }}</b>
  {{ .Annotations.summary }}

  Node: <b>{{ .Annotations.Node }}</b>
  {{ .Annotations.AlertValues }}
  <a href="https://{grafana_URL}/alerting/silence/new?matcher=node%3D{{ .Annotations.Node }}">Silence all alerts for this node</a> - <a href="https://status.astar.network/alerting/silence/new?matcher=alertname%3D{{ .Labels.alertname }}">Silence this alert for all nodes</a>
  <a href="https://{grafana_URL}/d/{dashboard_id/name}">Access dashboard</a>{{ end }}

telegram_alert_resolved

{{ define "telegram_alert_resolved" }}
  <b>{{ .Labels.alertname }}</b>
  
  Node: <b>{{ .Annotations.Node }}</b>
  {{ .Annotations.AlertValues }}
  <a href="https://{grafana_URL}/d/{dashboard_id/name}">Access dashboard</a>{{ end }}

Contact points

Slack
Title = {{ template "slack_title" . }}
Text Body = {{ template "slack_message" . }}

Telegram
Message = {{ template "telegram_message" . }}

Notification policies

Set contact points with criteria you need, tick “Continue matching subsequent sibling nodes”.

I hope these example can help builders struggling to setup alerts templating, please let me know in answers.

7 Likes
2

Great job @bLd759 ! These templates are great and should be very useful to others.

3 Likes
3

This is awesome!! My idea is this should be part of grafana as stepper wizard that fill in this code for you based on parameters you select such message host type: team, slack etc

3 Likes
4

Hey @bLd759,

This is really great :sunglasses: and many community users keep asking for this. I will add more tags so that it gets more visibility while searching.

Thanks and if you have more examples then keep sharing them !! :pray:

1 Like
5

See the initial message posted by @bLd759 which contains the actual and complete solution.