Obtain the metric measurement value for alert notification

Grafana Alerting
,
1

  • What Grafana version and what operating system are you using?
    v8.2.2 (6232fe07c0), Ubuntu Linux

  • What are you trying to achieve?
    I am trying to display in a notification the metric measurement that was sent to the alert.

  • How are you trying to achieve it?
    I am using a template. The contents of ValueString usually look something like this:

[
	metric='up{instance="prometheus.local:8080", job="test"}'
	labels={__name__=up, instance=prometheus.local:8080, job=test}
	value=0.9501661129568106
]

If the ValueString has contents, I am trying to print just the value of “value”, which is the metric measurement.

Since ValueString is of type String, I cannot use any of the KeyValue methods on it.

Using match only provides a boolean result.

{{if gt (len .ValueString) 0 }}
Metric Measurement: {{match "value=([0-9.-]+)" .ValueString}}
{{end}}
-------------
Metric Measurement: true

I tried the stringSlice method like this:

{{ .ValueString.stringSlice }}

… it did not display anything and everything in the template after that point failed to display, which I assume means there was an error (not sure where that gets logged, if it does).

I am still playing with several other methods and have yet to be able to figure out the best way to just grab that value. I probably need to brush up on my golang templating skills. In the meantime, does anyone know of a good way to grab that measurement value?

1 Like
2

Hi @obiwan

Can you confirm: are you using the new Unified Alerting platform or the legacy platform?

3

@mattabrams, yes, I confirm that it is the new Unified Alerting.

From my admin/settings page:

[alerting]
  enabled	false

[unified_alerting]
  enabled	true
1 Like
4

Note: I am now running version: 8.3.3

So far, this is what I have been able to come up with:

{{with .ValueString}}
  Metric Measurement: {{. | reReplaceAll `metric='.*' labels={.*} value=` ""}} 
{{end}}

I strip out the extra stuff so all that is left is the raw value:

Metric Measurement: [ 0.6710963455149501 ]

Maybe I can figure out how to format the number (which is still a string at this point).

2 Likes
5

May I ask how to solve it, I have submitted a problem from github, but it has not been solved so far. I think your problem is similar to my expectation. Can you help guide me?
github：Alerting: $value does not display in expected format in message template · Issue #50865 · grafana/grafana · GitHub

6

I applied similar approach to @obiwan - here’s my alert format

{{ define "alertMessageTitle" }}
  {{if eq .Status "firing" }} [Alerting]{{end}}{{ if gt (.Alerts.Resolved | len) 0 }} [OK] {{end}} - {{range .CommonLabels.SortedPairs }} {{ if eq .Name "alertname"}} {{ .Value }}{{end}}{{end}}
{{ end }}

{{ define "alertBody" }}
*{{ or .Annotations.message .Labels.alertname }}*
	{{ with .ValueString }}
{{- . | reReplaceAll `\[\s` "" | reReplaceAll `\],\s` "\n" | reReplaceAll `\]` "" | reReplaceAll `var='[A-Z0-9a-z]+'\s` "" | reReplaceAll `labels=\{[A-Za-z0-9_\-\=\s\.:,]+\}\s` "" | reReplaceAll `metric='(.*)' ` " *$1* " | reReplaceAll `value=([0-9\.]+)` "($1)" }} 
	{{ end }}
{{- if gt (len .GeneratorURL) 0 }}{{ printf "\n%s%s\n" "Alert: " .GeneratorURL }}{{ end }}
{{- if gt (len .SilenceURL) 0 }}Silence: {{ printf "%s\n" .SilenceURL }}{{ end }}
{{- if gt (len .DashboardURL) 0 }}Dashboard: {{ printf "%s\n" .DashboardURL }}{{ end }}
{{- if gt (len .PanelURL) 0 }}Panel: {{ printf "%s\n" .PanelURL }}{{ end }}
{{- end }}

{{ define "alertMessage" }}
  {{ if gt (len .Alerts.Firing) 0 }}
    {{ range .Alerts.Firing }} {{ template "alertBody" .}} {{ end }}
  {{ end }}
  {{ if gt (len .Alerts.Resolved) 0 }}
    {{ range .Alerts.Resolved }} {{ template "alertBody" .}} {{ end }}
  {{ end }}
{{ end }}
3 Likes
7

Thanks for sharing @jlcrow! This is exactly the sort of formatting I was looking for. It’s really not easy to format these strings to get back to the pre-unified alerting sort of formatting.

1 Like
8

Yeah, I was taken off guard too, considering most are probably using the information from the old alert style and the template doesn’t give you a real way to deal with it, just a ValueString. I’d love to figure out how to get my dashboard images back in slack too.

9

Were you using the grafana image renderer? I know I had to re-add some configuration (Grafana 9 Alert Migrations: legacy alert screenshot settings aren't automatically migrated to new alerts) to get those back showing up.

1 Like
10

*edit: I figure it out; it was the regex for labels not expecting to be labels={}. Updated the template at the very end of this post; just swapped the + for * to match zero to unlimited instead of 1 to unlimited.

So I’m seeing a scenario where I am getting output with the following from the above set of reReplaceAlls and I can’t wrap my head around why it is keeping labels={} in here:

**Network 1 2u via pfsense** : labels={} 96

The fille ValueString is:
[ var='E0' metric='Network 1 (2u) via pfsense' labels={} value=92.83333333333333 ]

I thought it might be parenthesis so I tried to remove them and then the ValueString was:
[ var='E0' metric='Network 1 2u via pfsense' labels={} value=95.8 ]

But I still have the same output.

Here is my full contact point template:

{{ define "__discord_text_alert_list" }}{{ range . }}{{ with .ValueString }}
{{- . | reReplaceAll `\[\s` "" | reReplaceAll `\],\s` "\n" | reReplaceAll `\]` "" | reReplaceAll `var='[A-Z0-9a-z]+'\s` "" | reReplaceAll `labels=\{[A-Za-z0-9_\-\=\s\.:,]+\}\s` "" | reReplaceAll `metric='(.*)' ` "**$1**: " | reReplaceAll `value=([0-9\.]+)` "$1" }}
{{ end }}
{{ if gt (len .PanelURL) 0 }}**Panel**: {{ .PanelURL }}{{ end }}
{{ end }}{{ end }}

{{ define "discord.message" }}{{ if gt (len .Alerts.Firing) 0 }}**[ALERTING]**
{{ template "__discord_text_alert_list" .Alerts.Firing }}{{ if gt (len .Alerts.Resolved) 0 }}
{{ end }}{{ end }}{{ if gt (len .Alerts.Resolved) 0 }}**[RESOLVED]**
{{ template "__discord_text_alert_list" .Alerts.Resolved }}{{ end }}{{ end }}

I have other scenarios where I am seeing the output just as I expect:

**google** : 17.617466666666665
**wan2 gw** : 7.962100000000001

Any thoughts as to what is going on here? This syntax makes my head want to explode!

*edit: here is the updated regex to deal with the possibility of labels={}:

{{ define "__discord_text_alert_list" }}{{ range . }}{{ with .ValueString }}
{{- . | reReplaceAll `\[\s` "" | reReplaceAll `\],\s` "\n" | reReplaceAll `\]` "" | reReplaceAll `var='[A-Z0-9a-z]+'\s` "" | reReplaceAll `labels=\{[A-Za-z0-9_\-\=\s\.:,]*\}\s` "" | reReplaceAll `metric='(.*)' ` "**$1**: " | reReplaceAll `value=([0-9\.]+)` "$1" }}
{{ end }}
{{ if gt (len .PanelURL) 0 }}**Panel**: {{ .PanelURL }}{{ end }}
{{ end }}{{ end }}

{{ define "discord.message" }}{{ if gt (len .Alerts.Firing) 0 }}**[ALERTING]**
{{ template "__discord_text_alert_list" .Alerts.Firing }}{{ if gt (len .Alerts.Resolved) 0 }}
{{ end }}{{ end }}{{ if gt (len .Alerts.Resolved) 0 }}**[RESOLVED]**
{{ template "__discord_text_alert_list" .Alerts.Resolved }}{{ end }}{{ end }}
3 Likes
11

One more update; found that I had some forward slashes in labels due to docker image names (e.g. - mbentley/nginx:latest) so just and updated regex:

{{ define "__discord_text_alert_list" }}{{ range . }}{{ with .ValueString }}
{{- . | reReplaceAll `\[\s` "" | reReplaceAll `\],\s` "\n" | reReplaceAll `\]` "" | reReplaceAll `var='[A-Z0-9a-z]+'\s` "" | reReplaceAll `labels=\{[A-Za-z0-9_\-\=\s\.:,\\/]*\}\s` "" | reReplaceAll `metric='(.*)' ` "**$1**: " | reReplaceAll `value=([0-9\.]+)` "$1" }}
{{ end }}
{{ if gt (len .PanelURL) 0 }}**Panel**: {{ .PanelURL }}{{ end }}
{{ end }}{{ end }}

{{ define "discord.message" }}{{ if gt (len .Alerts.Firing) 0 }}**[ALERTING]**
{{ template "__discord_text_alert_list" .Alerts.Firing }}{{ if gt (len .Alerts.Resolved) 0 }}
{{ end }}{{ end }}{{ if gt (len .Alerts.Resolved) 0 }}**[RESOLVED]**
{{ template "__discord_text_alert_list" .Alerts.Resolved }}{{ end }}{{ end }}
1 Like
12

Grafana v9.0.2.
My ValueString returns something like this:
[ var='C' labels={device=/dev/mapper/vg_system-lv_root, env=test, fstype=xfs, host=host03, instance=10.10.10.10:9100, job=node-exporter, mountpoint=/} value=1 ], [ var='B' labels={device=/dev/mapper/vg_system-lv_root, env=test, fstype=xfs, host=host03, instance=10.10.10.10:9100, job=node-exporter, mountpoint=/} value=87.20739379429632 ]
I described it in the section ‘Alerting’ - ‘Alert rules’ - ‘Summary and annotations’ like this:
{{ $value | reReplaceAll [\s"" | reReplaceAll],\s"\n" | reReplaceAll]"" | reReplaceAllvar=‘[A-Z0-9a-z]+’\s"" | reReplaceAlllabels={[A-Za-z0-9_-=\s.:,\/]}\s"" | reReplaceAllmetric='(.)’ "**$1**: " | reReplaceAllvalue=([0-9.]+) "$1" }}%
And the message comes with two values: 87.35772882917088 and 1. Like this:
“mountpoint: / , host: host03, current value: 87.35772882917088
1”
Could tell me please how to remove value 1 which is in the line [ var='C' labels={device=/dev/mapper/vg_system-lv_root, env=test, fstype=xfs, host=host03, instance=10.10.10.10:9100, job=node-exporter, mountpoint=/} value=1 ]?

13

Looking at your output:

it’s showing that there are two queries that are returning values that are being triggered by one or more conditions:

[ var='C' labels={device=/dev/mapper/vg_system-lv_root, env=test, fstype=xfs, host=host03, instance=10.10.10.10:9100, job=node-exporter, mountpoint=/} value=1 ]
and
[ var='B' labels={device=/dev/mapper/vg_system-lv_root, env=test, fstype=xfs, host=host03, instance=10.10.10.10:9100, job=node-exporter, mountpoint=/} value=87.20739379429632 ]

Look at the alert expression to see why it is triggering both B and C.

1 Like
14

Would be great to have Value on Create Alert page on “Summary and annotations section.” with ability to use template values there ($labels $values $value).
Creating new template for just Value formatting is overkill.

2 Likes
15

I just wondering why not created default template to have behaviour identical to times before to unified alerting?

I can see value of “value_string” only in Alerting Groups, why not to have back that magic button Alert Preview.
And no matter how I modify message, I can’t get value of ValueString, is it working only in template?

Grafana 9.4.7