How to sort items in aggregated host alert by value

Grafana Alerting
,
1

Hello all,

I have a working alert that tells me about dropped packets, grouped by hosts that works well so far:

dropped_packets, host=A,  Incoming network eth1,  drops=2786
dropped_packets, host=B,  Incoming network eth1,  drops=1.048778e+06
dropped_packets, host=C,  Incoming network eth0,  drops=9789
dropped_packets, host=D,  Incoming network eth0,  drops=5345
dropped_packets, host=E,  Incoming network eth0,  drops=127392

But as you can see I have 2 issues:

  • Numbers are not formatted
  • Grouped messages are not sorted by drops by rather by the host (or it seems)

I have the following templates on the notifications ā†’ contact points ā†’ Message templates

{{ define "droppedalert" }}
  {{ if gt (len .Annotations) 0 }}{{ range .Annotations.SortedPairs }}{{ if  eq .Name  "description" }}{{ .Value }}{{ end }}{{ end }}{{ end }}
{{ end }}

And then:

{{ define "droppedmessage" }}
  {{ if gt (len .Alerts.Firing) 0 }}
      Detected packet drops on  {{ len .Alerts.Firing }} hosts:
      {{ range .Alerts.Firing }} {{ template "droppedalert" .}} {{ end }}
  {{ end }}
{{ end }}

I was checking the official template documentation but so far I could not find a way to do what I need.

Does any of you have some experience doing something similar?

Thanks in advance,

2

So looking into the Prometheus alert manager code I found than SortedPairs treats the values as strings:

// SortedPairs returns a sorted list of key/value pairs.
func (kv KV) SortedPairs() Pairs {
	var (
		pairs     = make([]Pair, 0, len(kv))
		keys      = make([]string, 0, len(kv))
		sortStart = 0
	)
	for k := range kv {
		if k == string(model.AlertNameLabel) {
			keys = append([]string{k}, keys...)
			sortStart = 1
		} else {
			keys = append(keys, k)
		}
	}
	sort.Strings(keys[sortStart:])

	for _, k := range keys {
		pairs = append(pairs, Pair{k, kv[k]})
	}
	return pairs
}

Iā€™m not very familiar with Golang but is possible to create a function that returns a list sorted by value without editing the template.go that comes with Grafana?

Note: Assuming this cannot be done and instead you need to rely on a different component like OnCall to get it done (tried with Grafana 9.2.0).