Multiple instances alert appears as single isntance alert in grafana

Grafana Prometheus
1

Hello,

We are doing some research now to see if we should use Prometheus and Alert manager or the new Grafana unified alerting system to manage our alerts. I have observed a different behavior between a same alerts when it is defined in Grafana or directly in Prometheus and I do not know if it is a bug or not (hope you can help me with that).

Imagine that you define an alert both in Grafana and Prometheus. This alert is really simple and checks that the current value of a metric is above 5. The prometheus expression of the alert, when evaluated returns 2 metrics instances (one for each server impacted).
When I check the alert status in the alert manager (I’ve configured grafana to forward alert to the Prometheus alert manager) I can see two alert instances for the alert defined in prometheus while I only see one for the one defined in grafana. This is also confirmed when I look at the grafana alert rules panel where it also shows only one instance for that rule. Also when I look at the alert status details, I can see that the value field is filled with an array representing the different prometheus expressions instances.

Is it the expected behavior or should it be flagged as a bug?

I use the latest grafana community edition 8.3.4

Thanks for you help!

2

After reading more carefully the documentation, I realize that the alert based on a classic condition are always single dimension alert.
Therefore I’ve changed my alert definition to use a ‘Math’ condition, as explained in Annotations and labels for alerting rules | Grafana Labs. I have also add a label to uniquely identify each alert instance. This label will be set to one of the label of my first query by using template variable.
The alert fires up but has an error stating : “invalid format of evaluation results for the alert definition : frame cannot uniquely be identified by its labels: has duplicate results with labels {}”

It is as if the template variable were not correctly expanded.

Below are the details of my alert:

  • Grafana version: 8.3.4
  • Alerting system: unified alerting system
  • Rule type: Grafana managed alert
  • Alert query A: ibm_mq_statq_queue_depth{queue = “SYSTEM.MQTT.TRANSMIT.QUEUE”} (this metrics has the qmname label)
  • Expression of type Math: $A > 0
  • custom label qm = {{ $labels.qmname }}

My grafana config file contains:
[unified_alerting]
enabled = true

Did I do something wrong?