- What Grafana version and what operating system are you using?
Grafana v8.4.2 (9fc02d717f)
Self-hosted on CentOS 7
- What are you trying to achieve?
Receive a single notification for an alert with multi-series data.
- How are you trying to achieve it?
Query A:
100-node_filesystem_free_bytes{mountpoint="/data",job=~"[MASKED]\\d\\d-gitlab-runner"}/node_filesystem_size_bytes{mountpoint="/data",job=~"[MASKED]\\d\\d-gitlab-runner"}*100
Expression B:
Classic condition
WHEN last() A IS ABOVE 80
- What happened?
The contact point received several E-Mails with the following alert values:
E-Mail 1
[
metric='{device="/dev/sdb", fstype="xfs", instance="[MASKED].214:9100", job="[MASKED]03-gitlab-runner", mountpoint="/data"}'
labels={device=/dev/sdb, fstype=xfs, instance=[MASKED].214:9100, job=[MASKED]03-gitlab-runner, mountpoint=/data}
value=80.08562988519785
]
E-Mail 2
[
metric='{device="/dev/sdb", fstype="xfs", instance="[MASKED].214:9100", job="[MASKED]03-gitlab-runner", mountpoint="/data"}'
labels={device=/dev/sdb, fstype=xfs, instance=[MASKED].214:9100, job=[MASKED]03-gitlab-runner, mountpoint=/data}
value=84.8801836529067
], [
metric='{device="/dev/sdb", fstype="xfs", instance="[MASKED]:9100", job="[MASKED]06-gitlab-runner", mountpoint="/data"}'
labels={device=/dev/sdb, fstype=xfs, instance=[MASKED]:9100, job=[MASKED]06-gitlab-runner, mountpoint=/data}
value=80.21943774425989
]
E-Mail 3
[
metric='{device="/dev/sdb", fstype="xfs", instance="[MASKED].146:9100", job="[MASKED]01-gitlab-runner", mountpoint="/data"}'
labels={device=/dev/sdb, fstype=xfs, instance=[MASKED].146:9100, job=[MASKED]01-gitlab-runner, mountpoint=/data}
value=80.39159745969712
],[
metric='{device="/dev/sdb", fstype="xfs", instance="[MASKED].214:9100", job="[MASKED]03-gitlab-runner", mountpoint="/data"}'
labels={device=/dev/sdb, fstype=xfs, instance=[MASKED].214:9100, job=[MASKED]03-gitlab-runner, mountpoint=/data}
value=86.63112023693209
], [
metric='{device="/dev/sdb", fstype="xfs", instance="[MASKED].218:9100", job="[MASKED]06-gitlab-runner", mountpoint="/data"}'
labels={device=/dev/sdb, fstype=xfs, instance=[MASKED].218:9100, job=[MASKED]06-gitlab-runner, mountpoint=/data}
value=80.2119252259404
]
E-Mail 4
[
metric='{device="/dev/sdb", fstype="xfs", instance="[MASKED].146:9100", job="[MASKED]01-gitlab-runner", mountpoint="/data"}'
labels={device=/dev/sdb, fstype=xfs, instance=[MASKED].146:9100, job=[MASKED]01-gitlab-runner, mountpoint=/data}
value=83.04804897410845
],[
metric='{device="/dev/sdb", fstype="xfs", instance="[MASKED].214:9100", job="[MASKED]03-gitlab-runner", mountpoint="/data"}'
labels={device=/dev/sdb, fstype=xfs, instance=[MASKED].214:9100, job=[MASKED]03-gitlab-runner, mountpoint=/data}
value=87.67082086590132
], [
metric='{device="/dev/sdb", fstype="xfs", instance="[MASKED].217:9100", job="[MASKED]05-gitlab-runner", mountpoint="/data"}'
labels={device=/dev/sdb, fstype=xfs, instance=[MASKED].217:9100, job=[MASKED]05-gitlab-runner, mountpoint=/data}
value=80.44677347948218
], [
metric='{device="/dev/sdb", fstype="xfs", instance="[MASKED].218:9100", job="[MASKED]06-gitlab-runner", mountpoint="/data"}'
labels={device=/dev/sdb, fstype=xfs, instance=[MASKED].218:9100, job=[MASKED]06-gitlab-runner, mountpoint=/data}
value=83.67876312896922
]
After the value of each time series becomes above 80 an e-mail is sent.
- What did you expect to happen?
I expect to receive only 1 E-Mail because the documentation says:
Use the classic condition expression to create a rule that triggers a single alert when its condition is met. For a query that returns multiple series, Grafana does not track the alert state of each series. As a result, Grafana sends only a single alert even when alert conditions are met for multiple series
Am I getting the documentation wrong? How to achieve only 1 notification after the first of time series reaches its value above 80?