Alert annotation in panel

Grafana Alerting
1

hi,

i replaced my old panel-based alerts with the new alerts. so i created alert rules which contain annotations with a Dashboard UID and a Panel ID. this also works, so i can seee the green/red heart symbol in the dashboard panel if this alert fires.

but there is no annotation for start/end as with the old alerts (red/green annotation line). i opened the sqlite database to query the database and there is an entry in the annotations which has the alert id, but the columns for dashboard-id (id, not uid) and the panel-id are both 0.

is there anything i can do, so that the unified alerting system creates an annotation which is shown in a specific dashboard/panel?

thanks

3 Likes
2

Hi! Thanks for this. I will see if I can find some time to look into this later this week and come back with an answer.

2 Likes
3

thanks.

i also have a problem with the the alert-system: it looks like that the alert-id’s were re-used. but i think this is a bug, so i created a ticket (Wrong alert state history · Issue #59813 · grafana/grafana · GitHub). only wanted to mention, perhaps there is a correlation (also i don’t think so).

4

i replaced my old panel-based alerts with the new alerts. so i created alert rules which contain annotations with a Dashboard UID and a Panel ID. this also works, so i can seee the green/red heart symbol in the dashboard panel if this alert fires.

but there is no annotation for start/end as with the old alerts (red/green annotation line). i opened the sqlite database to query the database and there is an entry in the annotations which has the alert id, but the columns for dashboard-id (id, not uid) and the panel-id are both 0.

is there anything i can do, so that the unified alerting system creates an annotation which is shown in a specific dashboard/panel?

is there any news on this topic, we have exactly the same problem. We use the latest Grafana version 9.3.1

Thanks, BR

5

Hi, same confusion here about whether the alert state annotations disappeared from any panel at some point.

6

Hi,

after updating to 9.3.2 my alert annotations also disapeared.

The Red/Green hearts on the panels title are working correctly.

I hope this could be fixed in future releases.

Thanks for the great grafana Software!

7

same for me. After we updated to 9.3.2 today the annotations disappeared from the panels.

8

We’ve found the issue that broke annotations in 9.3. It should be fixed in 9.3.3. Alerting: Fix missing dashboard/panelID links in annotations by alexweav · Pull Request #60926 · grafana/grafana · GitHub and Annotations: Fix EpochEnd being zero for Alert-generated annotations by alexweav · Pull Request #60931 · grafana/grafana · GitHub.

9

Hi,
can it be that the fix Annotations: Fix EpochEnd being zero for Alert-generated annotations by alexweav · Pull Request #60931 · grafana/grafana · GitHub
is still not included in version 9.3.4 or 9.3.6? ty

10

Hi! Due to an error the fix was not backported to 9.3.x. It has just been fixed in 9.3.8 and 9.4.1, both of which were released this afternoon!

11

hi georgerobinson, thanks for the reply - fyi - the problem with alert annotation in panel was solved with Grafana version 9.4.1 at our site.

12

Hi! That’s good to know! I’d recommend upgrading to 9.4.3 as soon as possible due to some bugs found in 9.4.1 and 9.4.2 https://github.com/grafana/grafana/blob/main/CHANGELOG.md#943-2023-03-02.

13

The annotations is now showing in the normal panels, but I still have an issue with the Annotation panel, it only shows the manually created annotations and not the ones created by the alerting.
Is there a solution to this?
(Using 9.4.3)

14

Hi! I upgraded to 9.4.3 but I do not see Annotations created from alerting. Are there any updates on this issue? Many thanks!

15

Hi! Can you show us your alert rule, including the Dashboard UID and Panel ID annotations?

16

image
image917Ă—906 52.5 KB

I created the rule through the “Create alert rule from this panel” button…

But also manually added annotations do not show up after adding.

image

I see the marker when trying to add, but after pressing Save the marker disappears.
Alerting also with rendering and sending into a Telegram chat works just fine.

Many thanks in advance!

EDIT: After investigating the issue and restarting several times and also changing annotiation configuration several times it seems to work now, I tried to figure out the one step that fixed the issue but unfortunately I am not able to provide the information…

Thanks again, it seems the problem is fixed!