Alert Content disappeared in MS Teams

Grafana Alerting
1

Hello all,

we are using the MS Teams contact point to receive alerts created by Grafana. Until recently this worked pretty well, but suddenly all the alert message content disappeared. Now all the alerts look like this:

image
image1238×659 31.7 KB

The strange thing is that the content of the old alert messages - even those sent weeks ago - has also disappeared. I think there can only be two causes for this problem:

  1. The alert messages sent by Grafana via Webhook may load additional content from Grafana. In this case, due to e.g. a network error, Teams could not access the data to be (possibly) loaded and therefore the content is no longer displayed. (Here I have to add that the alerts do not contain images. They only contain additional text when displayed correctly. That is why I think this options is less likely.)

  2. The messages sent by Grafana Alerts are displayed as a card. Possibly the syntax required by Teams has changed and now all alert messages (even the older ones) cannot be displayed properly.

Unfortunately, I cannot see the exact message sent from Grafana to MS Teams. This makes troubleshooting really hard. If you have more details about Grafana Alerting and the MS Teams contact point please let me know. Or maybe someone is experiencing the same issue.

Grafana Version: v8.5.3 (409e9bc9a5)
We are not using any message templates.

Thank you all in advance!

1 Like
2

Hi! Thanks for writing such a comprehensive description of the issue! This is not an issue I have seen before. That said, I would recommend upgrading both Grafana from v8.5.x to the latest version of Grafana, and I would also recommend migrating from old alerting to the new unified alerting experience (Upgrade Alerting | Grafana documentation).

This issue should be fixed in Grafana 9.1 and later as we migrated Teams notifications from Office365 connector cards to Adaptive Cards. However, you must be using the new unified alerting experience to get this feature. I don’t think it will be backported to old alerting as that is now deprecated, so upgrading is recommended!

Kind regards

1 Like
3

@danielsous Did you find any resolution to this? We are having similar issue since today morning. Weirdly it affected old alerts as well.

Seems like Teams changed something on their end. Now, it’s clipping everything after ‘\n’ in the payload.

Let me know if you find something.

1 Like
4

I have not been able to fix the issue yet, but I plan to upgrade Grafana hoping that it will resolve the issue. If anyone has new information about a possible solution, please let us know.

5

We are already using unified alerting. I seems like MS Teams changed the format of these “Office365 connector cards”. So, updating seems to be the only possible solution.

6

We are experiencing the same problem, which also affects past alerts, Grafana v9.0.6 and MS Teams v1.5.00.31168. Weirdly enough, it looks fine on one of our colleague’s @edromanenco MS Teams…

2 Likes
8

@maxikem @danielsous

The weird part is when I open Teams from my phone, I see alerts having complete information. However, when I check from the Desktop Client, it is clipping of the information after first ‘\n’.

Is anyone else having the same issue?

1 Like
9

The most weird thing is that today it suddenly is working again. We did not upgrade Grafana.
Im sure this was caused by some changes in MS Teams in the background.

2 Likes
10

Yes, it fixed for us as well without making any changes on our end. I just had to restart my machine.