Gents, first let me start by thanking everyone at grafana and the community here. I love the project and have a loyal user of the onprem version for years now.
Recently, we started to test grafana cloud just for the OnCall function. Everything is working great and I can see this being a real game changer. However, starting yesterday we started noticing something now.
There have been blank alerts coming in. Our template is simple, but what I think is happening is that we have empty payload getting sent over to the webhook address.
I was hoping there was a way to get more information from some of these incoming alerts. For example the IP so we can see if this is an accidental alert from one of our sites or someone on the internet just guessed our webhook URL and is playing with us.
Hey there! The Support team might be able to check the Grafana server logs for more details. I recommend opening a ticket and include the timestamp when the alert was received so they can review since those logs are not exposed for you to review directly. Hopefully that will give more insight, and the team can also check for any known issues that might be related.
@acl at first, thank you a lot for trying the product! Unfortunately for now it’s not possible to gather information about the IP address that sent us an alert. Such logging is in our backlog and we hope to roll it out in the mid-term.
For now, as a hotfix, I could suggest recreating the integration. It will generate the new unique link which you could put in your alert sources one by one (if you have a few of them) and try to identify the source generating such a payload.
We use librenms to send alarms via webhook. After a recent update, we just stared receiving blank payload. What helped us was the time stamps, we were able to correlate it back to the source. Thanks again for the help.
