Hi,
I’ve recently been working with Grafana v6.1.3
I’ve built some Alert rules where I can adjust the threshold to get the alert rule into an ‘Alerting’ state.
I have a webhook type alert set up as the default notification channel on all alerts, however when the alert goes into an ‘Alerting’ state, I can’t see that the webhook call is being made.
From the Logfiles I can see;
t=2020-06-12T14:17:01+0100 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=1 newState=alerting prev state=pending
t=2020-06-12T14:17:01+0100 lvl=info msg=“Sending webhook” logger=alerting.notifier.webhook
t=2020-06-12T14:17:31+0100 lvl=eror msg=“Failed to send webhook” logger=alerting.notifier.webhook error=“context deadline exceeded” webhook=“Spark Incident”
t=2020-06-12T14:17:31+0100 lvl=eror msg=“failed to send notification” logger=alerting.notifier uid=X-kP8PiMk error=“context deadline exceeded”
I can make a GET request to the webhook URL and see that this works as expected, however when i expect the system to perform the webhook as a result of the alert threshold being breached, nothing seems to be generated.