Alerts stops flowing via webhook aftersome time ( not all of them )

Grafana Alerting
, ,
1

→

  • What Grafana version and what operating system are you using?
    Grafana v9.1.3

  • What are you trying to achieve?
    Alert notification to discord

  • How are you trying to achieve it?
    Manual Dashboard/Alerts from UI / helm manifests ( dashboard from export, Alerts taken from API )

  • What happened?
    Alerts works just fine, but after some time ( few days ) i see 400 bad request or 404 page not found in logs and alerts are not sent to discord. Howeever not all alerts are affected.

  • What did you expect to happen?
    We expect it to send allerts based on our configuration.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

logger=alerting.notifier.discord t=2022-09-26T10:40:33.557434061Z level=error msg=“failed to send notification to Discord” err=“webhook response status 400 Bad Request” ││ logger=alertmanager org=1 t=2022-09-26T10:40:33.557518479Z level=error component=dispatcher msg=“Notify for alerts failed” num_alerts=3 err="Ops - Alerts/discord[0]: notify retry canceled due to unrecoverable error after 1 attempts: ││ logger=context traceID=00000000000000000000000000000000 userId=1 orgId=1 uname=admin t=2022-09-26T10:40:43.496066756Z level=error msg=“404 page not found\n” error=“404 page not found\n” remote_addr=172.26.1.98 traceID=000000000000000