Grafana new alerting: no PagerDuty custom details?

Grafana Alerting
,
1

  • What Grafana version and what operating system are you using?
    Grafana v8.3.0 (914fcedb7) on docker

  • What are you trying to achieve?
    Sending alerts to PagerDuty with labels, so they can be handled appropriately at PagerDuty’s side

  • How are you trying to achieve it?
    Switched to new Grafana alerting on that Grafana instance, then tested an alert using PagerDuty API v2 as the contact point.

  • What happened?

All labels and tags of the alert just appear in a details.firing field, but not as separate key-value pairs in the payload. This is the JSON that PagerDuty received:

{
  "client": "Grafana",
  "client_url": "https://grafana.example.com/",
  "contexts": [
    {
      "href": "https://grafana.example.com/",
      "text": "External URL",
      "type": "link"
    }
  ],
  "description": "[FIRING:1]  (alert name)",
  "event_type": "trigger",
  "incident_key": "480b3d163bb8995e036cf2a22e7ac3b9295...",
  "service_key": "36b0c8beeb4647...",
  "details": {
    "firing": "
Value: [ var='B' labels={environment=prod} value=6 ], [ var='C' labels={environment=prod} value=1 ]
Labels:
 - alertname = my alert name
 - env = prod
 - rule_uid = 0g96-O2..
 - environment = prod
Annotations:
 - custom_details.environment = prod
 - description = my alert description
 
Source: https://grafana.example.com/alerting/0g96-.../edit
Silence: https://grafana.example.com/alerting/silence/new?...
Dashboard: https://grafana.example.com/d/I0hydA...
Panel: https://grafana.example.com/d/I0hyd...?viewPanel=6
",
    "num_firing": "1",
    "num_resolved": "0",
    "resolved": ""
  }
}
  • What did you expect to happen?

My labels or tags should appear in the JSON as separate fields. For example:

"details": {
    "environment": "prod",
  }

I’ve looked at the code for the new PagerDuty integration: grafana/pagerduty.go at main · grafana/grafana · GitHub

I don’t see any logic that can set custom fields.

  • Can you copy/paste the configuration(s) that you are having problems with?

Don’t know what to paste here that could be relevant

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

No errors.

  • Did you follow any online instructions? If so, what is the URL?

No

1 Like