Notification Provisioning not creating policies

Grafana Alerting
1

Hi!

I’m trying to provision a notification policy via kustomize and helm, and the logs show that the configmap is being read, but nothing gets created, and there are no further logs. The logs are from setting the log level to DEBUG in the helm-values yaml.

This is the test notification policy yaml I’m using:

apiVersion: 1

policies:
- orgId: 1
  receiver: External Alerts
  routes:
  - receiver: "Critical"
    group_by: ["grafana_folder", "alertname"]
    matchers:
      -  alert_level =~ "critical_production|security"
    object_matchers:
      - ["alert_level", "=", "critical_production|security"]
    mute_time_intervals:
      - WorkingHours
  - receiver: "Non-Critical - MS Teams"
    group_by: ["grafana_folder", "alertname"]
    matchers:
      -  alert_level =~ "non-critical_production|non-critical_non_production|critical_non-production"
    object_matchers:
      - ["alert_level", "=~", "non-critical_production|non-critical_non_production|critical_non-production"]
    mute_time_intervals:
      - WorkingHours
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

I have already provisioned the WorkingHours mute time interval, and all other provisioning I’ve attempted as worked, including contact points and alerts. It’s only the notification policies that seem to silently fail.

I have tried multiple different policy yamls, including creating the policies via the UI and using the exported JSON, to no success. I’ve tried various combinations of object matchers vs matchers, simplified and more complicated yamls, but nothing provisions.

Directly using the HTTP API to create the policies works, but this is less than ideal for our setup.

Thank you for your help in advance!

2

welcome @drdjing

is it possible that matchers is missing quotation marks?

3

Hi!

Sorry it took me so long to reply, busy times.

According to the documentation the matchers doesn’t need quotation marks around the key, and if I try to add them the yaml errors.

The debug logs for grafana show that the configmap is being read and configured, but it just silently does nothing after that.