Mute alert for specific helm charts

Grafana
1

Hello all

We use Grafana together with Kubernetes and Helm.

To alert we have different notifiers.

These are defined in the helm charts like ~so:

notifiers:
  notifiers.yaml:
    apiVersion: 1
    notifiers:
      - name: slack-notification-channel
        type: slack
        uid: notification-notifier
        is_default: false
        send_reminder: true
        frequency: 1h
        disable_resolve_message: false
        settings:
          username: Grafana XY
          uploadImage: false
          url: https://hooks.slack.com/services/xxxxxxxxx
    delete_notifiers:
      - name: slack-notification-channel
        uid:notification-notifier

You see for deployment we also need a workaround (delete and recreate the same notifier) but this is only a side effect towards the question.

What I am looking for, for our dev environments I’d like to swallow these alerts.
As we use the same BOARD everywhere but not the same notifier I want to set this somehow on notifier level, e.g. like so:

notifiers:
      notifiers.yaml:
        apiVersion: 1
        notifiers:
          - name: slack-notification-channel
            type: MUTE   <----------------------------!!
...
delete_notifiers:
          - name: slack-notification-channel

I already thought about two options but none make me really happy:

  • Use a second slack webhook and spam a random channel
  • Use a fake webhook somehow and send the alert to Nirvana

What would be a graceful swallow of dev “errors”?

Thanks community

2

Hi dominikmeyer01,
Did you manage to find a better solution to your situation?