Multiple Grafana w/ MySQL (AKS HA setup) :: Alert notification provisioning error

Grafana Alerting
, , ,
1
  • What Grafana version and what operating system are you using?

Grafana: v8.0.5 (cbb2aa5001)
OS: Linux prometheus-dbp01-grafana-85dcc79d6-4btbw 5.4.0-1059-azure #62~18.04.1-Ubuntu SMP Tue Sep 14 17:53:18 UTC 2021 x86_64 Linux

note: additional sidecars for provision of datasources, notifier and dashboards

  • What are you trying to achieve?

Goal is to

  • use MySQL instead of SQlite3 as database to circumvent the “Database is locked” issue
  • have more than 1 pod so a rolling update won’t cause service interruption for the Grafana users.
  • How are you trying to achieve it?

We changed to 1-pod/SQlite setup of the prometheus-operation config to have two replicas and use mysql instead of SQlite3.

  • What happened?

First pod (instance) starts up without any issues, 2nd pods errors out and is not able to start

  • What did you expect to happen?

I expected both instances to be able to share one database and both pods (instances) to start up.

  • Can you copy/paste the configuration(s) that you are having problems with?
grafana:
  replicas: 2
  podDisruptionBudget:
    maxUnavailable: 1
  grafana.ini:
    database:
      type: mysql
      host: 10.0.8.15:3306
      name: grafana-cluster-stage
      user: 'grafana-cluster-stage@some-mysql-dbms'
      password: ****************
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

service init failed: Alert notification provisioning error: alert notification with same name already exists

  • Did you follow any online instructions? If so, what is the URL?

DB created, pointed 1st Grafana to it, Grafana created tabled, all good.
Started 2nd Grafana … not working

1st Grafana works, 2nd not

2

support for Unified Alerting in HA setups introduced in Grafana 8.2

3

Hi Matt.

Thanks for the instant reply. I updated our setup to go for Grafana 8.2.3 in the prometheus-operator Chart values.yaml and we are really now presented with v8.2.3.

grafana:
  image:
    repository: grafana/grafana
    tag: 8.2.3

but sadly I get the same result of service init failed: Alert notification provisioning error: alert notification with same name already exists. I also truncate all tables in the MySQL DB but same issue. enabled unified_alerting and disabled alerting.

Do I still miss something that needs to be enabled for alert deduplication or do I just need more patience till HA setup is in a stable state?

4

Issue is fixed (See: Alert notification provisioning error: alert notification with same name already exists · Issue #42229 · grafana/grafana · GitHub)

If the notifier uid is supposed to be not longer than 40 chars.

1 Like
5

thanks for working with the squad to get this sorted @manfredackermann . super interesting edge case

1 Like
6

This topic was automatically closed after 365 days. New replies are no longer allowed.