Configmap Helm Chart for managing Dashboards in Grafana 9

Grafana Dashboards
1

I’ve been experimenting with using a combination of dashboards created in the database and helm/configmaps and found that dashboards, in the same folder, with the same name or UID interact with each other with unintended consequences. Am I doing something wrong? Is this same behavior in Grafana 10?

Using the configuration below, I found the following to be true.

  1. Dashboards that are provisioned by helm/configmap cannot be edited or deleted from with the Grafana interface. This is a good thing, protecting against accidental editing.

  2. Dashboard definitions that have the same name and/or UID in the same folder will allow overwriting existing dashboards, regardless of being initially created in Grafana or by helm/configmap. This is undesirable behavior, but can be recovered, see #3.

  3. Dashboards definitions that have first been created using the database and Grafana and overridden database by helm/configmap changes can be reverted by first removing the helm chart and reverting the dashboard version to a value prior to the helm changes. Helm changes are identified by the user ‘anonymous’. This is annoying and undesirable behavior.

  4. Grafana must be configured not to allow configmaps to delete dashboards, because of the risk of a helm chart with a dashboard of duplicate identification also deleting an existing dashboard created in the database. This means that in order to delete a dashboard created by helm, you have to both delete the helm chart and then manually delete the dashboard from the Grafana GUI. This is annoying and undesirable behavior.

Grafana Configuration
    dashboardProviders:
      dashboardproviders.yaml:
        apiVersion: 1
        providers:
        - name: 'default'
          orgId: 1
          folder: 'Default'
          type: file
          disableDeletion: true
          editable: true
          options:
            path: /var/lib/grafana/dashboards/default
        - name: 'apps'
          orgId: 1
          folder: 'apps'
          type: file
          disableDeletion: true
          editable: true
          options:
            path: /var/lib/grafana/dashboards/apps

Here are the detailed test results

First, create a dashboard using the DB in folder apps, db_created_dashboard with same uid. Look at the panel name to reflect any changes in the dashboard. Each JSON dashboard has a panel with the test name.

  1. Add a helm dashboard with same folder, name and different uid, what happens? Helm install -f values.yaml test-1-1 . This overwrites the db dashboard, but it can be recovered after the helm chart is deleted.

  2. Add a helm dashboard with different name and same uid and folder, what happens? Helm install -f values.yaml test-1-2 This overwrites the db dashboard, but it can be recovered after the helm chart is deleted. The dashboard name is changed as part of the helm change and reverted as part of the database version restore.

  3. Add a helm dashboard with different name and different uid and the same folder, what happens? Helm install -f values test-1-3 .This creates a new, unique dashboard without changing the other dashboard.

  4. Update a helm dashboard with same name and different uid, what happens? Helm upgrade -f values.yaml test-1-1 . It may be necessary to revert the db dashboard and to re-create the helm configmap with test 1 (from scratch), if that chart has been deleted or modified since test 1. This overwrites the db dashboard, but it can be recovered after the helm chart is deleted.

  5. Update a helm dashboard with different name and same uid, what happens? Helm upgrade -f values.yaml test-1-2 . It may be necessary to revert the db dashboard and to re-create the helm configmap with test 2 (from scratch), if that chart has been deleted or modified since test 1. This overwrites the db dashboard, but it can be recovered after the helm chart is deleted.

  6. Update a helm dashboard with different name and different uid, what happens? This succeeds.

  7. Delete a helm dashboard with same name and different uid, what happens? Helm uninstall 1-1 The dashboard persists with the latest helm values and can then be reverted to a database backup. Kubernetes/Helm does not notify the user that db created dashboard exists and has been overwritten.

  8. Delete a helm dashboard with different name and same uid, what happens? Helm uninstall 1-2 The dashboard persists with the latest helm values and can then be reverted to a database backup. Kubernetes/Helm does not notify the user that db created dashboard exists and has been overwritten.

  9. Delete a helm dashboard with different name and different uid, what happens? This does not delete the dashboard from Grafana, but does allow for it to be deleted from manually from Grafana.