Grafana pvc doesnt save data

Grafana Configuration
1

Im using grafan 9.3.1 with kubernetes created with helm. I have set in the values.yaml of grafana chart. the pvc settings:

persistence:
  type: pvc
  enabled: true
  storageClassName: longhorn
  accessModes:
    - ReadWriteOnce
  size: 10Gi
  finalizers:
    - kubernetes.io/pvc-protection

I can see that the volume is bond:

  grafana:
    Mounts:
      /tmp/dashboards from sc-dashboard-volume (rw)

– kubectl describe pvc

Name:          prometheus-grafana
Namespace:     monitoring
StorageClass:  longhorn
Status:        Bound
Volume:        pvc-e42a3d09-d4d0-4b50-bf87-dbfb5b189c18
Labels:        app.kubernetes.io/instance=prometheus
               app.kubernetes.io/managed-by=Helm
               app.kubernetes.io/name=grafana
               app.kubernetes.io/version=9.3.1
               helm.sh/chart=grafana-6.49.0
Annotations:   meta.helm.sh/release-name: prometheus
               meta.helm.sh/release-namespace: monitoring
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io
               volume.kubernetes.io/storage-provisioner: driver.longhorn.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       prometheus-grafana-86b496fffd-gkld7
Events:        <none>

— kubectl get pvc

prometheus-grafana                   Bound    pvc-e42a3d09-d4d0-4b50-bf87-dbfb5b189c18   10Gi       RWO            longhorn       24h

I see that the mount path is to /tmp/dashboards by default and i have tried to change it to /var/lib/grafana or /app/dashboards but it doesn’t help and also if i exec inside the pod I don’t see the mount directory.

If i delete the pod all the dashboard data is deleted. haven’t found any docs on how to solve this.

How can i make it work?

2

I found the issue was me working on an old pod and i had another pod that was failing which was the new pod with the pvc settings and the problem was that busybox image was not found and apperntly if i set pvc then the busybox does some settings and is needed.