Cannot login to grafana kubernetes

Grafana Installation
1

Hello,

I have Grafana 6.7.4 installed on kubernetes. I can connect to grafana over an load balancer, I have definied a secret with admin, admin as user and password, but I can’t login to grafana, because there is always the error message invalid username or password.

These are my yamls:
configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
name: ini
labels:
component: grafana
data:
grafana.ini: |
[analytics]
check_for_updates = true
[grafana_net]
url = https://grafana.net
[log]
mode = console
[paths]
data = /var/lib/grafana/data
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
—
apiVersion: v1
kind: ConfigMap
metadata:
name: datasources
labels:
component: grafana
data:
datasources.yaml: |
apiVersion: 1
datasources:
- access: proxy
isDefault: true
name: prometheus
type: prometheus
url: No link because I am a new user prometheus:9090
version: 1
—
apiVersion: v1
kind: ConfigMap
metadata:
name: dashboardproviders
labels:
component: grafana
data:
dashboardproviders.yaml: |
apiVersion: 1
providers:
- disableDeletion: false
editable: true
folder: “”
name: default
options:
path: /var/lib/grafana/dashboards
orgId: 1
type: file

serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: grafana
---

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: grafana
  labels:
    component: grafana
spec:
  selector:
    component: grafana
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 3000

statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: grafana
  labels:
    component: grafana
spec:
  serviceName: grafana
  replicas: 1
  selector:
    matchLabels:
      component: grafana
  template:
    metadata:
      labels:
        component: grafana
    spec:
      serviceAccountName: grafana
      securityContext:
        runAsUser: 5000
        runAsGroup: 5000
        fsGroup: 5000
      containers:
      - name: grafana
        image: grafana/grafana:6.7.4
        imagePullPolicy: Always
        volumeMounts:
        - name: config
          mountPath: /etc/grafana
        - name: dashboards
          mountPath: /var/lib/grafana/dashboards
        - name: datasources
          mountPath: /etc/grafana/provisioning/datasources/
        - name: dashboardproviders
          mountPath: /etc/grafana/provisioning/dashboards/
        - name: notifiers
          mountPath: /etc/grafana/provisioning/notifiers/
        - name: data
          mountPath: /var/lib/grafana
        ports:
        - name: grafana
          containerPort: 3000
          protocol: TCP
        env:
        - name: GF_SECURITY_ADMIN_USER
          valueFrom:
            secretKeyRef:
              name: grafana-secret
              key: admin-user
        - name: GF_SECURITY_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: grafana-secret
              key: admin-password
        livenessProbe:
          httpGet:
            path: /api/health
            port: 3000
        readinessProbe:
          httpGet:
            path: /api/health
            port: 3000
          initialDelaySeconds: 10
          timeoutSeconds: 30
          failureThreshold: 10
          periodSeconds: 10
        resources:
          limits:
            cpu: 50m
            memory: 100Mi
          requests:
            cpu: 50m
            memory: 100Mi
      volumes:
        - name: config
          configMap:
            name: ini
        - name: notifiers
          emptyDir: {}
        - name: datasources
          configMap:
            name: datasources
        - name: dashboardproviders
          configMap:
            name: dashboardproviders
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      storageClassName: managed-nfs-storage
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 2Gi
  - metadata:
      name: dashboards
    spec:
      storageClassName: managed-nfs-storage
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 2Gi

secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: grafana-secret
  labels:
    component: grafana
type: Opaque
data:
  admin-user: YWRtaW4=
  admin-password: YWRtaW4=

This is the pod log:

kubectl logs -n monitoring grafana-0
t=2020-06-10T13:45:31+0000 lvl=info msg="Starting Grafana" logger=server version=6.7.4 commit=8e44bbc5f5 branch=HEAD compiled=2020-05-26T17:35:38+0000
t=2020-06-10T13:45:31+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2020-06-10T13:45:31+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
t=2020-06-10T13:45:31+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"
t=2020-06-10T13:45:31+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.logs=/var/log/grafana"
t=2020-06-10T13:45:31+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.plugins=/var/lib/grafana/plugins"
t=2020-06-10T13:45:31+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.provisioning=/etc/grafana/provisioning"
t=2020-06-10T13:45:31+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.log.mode=console"
t=2020-06-10T13:45:31+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana"
t=2020-06-10T13:45:31+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana"
t=2020-06-10T13:45:31+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
t=2020-06-10T13:45:31+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
t=2020-06-10T13:45:31+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_SECURITY_ADMIN_USER=admin"
t=2020-06-10T13:45:31+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_SECURITY_ADMIN_PASSWORD=*********"
t=2020-06-10T13:45:31+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana
t=2020-06-10T13:45:31+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana
t=2020-06-10T13:45:31+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana
t=2020-06-10T13:45:31+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins
t=2020-06-10T13:45:31+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning
t=2020-06-10T13:45:31+0000 lvl=info msg="App mode production" logger=settings
t=2020-06-10T13:45:31+0000 lvl=info msg="Initializing SqlStore" logger=server
t=2020-06-10T13:45:31+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
t=2020-06-10T13:45:31+0000 lvl=info msg="Starting DB migration" logger=migrator
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing HTTPServer" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing BackendPluginManager" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing PluginManager" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Starting plugin search" logger=plugins
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing HooksService" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing OSSLicensingService" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing InternalMetricsService" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing RemoteCache" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing RenderingService" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing AlertEngine" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing QuotaService" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing ServerLockService" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing LoginService" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing SearchService" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing TracingService" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing UsageStatsService" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing CleanUpService" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing NotificationService" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server
t=2020-06-10T13:45:32+0000 lvl=info msg="Initializing Stream Manager"
t=2020-06-10T13:45:32+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=http subUrl= socket=
t=2020-06-10T13:45:32+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS
t=2020-06-10T13:45:32+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS
t=2020-06-10T13:46:27+0000 lvl=eror msg="Invalid username or password" logger=context userId=0 orgId=0 uname= error="Invalid Username or Password" remote_addr=10.20.0.10
t=2020-06-10T13:46:27+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=POST path=/login status=401 remote_addr=10.20.0.10 time_ms=16 size=42 referer=http://10.20.1.1/login
t=2020-06-10T13:46:58+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/ status=302 remote_addr=10.20.0.10 time_ms=0 size=29 referer=
t=2020-06-10T13:47:32+0000 lvl=eror msg="Invalid username or password" logger=context userId=0 orgId=0 uname= error="Invalid Username or Password" remote_addr=10.20.0.10
t=2020-06-10T13:47:32+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=POST path=/login status=401 remote_addr=10.20.0.10 time_ms=108 size=42 referer=http://10.20.1.1/login
t=2020-06-10T13:47:40+0000 lvl=eror msg="Invalid username or password" logger=context userId=0 orgId=0 uname= error="Invalid Username or Password" remote_addr=10.20.0.10
t=2020-06-10T13:47:40+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=POST path=/login status=401 remote_addr=10.20.0.10 time_ms=12 size=42 referer=http://10.20.1.1/login

And this is the screenshot:

image
image3820×2022 400 KB

Can you point me in the right direction? I cannot see any problem? I have a nfs provisioner installed this is also running under user id 5000 and group id 5000. I don’t know, I think it’s an permissions problem but I don’t know how to solve it.

2

I would start by checking whether the problem is because you are connecting
through a load balancer (what type is it, by the way?), or whether Grafana is
doing this when you simply connect to a single instance directly from the
browser to Grafana.

There’s no point in spending time debugging the wrong problem, after all.

Antony.

3

Thank you very much. It was my pv that exists after the complete delete of the statefulset. After I removed the files manual, Grafana installed complete new.

Sorry, it was a silly mistake!

1 Like
4

grafana will store the password in the database on the first execution only so if you are reusing the persistent volume, the password won’t change even if you change the secret (GF_SECURITY_ADMIN_PASSWORD enviroment variable).

In order to reset the password you need to use grafana-cli on the container running grafana:

kubectl exec -ti -n grafana <pod> /bin/bash # get a shell to the pod / container running grafana
bash-5.0$ grafana-cli admin reset-admin-password $GF_SECURITY_ADMIN_PASSWORD
INFO[09-02|20:25:36] Connecting to DB                         logger=sqlstore dbtype=sqlite3
INFO[09-02|20:25:36] Starting DB migration                    logger=migrator
2 Likes
5

Worked for me.

Thanks!

1 Like