Hello,
I’m setting up a new Grafana instance. First one with version 9 (9.1.1.) for me and first one to use RDS PostgreSQL as the database.
I am using the official Helm chart but my Kubernetes manifests are being applied by ArgoCD, which renders the Helm chart and does a kubectl apply in the background.
I have used Helm with Grafana in the past and after a helm upgrade --install grafana ... you see the admin password on your console and I have never had a problem logging in. However I have also until now always just used the internal sqlite3 database.
I have configured Google OAuth and that works fine. I can connect to PSQL and I see that new users are added to the user_oauth (I believe) table
Grafana startup logs
$ k logs grafana-86b5b9cb57-jd4vw -n grafana-rds
logger=settings t=2022-09-09T12:40:59.493396743Z level=info msg="Starting Grafana" version=9.1.1 commit=9e60a1bff1 branch=HEAD compiled=2022-08-23T09:04:10Z
logger=settings t=2022-09-09T12:40:59.493676744Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2022-09-09T12:40:59.493704937Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2022-09-09T12:40:59.493730642Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana/"
logger=settings t=2022-09-09T12:40:59.493766438Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
logger=settings t=2022-09-09T12:40:59.493782517Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
logger=settings t=2022-09-09T12:40:59.493799359Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2022-09-09T12:40:59.493824452Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
logger=settings t=2022-09-09T12:40:59.493850167Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana/"
logger=settings t=2022-09-09T12:40:59.493885382Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
logger=settings t=2022-09-09T12:40:59.493904438Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
logger=settings t=2022-09-09T12:40:59.493930919Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
logger=settings t=2022-09-09T12:40:59.493959655Z level=info msg="Config overridden from Environment variable" var="GF_DATABASE_PASSWORD=*********"
logger=settings t=2022-09-09T12:40:59.493990729Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_USER=admin"
logger=settings t=2022-09-09T12:40:59.494009292Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_PASSWORD=*********"
logger=settings t=2022-09-09T12:40:59.494025652Z level=info msg="Config overridden from Environment variable" var="GF_AUTH_GOOGLE_CLIENT_SECRET=*********"
logger=settings t=2022-09-09T12:40:59.494040745Z level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2022-09-09T12:40:59.49406696Z level=info msg="Path Data" path=/var/lib/grafana/
logger=settings t=2022-09-09T12:40:59.494089955Z level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2022-09-09T12:40:59.494107683Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2022-09-09T12:40:59.494122997Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2022-09-09T12:40:59.494138073Z level=info msg="App mode production"
logger=sqlstore t=2022-09-09T12:40:59.494234742Z level=info msg="Connecting to DB" dbtype=postgres
logger=migrator t=2022-09-09T12:40:59.585548579Z level=info msg="Starting DB migrations"
logger=migrator t=2022-09-09T12:40:59.616587855Z level=info msg="migrations completed" performed=0 skipped=442 duration=1.811243ms
logger=plugin.manager t=2022-09-09T12:40:59.719868691Z level=info msg="Plugin registered" pluginId=input
logger=secrets t=2022-09-09T12:40:59.720681679Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
logger=query_data t=2022-09-09T12:40:59.724851028Z level=info msg="Query Service initialization"
logger=live.push_http t=2022-09-09T12:40:59.735764755Z level=info msg="Live Push Gateway initialization"
logger=ticker t=2022-09-09T12:40:59.840674342Z level=info msg=starting first_tick=2022-09-09T12:41:00Z
logger=infra.usagestats.collector t=2022-09-09T12:40:59.926466067Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
logger=provisioning.alerting t=2022-09-09T12:40:59.959089195Z level=info msg="starting to provision alerting"
logger=provisioning.alerting t=2022-09-09T12:40:59.959328549Z level=info msg="finished to provision alerting"
logger=http.server t=2022-09-09T12:40:59.962646103Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket=
logger=ngalert t=2022-09-09T12:40:59.96329596Z level=info msg="warming cache for startup"
logger=grafanaStorageLogger t=2022-09-09T12:40:59.963603073Z level=info msg="storage starting"
logger=ngalert.multiorg.alertmanager t=2022-09-09T12:40:59.989439712Z level=info msg="starting MultiOrg Alertmanager"
But no matter what I try, I can’t login as the local admin user. I can see the password from e.g. the pod environment. Also, the same password is in the expected Kubernetes secret.
When I try to login using that password I get this in my logs
logger=context traceID=00000000000000000000000000000000 userId=0 orgId=0 uname= t=2022-09-09T12:56:41.918948203Z level=error msg="Invalid username or password" error="invalid username or password" remote_addr=79.225.xxx.xxx traceID=00000000000000000000000000000000
I have tried using grafana-cli to reset the password as well.
$ k exec grafana-5dbfbd74c4-xm667 -n grafana-rds -- grafana-cli admin reset-admin-password mynewpassword1
Defaulted container "grafana" out of: grafana, init-chown-data (init)
logger=settings t=2022-09-09T12:27:28.022682711Z level=info msg="Starting Grafana" version= commit= branch= compiled=1970-01-01T00:00:00Z
logger=settings t=2022-09-09T12:27:28.022883112Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2022-09-09T12:27:28.022899683Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana/"
logger=settings t=2022-09-09T12:27:28.022909151Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
logger=settings t=2022-09-09T12:27:28.022917676Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
logger=settings t=2022-09-09T12:27:28.022926951Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
logger=settings t=2022-09-09T12:27:28.022935608Z level=info msg="Config overridden from Environment variable" var="GF_DATABASE_PASSWORD=*********"
logger=settings t=2022-09-09T12:27:28.022944012Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_USER=admin"
logger=settings t=2022-09-09T12:27:28.02302491Z level=info msg="Config overridden from Environment variable" var="GF_SECURITY_ADMIN_PASSWORD=*********"
logger=settings t=2022-09-09T12:27:28.023036736Z level=info msg="Config overridden from Environment variable" var="GF_AUTH_GOOGLE_CLIENT_SECRET=*********"
logger=settings t=2022-09-09T12:27:28.023045768Z level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2022-09-09T12:27:28.023054478Z level=info msg="Path Data" path=/var/lib/grafana/
logger=settings t=2022-09-09T12:27:28.023062464Z level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2022-09-09T12:27:28.023070493Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2022-09-09T12:27:28.023078883Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2022-09-09T12:27:28.02308763Z level=info msg="App mode production"
logger=sqlstore t=2022-09-09T12:27:28.023284027Z level=info msg="Connecting to DB" dbtype=sqlite3
logger=sqlstore t=2022-09-09T12:27:28.023315544Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-rw---- expected=-rw-r-----
logger=migrator t=2022-09-09T12:27:28.057777161Z level=info msg="Starting DB migrations"
logger=migrator t=2022-09-09T12:27:28.066281875Z level=info msg="migrations completed" performed=0 skipped=442 duration=1.793587ms
Admin password changed successfully ✔
Same result. I’m a bit curious about seeing this ""Connecting to DB" dbtype=sqlite3. Is the admin user always configured in the local sqlite3 DB?
Any suggestions on how to get around this?
Cheers,
AB