Migration Grafana OSS from docker to Azure Kubernetes Service

Grafana Configuration
, , ,
1

Hi everyone!

I’m currently working on the migration from docker compose (running on a linux server) to aks. I’m using de 9.3.6 (oss) version. But when I want to log in into the new intance I get this error:

image

Tried to reset the admin password to try to access but I get “Error: ✗ failed to update user password: database is locked”.

The steps I followed were:

  • Stop the current instance (docker compose)
  • Copy the contents of the two directories (grafana-data and grafana-conf)
  • In our case, we are using an Storage Account to store this data. The copied directories were placed in a File Share.
  • Deploy the grafana pod in aks, using the file share with the data and conf as volumes.
  • Access the new instance and tried to log in.

In the original server we are using volumes to map the contents of the /etc/grafana and the /var/lib/grafana directories between the grafana container and the linux server. This were the ones I moved to the File Share.

This are the logs of the Grafana pod:

logger=settings t=2023-04-17T16:21:56.230150821Z level=info msg="Starting Grafana" version=9.3.6 commit=978237e7cb branch=HEAD compiled=2023-01-26T14:38:58Z
logger=settings t=2023-04-17T16:21:56.230386344Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2023-04-17T16:21:56.230412543Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2023-04-17T16:21:56.230425587Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
logger=settings t=2023-04-17T16:21:56.230439263Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
logger=settings t=2023-04-17T16:21:56.230451226Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
logger=settings t=2023-04-17T16:21:56.230463619Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2023-04-17T16:21:56.230477495Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
logger=settings t=2023-04-17T16:21:56.230491621Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
logger=settings t=2023-04-17T16:21:56.230503123Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
logger=settings t=2023-04-17T16:21:56.23052275Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
logger=settings t=2023-04-17T16:21:56.230539952Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
logger=settings t=2023-04-17T16:21:56.230555431Z level=info msg="Config overridden from Environment variable" var="GF_LOG_MODE=console file"
logger=settings t=2023-04-17T16:21:56.230567814Z level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2023-04-17T16:21:56.230579607Z level=info msg="Path Data" path=/var/lib/grafana
logger=settings t=2023-04-17T16:21:56.230595046Z level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2023-04-17T16:21:56.230607649Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2023-04-17T16:21:56.230620533Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2023-04-17T16:21:56.230633678Z level=info msg="App mode production"
logger=sqlstore t=2023-04-17T16:21:56.230714159Z level=info msg="Connecting to DB" dbtype=sqlite3
logger=sqlstore t=2023-04-17T16:21:56.238880783Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rwxrwxrwx expected=-rw-r-----
logger=migrator t=2023-04-17T16:21:56.350674745Z level=info msg="Starting DB migrations"
logger=migrator t=2023-04-17T16:21:56.354711619Z level=info msg="migrations completed" performed=0 skipped=464 duration=359.365µs
logger=plugin.loader t=2023-04-17T16:21:56.416395314Z level=info msg="Plugin registered" pluginID=input
logger=plugin.loader t=2023-04-17T16:21:56.613518553Z level=warn msg="Skipping loading plugin due to problem with signature" pluginID=grafana-groupedbarchart-panel status=unsigned
logger=secrets t=2023-04-17T16:21:56.628967467Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
logger=query_data t=2023-04-17T16:21:56.632641349Z level=info msg="Query Service initialization"
logger=live.push_http t=2023-04-17T16:21:56.634887652Z level=info msg="Live Push Gateway initialization"
logger=infra.usagestats.collector t=2023-04-17T16:21:57.033059894Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
logger=provisioning.alerting t=2023-04-17T16:21:57.137145584Z level=error msg="can't read alerting provisioning files from directory" path=/etc/grafana/provisioning/alerting error="open /etc/grafana/provisioning/alerting: no such file or directory"
logger=provisioning.alerting t=2023-04-17T16:21:57.137210175Z level=info msg="starting to provision alerting"
logger=provisioning.alerting t=2023-04-17T16:21:57.137252494Z level=info msg="finished to provision alerting"
logger=ngalert.state.manager t=2023-04-17T16:21:57.138713198Z level=info msg="Warming state cache for startup"
logger=http.server t=2023-04-17T16:21:57.148554341Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket=
logger=grafanaStorageLogger t=2023-04-17T16:21:57.14869237Z level=info msg="storage starting"
logger=ngalert.state.manager t=2023-04-17T16:21:57.168949369Z level=info msg="State cache has been initialized" states=0 duration=30.23553ms
logger=ngalert.multiorg.alertmanager t=2023-04-17T16:21:57.169070866Z level=info msg="starting MultiOrg Alertmanager"
logger=ticker t=2023-04-17T16:21:57.169088601Z level=info msg=starting first_tick=2023-04-17T16:22:00Z
logger=serviceaccounts t=2023-04-17T16:22:02.188221487Z level=warn msg="Failed to get usage metrics" error="database is locked"
logger=infra.usagestats t=2023-04-17T16:22:02.18841928Z level=error msg="Failed to get last sent time" error="database is locked"
logger=provisioning.dashboard t=2023-04-17T16:22:02.189219271Z level=warn msg="Failed to delete orphaned provisioned dashboards" err="database is locked"
logger=secret.migration t=2023-04-17T16:22:02.209594751Z level=error msg="Server lock for secret migration already exists"
2

Hi @fedsecch,

Welcome to the :grafana: community support forums !!

We are excited that you joined our OSS community. Please read about some of the FAQs in the community :slight_smile:

I do not know that well about AKS or Azure. However I checked your logs and see the lines saying:

logger=serviceaccounts t=2023-04-17T16:22:02.188221487Z level=warn msg="Failed to get usage metrics" error="database is locked"
logger=infra.usagestats t=2023-04-17T16:22:02.18841928Z level=error msg="Failed to get last sent time" error="database is locked"
logger=provisioning.dashboard t=2023-04-17T16:22:02.189219271Z level=warn msg="Failed to delete orphaned provisioned dashboards" err="database is locked"

For this error, please view the following GitHub issue as there is a workaround and also a proposed solution to switch from default SQLite backend database to either MySQL or PostgreSQL.

I hope this helps.

3

Hi @usman.ahmad ! Thank you for your reply.

I solved the problem setting an fsGroup in the manifest, which allows you to set the group ID that owns the volume, and thus who can write to it.

But now I have another problem… The SQLite3 DB wipes in the deployment, erasing all the data:

logger=sqlstore t=2023-05-10T13:34:40.484984559Z level=info msg="Connecting to DB" dbtype=sqlite3
logger=sqlstore t=2023-05-10T13:34:40.485008293Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-rw---- expected=-rw-r-----
logger=migrator t=2023-05-10T13:34:40.496804372Z level=info msg="Starting DB migrations"
logger=migrator t=2023-05-10T13:34:40.496910581Z level=info msg="Executing migration" id="create migration_log table"
logger=migrator t=2023-05-10T13:34:40.503043852Z level=info msg="Executing migration" id="create user table"
etc...

and then…

logger=context t=2023-05-10T13:35:28.775683649Z level=error msg="Anonymous access organization error." org_name="[My Organization Name]" error="[org.notFound] failed to get org by name: [My Organization Name]"
logger=context userId=0 orgId=0 uname= t=2023-05-10T13:35:28.776279639Z level=error msg="Invalid username or password" error="user not found" remote_addr=[IP Address] traceID=

Is there a way to prevent this?

4

Found this GitHub issue which is a close match to your error message. Try out the suggestion and see if that helps.

5

This is already specified in the grafana.ini file. The problem, I think, it comes from the docker image which won’t preserve the data in the SQLite DB.

6

Turns out I re-deployed the pod as a Statefulset and removed the wal: true from the grafana.ini file. It solved the whole problem! :smiley:

1 Like
7

Glad to hear I will cross reference this post link into this one which is all about Grafana migration.