We are using grafana hosted in an azure-kubernetes-cluster (aks) to access seveal system metrics collected with telegraf an stored in influxdb.
Everything was performing very well - grafana renders all our dashboards within miliseconds and all our dev-ops-people are very happy with it.
Since the number of alerts and dashboards increased we decided to migrate from local sqlite-storage to a “real database”. Since our kubernetes-cluster is already connected to an managed azure postgesql-service we decided to host the grafana database there.
The migration itself was a bit tricky but finally suceeded without any information-loss.
But now the performance of rendering our dashboards degraded massively!
A simple dashboard that used to show up in less than a second now takes more that 10 second to load like this:
If I take a look at my ressource utilization of the postgresql service in azure portal it looks like this:
A look into the logs of my postgres-service shows lots and lots of entries like this:
2019-07-01 12:49:49 UTC-5d1a016d.c65b8-LOG: connection received: host=aaa.bbb.ccc.ddd port=38154 pid=812472 2019-07-01 12:49:49 UTC-5d1a016d.c65b4-LOG: connection received: host=aaa.bbb.ccc.ddd port=60867 pid=812468 2019-07-01 12:49:49 UTC-5d1a016d.c65bc-LOG: connection received: host=aaa.bbb.ccc.ddd port=24980 pid=812476 2019-07-01 12:49:49 UTC-5d1a016d.c65b0-LOG: connection received: host=aaa.bbb.ccc.ddd port=28318 pid=812464 2019-07-01 12:49:49 UTC-5d1a016d.c65ac-LOG: connection authorized: user=XXXXXX=grafana SSL enabled (protocol=TLSv1.1, cipher=ECDHE-RSA-AES256-SHA, compression=off) 2019-07-01 12:49:50 UTC-5d1a016d.c65bc-LOG: connection authorized: user=XXXXXX=grafana SSL enabled (protocol=TLSv1.1, cipher=ECDHE-RSA-AES256-SHA, compression=off) 2019-07-01 12:49:50 UTC-5d1a016d.c65b4-LOG: connection authorized: user=XXXXXX=grafana SSL enabled (protocol=TLSv1.1, cipher=ECDHE-RSA-AES256-SHA, compression=off) 2019-07-01 12:49:50 UTC-5d1a016d.c65b8-LOG: connection authorized: user=XXXXXX=grafana SSL enabled (protocol=TLSv1.1, cipher=ECDHE-RSA-AES256-SHA, compression=off) 2019-07-01 12:49:50 UTC-5d1a016d.c65a0-LOG: connection authorized: user=XXXXXX=grafana SSL enabled (protocol=TLSv1.1, cipher=ECDHE-RSA-AES256-SHA, compression=off) 2019-07-01 12:49:50 UTC-5d1a016e.c65c0-LOG: connection received: host=aaa.bbb.ccc.ddd port=57280 pid=812480
For me it looks like grafana is constantly establishing ssl-connections to the database which puts the database under unneccecary load. This might be a reason for the bad performance.
So do we have to move back to local sqlite-storage - or is there something we can do about it?
Grafana-Version 6.2.4 - helm-chart-Version: 3.5.5