I have a Docker Container with Grafana Running. I’m using the grafana_mail plugin to generate mails with images of reports (see https://github.com/Isma399/grafana_mail ). That all worked fine until last week. Then I did an upgrade from Grafana 6 to Grafana 7. Today I got an email report with a hint that for rendering images, I need to install the “Grafana Image Renderer Plugin”. I did that and also installed the dependencies listed here:
https://grafana.com/docs/grafana/latest/administration/image_rendering/
After I got some errors about missing libraries, I installed the alsa-lib package as well (The Container is based on CentOS 7) and then it seemed to work. However, I ran into timeout errors. This always happens:
First I get a few lines with render requests in the log:
t=2020-06-08T06:23:44+0000 lvl=info msg=Rendering logger=rendering renderer=plugin path="d-solo/h96K9uxWz/e-mail-reports?panelId=7&from=1588809600000&to=1591401599000&width=1000&height=500&var-Location=1&orgId=1"
t=2020-06-08T06:23:44+0000 lvl=info msg=Rendering logger=rendering renderer=plugin path="d-solo/h96K9uxWz/e-mail-reports?panelId=2&from=1588809600000&to=1591401599000&width=1000&height=100&var-Location=1&orgId=1"
t=2020-06-08T06:23:44+0000 lvl=dbug msg="Calling renderer plugin" logger=rendering renderer=plugin req="url:\"http://grafana.staged-by-discourse.com/d-solo/h96K9uxWz/e-mail-reports?panelId=7&from=1588809600000&to=1591401599000&width=1000&height=500&var-Location=1&orgId=1&render=1\" width:1000 height:500 deviceScaleFactor:1 filePath:\"/var/lib/grafana/png/34wD4XSuxSRt4yB9RjHE.png\" renderKey:\"4XEgKjKgo6j1nD0Vi5Ovwx5Y3RG08BKS\" domain:\"localhost\" timeout:60 "
t=2020-06-08T06:23:44+0000 lvl=info msg=Rendering logger=rendering renderer=plugin path="d-solo/h96K9uxWz/e-mail-reports?panelId=5&from=1588809600000&to=1591401599000&width=1000&height=100&var-Location=1&orgId=1"
t=2020-06-08T06:23:44+0000 lvl=dbug msg="Calling renderer plugin" logger=rendering renderer=plugin req="url:\"http://grafana.staged-by-discourse.com/d-solo/h96K9uxWz/e-mail-reports?panelId=2&from=1588809600000&to=1591401599000&width=1000&height=100&var-Location=1&orgId=1&render=1\" width:1000 height:100 deviceScaleFactor:1 filePath:\"/var/lib/grafana/png/0ZeqlVK9Vf3bJ7EGRMA0.png\" renderKey:\"wDmVI2pQx6XwVhVcAlgsX13BjN0Q7ubE\" domain:\"localhost\" timeout:60 "
t=2020-06-08T06:23:44+0000 lvl=info msg=Rendering logger=rendering renderer=plugin path="d-solo/h96K9uxWz/e-mail-reports?panelId=8&from=1588809600000&to=1591401599000&width=1000&height=500&var-Location=1&orgId=1"
Then, for about 30 seconds (probably because the timeout shows as 30s) nothing happens. Then, I get a bunch of timeout errors:
t=2020-06-08T06:24:16+0000 lvl=eror msg="Render request failed" logger=plugins.backend pluginId=grafana-image-renderer url="http://grafana.staged-by-discourse.com/d-solo/h96K9uxWz/e-mail-reports?panelId=7&from=1588809600000&to=1591401599000&width=1000&height=500&var-Location=1&orgId=1&render=1" error="TimeoutError: Navigation timeout of 30000 ms exceeded"
t=2020-06-08T06:24:16+0000 lvl=eror msg="Rendering failed." logger=context userId=0 orgId=1 uname= error="Rendering failed: TimeoutError: Navigation timeout of 30000 ms exceeded"
t=2020-06-08T06:24:16+0000 lvl=eror msg="Request Completed" logger=context userId=0 orgId=1 uname= method=GET path=/render/d-solo/h96K9uxWz/e-mail-reports status=500 remote_addr=172.18.0.1 time_ms=32407 size=1723 referer=
t=2020-06-08T06:24:16+0000 lvl=eror msg="Render request failed" logger=plugins.backend pluginId=grafana-image-renderer url="http://grafana.staged-by-discourse.com/d-solo/h96K9uxWz/e-mail-reports?panelId=2&from=1588809600000&to=1591401599000&width=1000&height=100&var-Location=1&orgId=1&render=1" error="TimeoutError: Navigation timeout of 30000 ms exceeded"
So I tried to call the URL manually with CURL and get this result:
[root@4753564318ec /]# curl 'http://grafana.staged-by-discourse.com/d-solo/h96K9uxWz/e-mail-reports?panelId=7&from=1588809600000&to=1591401599000&width=1000&height=500&var-Location=1&orgId=1&render=1'
<a href="/login">Found</a>.
This seems to make sense since in the Generator Script I provide the API Key for the access, so without provideing that in curl it makes sense I get the login page only while it should work from the script.
If I replace the localhost:3000 part in the URL with the public URL of the Grafana Server I get a rendered image of the graph but I don’t see anything in the Grafana Log.
The grafana_mail script is running on the host machine and calling the Grafana Installation by it’s public URL (https://xxx.yz) so since the requests are coming to the Grafana Logs, I assume the issue is on the Grafana Side and not on the Mail Generation Script side.
Does anyone have an idea what the issue might be or how to troubleshoot that?