I have a docker stack with Grafana and it’s image renderer plugin configured as follows:
grafana:
image: grafana/grafana
restart: always
environment:
- GF_RENDERING_SERVER_URL=http://renderer:8081/render
- GF_RENDERING_CALLBACK_URL=http://grafana:3000/
- GF_LOG_FILTERS= rendering:debug
...
- GF_SERVER_PROTOCOL=https
- GF_LOG_MODE=file
ports:
- 6030:3000
networks:
- tpin_net
renderer:
image: grafana/grafana-image-renderer
restart: always
ports:
- 8081
networks:
- tpin_net
but when trying to render a panel I get an error “”. below logs from containers
Grafana
logger=rendering renderer=http t=2022-03-21T19:29:24.36+0000 lvl=info msg=Rendering path="d-solo/PUlb2g27z/healthcheck_dashboard?orgId=1&var-Country=SINGAPOR&var-Customer=SINGTEL&from=1647718128309&to=1647919728309&panelId=4&width=1000&height=500&tz=Asia%2FQatar"
logger=rendering renderer=http t=2022-03-21T19:29:24.37+0000 lvl=dbug msg="calling remote rendering service" url="http://renderer:8081/render?deviceScaleFactor=1.000000&domain=grafana&encoding=&height=500&renderKey=24RLoc72xoFtKjLA4tViQJ4RSuyHHG63&timeout=60&timezone=Asia%2FQatar&url=http%3A%2F%2Fgrafana%3A3000%2Fd-solo%2FPUlb2g27z%2Fhealthcheck_dashboard%3ForgId%3D1%26var-Country%3DSINGAPOR%26var-Customer%3DSINGTEL%26from%3D1647718128309%26to%3D1647919728309%26panelId%3D4%26width%3D1000%26height%3D500%26tz%3DAsia%252FQatar%26render%3D1&width=1000"
logger=rendering renderer=http t=2022-03-21T19:30:02.85+0000 lvl=eror msg="Failed to send request to remote rendering service" error="Get \"http://renderer:8081/render?deviceScaleFactor=1.000000&domain=grafana&encoding=&height=500&renderKey=24RLoc72xoFtKjLA4tViQJ4RSuyHHG63&timeout=60&timezone=Asia%2FQatar&url=http%3A%2F%2Fgrafana%3A3000%2Fd-solo%2FPUlb2g27z%2Fhealthcheck_dashboard%3ForgId%3D1%26var-Country%3DSINGAPOR%26var-Customer%3DSINGTEL%26from%3D1647718128309%26to%3D1647919728309%26panelId%3D4%26width%3D1000%26height%3D500%26tz%3DAsia%252FQatar%26render%3D1&width=1000\": context canceled"
logger=rendering t=2022-03-21T19:30:02.85+0000 lvl=eror msg="Failed to delete render key" error="context canceled"
logger=context t=2022-03-21T19:30:02.85+0000 lvl=eror msg="Rendering failed." error="failed to send request to remote rendering service: Get \"http://renderer:8081/render?deviceScaleFactor=1.000000&domain=grafana&encoding=&height=500&renderKey=24RLoc72xoFtKjLA4tViQJ4RSuyHHG63&timeout=60&timezone=Asia%2FQatar&url=http%3A%2F%2Fgrafana%3A3000%2Fd-solo%2FPUlb2g27z%2Fhealthcheck_dashboard%3ForgId%3D1%26var-Country%3DSINGAPOR%26var-Customer%3DSINGTEL%26from%3D1647718128309%26to%3D1647919728309%26panelId%3D4%26width%3D1000%26height%3D500%26tz%3DAsia%252FQatar%26render%3D1&width=1000\": context canceled"
logger=context t=2022-03-21T19:30:02.85+0000 lvl=eror msg="Request Completed" method=GET path=/render/d-solo/PUlb2g27z/healthcheck_dashboard status=500 remote_addr=10.0.0.2 time_ms=38509 size=1344 referer=
logger=rendering renderer=http t=2022-03-21T19:35:02.04+0000 lvl=info msg=Rendering path="d-solo/PUlb2g27z/healthcheck_dashboard?orgId=1&var-Country=SINGAPOR&var-Customer=SINGTEL&from=1647718128309&to=1647919728309&panelId=4&width=1000&height=500&tz=Asia%2FQatar"
logger=rendering renderer=http t=2022-03-21T19:35:02.05+0000 lvl=dbug msg="calling remote rendering service" url="http://renderer:8081/render?deviceScaleFactor=1.000000&domain=grafana&encoding=&height=500&renderKey=Zxu21wz36qSeCc1slQb6ssPA8Y4i2hTM&timeout=60&timezone=Asia%2FQatar&url=http%3A%2F%2Fgrafana%3A3000%2Fd-solo%2FPUlb2g27z%2Fhealthcheck_dashboard%3ForgId%3D1%26var-Country%3DSINGAPOR%26var-Customer%3DSINGTEL%26from%3D1647718128309%26to%3D1647919728309%26panelId%3D4%26width%3D1000%26height%3D500%26tz%3DAsia%252FQatar%26render%3D1&width=1000"
Image renderer
rpm_renderer.1.1huzg0b8pqxn@rpm | {"level":"error","message":"Browser console error","msg":"Failed to load resource: the server responded with a status of 400 (Bad Request)","url":"http://grafana:3000/d-solo/PUlb2g27z/healthcheck_dashboard?orgId=1&var-Country=SINGAPOR&var-Customer=SINGTEL&from=1647718128309&to=1647919728309&panelId=4&width=1000&height=500&tz=Asia%2FQatar&render=1"}
rpm_renderer.1.1huzg0b8pqxn@rpm | {"failure":"net::ERR_CONNECTION_RESET","level":"error","message":"Browser request failed","method":"GET","url":"http://grafana:3000/d-solo/PUlb2g27z/healthcheck_dashboard?orgId=1&var-Country=SINGAPOR&var-Customer=SINGTEL&from=1647718128309&to=1647919728309&panelId=4&width=1000&height=500&tz=Asia%2FQatar&render=1"}
rpm_renderer.1.1huzg0b8pqxn@rpm | {"level":"error","message":"Browser console error","msg":"Failed to load resource: net::ERR_CONNECTION_RESET","url":"http://grafana:3000/d-solo/PUlb2g27z/healthcheck_dashboard?orgId=1&var-Country=SINGAPOR&var-Customer=SINGTEL&from=1647718128309&to=1647919728309&panelId=4&width=1000&height=500&tz=Asia%2FQatar&render=1"}
rpm_renderer.1.1huzg0b8pqxn@rpm | {"err":"TimeoutError: Navigation timeout of 60000 ms exceeded\n at /usr/src/app/node_modules/puppeteer/lib/cjs/puppeteer/common/LifecycleWatcher.js:107:111","level":"error","message":"Error while trying to prepare page for screenshot","url":"http://grafana:3000/d-solo/PUlb2g27z/healthcheck_dashboard?orgId=1&var-Country=SINGAPOR&var-Customer=SINGTEL&from=1647718128309&to=1647919728309&panelId=4&width=1000&height=500&tz=Asia%2FQatar&render=1"}
rpm_renderer.1.1huzg0b8pqxn@rpm | {"err":"TimeoutError: waiting for function failed: timeout 60000ms exceeded\n at new WaitTask (/usr/src/app/node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:526:34)\n at DOMWorld.waitForFunction (/usr/src/app/node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:477:26)\n at Frame.waitForFunction (/usr/src/app/node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:993:32)\n at Page.waitForFunction (/usr/src/app/node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:2479:33)\n at /usr/src/app/build/browser/browser.js:309:33\n at Browser.<anonymous> (/usr/src/app/build/browser/browser.js:424:24)\n at Generator.next (<anonymous>)\n at /usr/src/app/build/browser/browser.js:8:71\n at new Promise (<anonymous>)\n at __awaiter (/usr/src/app/build/browser/browser.js:4:12)","level":"error","message":"Error while waiting for the panels to load","url":"http://grafana:3000/d-solo/PUlb2g27z/healthcheck_dashboard?orgId=1&var-Country=SINGAPOR&var-Customer=SINGTEL&from=1647718128309&to=1647919728309&panelId=4&width=1000&height=500&tz=Asia%2FQatar&render=1"}
rpm_renderer.1.1huzg0b8pqxn@rpm | {"level":"error","message":"Browser console error","msg":"Failed to load resource: the server responded with a status of 400 (Bad Request)","url":"http://grafana:3000/d-solo/PUlb2g27z/healthcheck_dashboard?orgId=1&var-Country=SINGAPOR&var-Customer=SINGTEL&from=1647718128309&to=1647919728309&panelId=4&width=1000&height=500&tz=Asia%2FQatar&render=1"}
rpm_renderer.1.1huzg0b8pqxn@rpm | {"failure":"net::ERR_CONNECTION_RESET","level":"error","message":"Browser request failed","method":"GET","url":"http://grafana:3000/d-solo/PUlb2g27z/healthcheck_dashboard?orgId=1&var-Country=SINGAPOR&var-Customer=SINGTEL&from=1647718128309&to=1647919728309&panelId=4&width=1000&height=500&tz=Asia%2FQatar&render=1"}
rpm_renderer.1.1huzg0b8pqxn@rpm | {"level":"error","message":"Browser console error","msg":"Failed to load resource: net::ERR_CONNECTION_RESET","url":"http://grafana:3000/d-solo/PUlb2g27z/healthcheck_dashboard?orgId=1&var-Country=SINGAPOR&var-Customer=SINGTEL&from=1647718128309&to=1647919728309&panelId=4&width=1000&height=500&tz=Asia%2FQatar&render=1"}
rpm_renderer.1.1huzg0b8pqxn@rpm | {"err":"TimeoutError: Navigation timeout of 60000 ms exceeded\n at /usr/src/app/node_modules/puppeteer/lib/cjs/puppeteer/common/LifecycleWatcher.js:107:111","level":"error","message":"Error while trying to prepare page for screenshot","url":"http://grafana:3000/d-solo/PUlb2g27z/healthcheck_dashboard?orgId=1&var-Country=SINGAPOR&var-Customer=SINGTEL&from=1647718128309&to=1647919728309&panelId=4&width=1000&height=500&tz=Asia%2FQatar&render=1"}
I looked around and found one answer saying that the SQL query is taking more than 60 seconds, but inside the dashboard the panel is loading within few seconds, and the logs about bad request shows instantly when I click on render image button