Grafana and prometheus issue

Grafana Prometheus
1

Hello Team,

All of a sudden alerts started coming with below error note.

tsdb.HandleRequest() error Post "http://localhost:9090/api/v1/query_range": dial tcp [127.0.0.1:9090](http://127.0.0.1:9090/): connect: connection refused

We checked that prometheus is running properly on 9090 port and is able to fetch the metrics from other servers as well. But this alert is coming intermittently and sometimes clears also.

We also checked grafana logs and found this snippet but doesn’t seem relevant to this particular error

t=2020-11-23T08:48:30+0800 lvl=info msg=Rendering logger=rendering renderer=plugin path="d-solo/dBA3kjnZo/nykaa-eretailweb-tg-tg?orgId=1&panelId=15"
t=2020-11-23T08:48:30+0800 lvl=eror msg="Render request failed" logger=plugins.backend pluginId=grafana-image-renderer url="http://grafana.staged-by-discourse.com/d-solo/dBA3kjnZo/nykaa-eretailweb-tg-tg?orgId=1&panelId=15&render=1" error="Error: Failed to launch chrome!\n/var/lib/grafana/plugins/grafana-image-renderer/chrome-linux/chrome: error while loading shared libraries: libXcomposite.so.1: cannot open shared object file: No such file or directory\n\n\nTROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md\n"
t=2020-11-23T08:48:30+0800 lvl=eror msg="Failed to render and upload alert panel image." logger=alerting.notifier ruleId=129 error="Rendering failed: Error: Failed to launch chrome!\n/var/lib/grafana/plugins/grafana-image-renderer/chrome-linux/chrome: error while loading shared libraries: libXcomposite.so.1: cannot open shared object file: No such file or directory\n\n\nTROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md\n"

PFA image for the alert

Capture
Capture646×520 15.1 KB

Environment:

  • Grafana version: v7.1.5 (9893b8c53d)
  • Data source type & version: prometheus2-2.20.1-2.el7
  • OS Grafana is installed on: cpe:2.3:o:amazon:amazon_linux:2
  • User OS & Browser: Windows 10 & Chrome
  • Grafana plugins: grafana-image-renderer @ 2.0.0, grafana-piechart-panel @ 1.6.0, snuids-trafficlights-panel @ 1.4.5

Image render plugin is already installed on the server. Please help to correct this.

Regards
Nitin G

2

I even tried to update the both the versions to below but no fate. Please help.

Grafana to v7.3.3
Prometheus2 to v2.22.2-1.el7

3

Never Mind … It is resolved by replacing Data Source Prometheus URL in Grafana UI from http://localhost:9090 to http://:9090

4

I made an account just to reply to this, I’ve been having the same issue and this has resolved it and now I have data source is working, you absolute legend!