Timeout awaiting response header error while querying

Grafana Configuration
1

Hello dear community,

We recently upgraded our Grafana to 8.2.5 from 7.1.2. We are facing some issues with our NA data source.

Here are some of the error logs:

t=2021-12-12T16:40:29-0500 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=elastic path=/api/datasources/proxy/1/_msearch remote_addr=10.101.56.246 referer="http://grafana.staged-by-discourse.com/d/x0KEGiKGz/elk-metrics?orgId=1" error="http: proxy error: net/http: timeout awaiting response headers"
t=2021-12-12T16:40:29-0500 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=elastic method=POST path=/api/datasources/proxy/1/_msearch status=502 remote_addr=10.101.56.246 time_ms=30001 size=0 referer="http://grafana.staged-by-discourse.com/d/x0KEGiKGz/elk-metrics?orgId=1"
t=2021-12-12T16:40:29-0500 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=elastic path=/api/datasources/proxy/1/_msearch remote_addr=10.101.56.246 referer="http://grafana.staged-by-discourse.com/d/x0KEGiKGz/elk-metrics?orgId=1" error="http: proxy error: net/http: timeout awaiting response headers"

When we try to query some data from elasticsearch, Grafana gets the “Bad Gateway” error and doesn’t show any data from that specific data source. When we get the “Bad Gateway” error, the Grafana logs are the logs above. Other data sources are fine. The problem is our elk cluster is also fine. It is green with all nodes are active & running.

When I set the timeout field on the Grafana data source settings to 300 seconds, It doesn’t show any errors but it took a long time to query the data. We don’t see any errors or logs but like I said it took more than a minute to query.

  • How can we speed up this process?
  • What can we do about it?
  • Where to look for this issue?
2

Hello dear community,
We get the same error when making requests to Clickhouse. Our version of grafana is 8.3.2-1.
Previously, our queries worked, but at some point everything broke. Who has any thoughts on this?

3

I had similar issue with grafana. For me the issue was the timeout setting in dataproxy.

The default timeout is set to 30 seconds. Try increasing that.

4

Thank you for your reply. Yes, this was temporarily fixed the issue. For our case, the main issue was the heap problem with our ELK stack we fixed that, and this error is fixed permanently.

5

Hello @mertkaant ,

We are experiencing same issue. How did you fix the heap problem of ES? Just increasing?

6

can you please tell me the file name where you made this change?

7

/etc/grafana/grafana.ini

[dataproxy] section

8

Hello @cagripinar

Increasing the timeout setting on Grafana > Data sources temporarily fixes the timeout problems with Grafana and it is not a fix actually. Our main problem was high heap usage in Elasticsearch. We fixed that and our problem was solved.

1 Like