Loki queries within grafana dashboards

Grafana Loki
1

Hi all,

I have quite simple queries which return data for 90 days without problem when used in explorer mode. But the same queries (some of them) ends with:

#### <html> <head><title>504 Gateway Time-out</title></head> <body> <center><h1>504 Gateway Time-out</h1></center> <hr><center>nginx</center> </body> </html>

when used on grafana dashboard.
Further inspection of failed query stats shows that it ends usually after 1 minute.

I have re-set various timeouts:

  • grafana (loki datasource timeout 120s)
    [dataproxy]
    timeout = 300
  • nginx (loki-distributed gateway)
    proxy_read_timeout 300s;
  • loki
    server:
    http_server_read_timeout: 300s
    http_server_write_timeout: 300s
    querier:
    query_timeout: 3m

but this happening still.

Did I forgot to something?

Best regards,
Vladimir

2

And answering to myself - yes I totally forgot that access to my grafana is managed by ingress so:

nginx.ingress.kubernetes.io/proxy-connect-timeout: “360”
nginx.ingress.kubernetes.io/proxy-read-timeout: “360”
nginx.ingress.kubernetes.io/proxy-send-timeout: “360”

solved the issue.

3

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.