Log retrieval fails with "Bad Gateway"... Cannot poll logs + 1 hour | Help with Config

Grafana Loki
1

My setup is as follows:

  1. Servers ship logs to a centralized RSYSLOG box.
  2. Log Server ships logs to loki via promtail
  3. Loki Ingests loks and grafana is used to sort and filter them

Problem:
I have a LOT of logs and each time I try to search / sort / query them I get:
image
but I can see logs if I just use {job=~“.+”}|=“error” but only for a period < 1 hour. It seems loki is timing out during the requests.However it will retrieve results if a lesser time is selected.

Question:
Should I NOT aggregate my logs before shipping and create custom promtail configs for each of my servers? I have a requirement to ship all logs in every folder so using …//* or …//*.log is advisable?

2

Open up your browsers dev tools, and then the network tab. it should tell you what part of loki is giving you the bad gateway error.

3

I see there is a backwards limit based on the error url:
....api/datasources/proxy/5/loki/api/v1/query_range?direction=BACKWARD&limit=373&query=count_over_time(%7Bjob%3D~%22.%2B%22%7D%7C~%22error%22%5B1m%5D)&start=1648761356143000000&end=1649366156143000000&step=1800

4

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.