I’m running into a weird problem. I have grafana and Prometheus running on the same box. Grafana is running behind nginx. I am trying to graph the following query:
topk(10, sum by (property_id) (increase(aasubmitter_seen_property_submissions_total{job="hoothoot",type="impression"}[1m])))
If I go to the Prometheus web console, I can graph that query in under 5 seconds. When Grafana makes that call, it takes more than 15 seconds and the Nginx proxy times out. Obviously I could just increase the Nginx timeout, but I don’t understand why it is taking so long in the first place. How can I troubleshoot this or speed up the query?
What is your step parameter option you defined in Grafana and what did you set it to when you issued it manually?
Grafana is using the query_range api that takes a step parameter that sets the number of points to query for. By default Grafana will set this to a low number to get one data point per pixel in the Graph, depending on time range this can be very slow in Prometheus.
Try manually setting a higher step option in Grafana, or change the Resolution option, this will cause Grafana to automatically set the step option to bigger jumps (but will get less data back)
