Tips on preventing Grafana from bogging down InfluxDB server?

Grafana InfluxDB
1

Hello there,

I’m currently using Grafana to view InfluxDB data. Occasionally, we will have users submit a very large queries with small update intervals (i.e. query last 7 days, update browser every 5 seconds).

Essentially, the Grafana query doesn’t respond within 5 seconds before a new query is issued. These queries seem to cascade overtime and bog down the server’s NIC.

Are there any methods to prevent this behavior? Specifically I’m looking for a way to disable these queries from piling up.

2

I’m not sure there is a perfect solution to this. You could look into some of the following:

  • set a lower threshold on the dashboard auto-refresh interval (though obviously doesn’t stop users hitting “refresh” repeatedly)
  • set a higher threshold on the aggregation interval (though that doesn’t stop users requesting a huge date range)
  • Influx itself has quite a few configuration options for limiting resource-intensive queries - see Configure InfluxDB OSS | InfluxDB OSS 1.8 Documentation, or for limiting network traffic - see e.g. Configure InfluxDB OSS | InfluxDB OSS 1.8 Documentation. Those may be a bit heavy-handed, but potentially relevant depending on your use case
3

Thanks svetb, I’ve looked into some of those options and have run into the fact that might be too restrictive when setting up correct dashboards. For example, if we’re querying data only from the last 5 minutes, a 5 second update interval seems valid.

Do you know if Grafana will cancel the previous InfluxDB query prior to sending a new one? Or am I correct in assuming the queries will just keep building?

4

Good question…I’m not really sure, but I don’t think so.

Once Grafana has already fired off a query to Influx, I don’t think there’s a “stop query” mechanism or anything like that. If Grafana explicitly breaks its network connection with Influx, Influx may understand that to mean “cancel all outstanding queries”, but I’m a bit hazy on the potential details. And yeah, I’m not aware of there being a “break” like that which Grafana applies. That would indeed be nice.