Grafana Too Many Points in a Series Error

[altuscilliers]

Hello

My grafana dashboard just started giving me this error:
returned too many points in series: 902

Any idea what it could be? It is on all the single stat and gauge panels(The number differs between panels, the graphs seem fine. If I make a snapshot of my dashboard it now displays wrong info…was working a few hours ago? I can provide the flux code or telegraf configs if needed
Edit: The graphs have started to show this error as well

[altuscilliers]

bumping post, still haven’t figured it out

[grant2]

How many points in a given time range are stored in your datasource (e.g. 10 readings per minute, 4 readings per second, 3 readings per hour, etc.)? What time range are you using to view the Graphs, Stat Panel or Gauge Panel?

[altuscilliers]

I’m not sure, how would I check the number of readings in a time range? I am querying one API every 2 minutes and another every hour(would like to add more in the future). The time range I have them set to past 24 hrs, but in the flux code I have range(start: v.timeRangeStart, stop:v.timeRangeStop)

[altuscilliers]

Using influxdb as a datasource for context, not sure why but re-selecting past 24 hours and clicking apply seems to have fixed them? Not sure if it is only temporarily or not, but they are working again and displaying the proper data.

Edit: worked for about a minute then some of them gave the error again

[grant2]

To see how many readings you have in your Influx database, log into it and write a SELECT query statement for a given time range. More help here: Data exploration using InfluxQL | InfluxDB OSS 1.7 Documentation

[jangaraj]

You query selects all records in the selected period (e.g. 24h) without any aggragetion. In theory you may have a infinite records in that selected period, but you want to show only one value in the used panel → that’s not very efficient query, because it is wasting resources and you may hit some limits (it looks like you have hit MD limit of the panel - max datapoints per panel, which is 451 in your screenshot).

The better option is to write query, which will return exactly one datapoint for selected datapoint, so you never hit any MD limit - it can be the most recent (last), average, min, max, … value of the selected InfluxDB field - select function, which fits your needs. IMHO last() function seems to be the best option for your use case.

[altuscilliers]

okay, still very new to grafana and flux, Is this what you are referring to?

or should I change my flux query code and just add |> last() at the end?

[grant2]

Keep your current query the same and make the Display setting as shown in your screenshot (with “Calculation” being Last (not null)) and report back.

[altuscilliers]

I’m confused, is it not set as that already…? that screenshot is from my panel’s display setting

[grant2]

OK, then I think your query is where the problem is. How did you write the query initially (using a help guide or something)? Is it just 1 query or several? Can you paste the text for each query here? I can see from your previous screenshot that you have query A, but maybe there are more? If this is a public Influx database you are using as your data source, please advise and I can try to connect to it and help figure it out.

Just to give us some broader context, have you had success in creating other Grafana panels (gauge, stat panel, etc.) with this same data source?

[altuscilliers]

I think I used the flux documentation from influxdb to write the query. Query A is the only query for this panel, I have more for other panels though if you want the code for them. It is not a public database, I am using a python script to query some APIs and inputting into my own database on influxdb.

I have various panels, some stat ,gauge, graph etc…will give an example of a stat and graph query below:
Stat panel:

from(bucket: "altus")
  |> range(start: v.timeRangeStart, stop:v.timeRangeStop)
  |> filter(fn: (r) =>
    r._measurement == "my_metric-ethermine" and
    r._field == "usdPerMin" 
  )
  |> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
  |> map(fn: (r) => ({
    r with
    _value: (r.usdPerMin*1440.00)
  }
  )
  )
  |> last()

Graph:

  |> range(start: v.timeRangeStart, stop:v.timeRangeStop)

  |> filter(fn: (r) =>

    r._measurement == "my_metric-ethermine" and

    r._field == "currentHashrate" or r._field == "averageHashrate" or r._field == "reportedHashrate"

  )

I added the |> last() at the end of my single stat query as @jangaraj suggested which seems to have fixed them, at least for now…will keep an eye on them to see if they break again

[grant2]

That’s great. If his suggestion fixes the issue, please mark it as Solution.