How to count events(errors) on selected period? (Loki/Prometheus)

Grafana Loki
1

Hello!
(Grafana newbie here.)

Really frustrated about simple counting of particular events during selected time.
I’ve got a simple dashboard, where I have log lines from LOKI (on the screenshot - filtered by “ERROR” keyword), some graphs, etc.

01
011381×723 65.7 KB

I want to count (just count, not average, etc.) my app’s errors during given time interval.
5m, 30m or user selected (with mouse on graph).

I’ve created a variable ($var_selected_interval, type=interval, Auto=ON) and used it in a panel:

errpanel
errpanel1046×645 51.8 KB

So… as we can see on the 1st screenshot Errors count in the logs really differs from the “calculated”.
What this " 1.9 " errors mean?
1.9 (even 19) in selected by user 5 minutes period? Wrong!
1.9 per second?.. Doubtfully
So, I need only a count. Count all error lines in my log.

I have tried count_over_time, sum, rate, etc… Nothing helps.

What I am doing wrong?
Please, help.

2

Hmmm… looks like I should use " $__range" variable, right?

3

do you have TOTAL selected in the stats value setting?

4

Not sure, where to find this “TOTAL” option…

5

HI. I got this to work in a similar use case. I wanted to tally up the total number of lines where a specific issue occurred. Here is what I did.

image
image1945×853 103 KB

One thing I ran into is that when the interval was very large the counts were not accurate, it was like it was dropping dome of the entries. This is why I set the max data points option, once I did that it was better unless I went to a really large interval.

Hope this helps!

2 Likes
6

Thank you, this worked!! Amazing when you dig up a bit of gold like this :tada:

Interestingly, if I go above 10000 it returns no data. Not 100% on why, but it works fine for now