We have logs getting through promtail to loki There are no labels. We would like to compute the standard deviation to be used for alerting. The queries queries are as follows:

avg(stddev_over_time({job=“job1”}[1m]) != 0)

The above did not work because we had to unwrap labels and we did not have labels.
We tried the following and we got a line which is zero:

stddev(rate({job=“job1”} |= `` [$__interval]))

Note that avg(rate({job=“job1”} |= `` [$__interval])) is returning non-zero values.

We are getting all the lines with errors. Then we are trying to get all the time where the count is greater than 2 standard deviation and send an alert. We will also try z-score as threshold for alert but haven’t reached that yet.

I think you may be misreading on what the function stddev and stddev_over_time do, and how they are different.

stddev_over_time(unwrapped-range): the population standard deviation of the values in the specified interval.

stddev: Calculate the population standard deviation over labels

So when you do something like stddev(rate({$label_name=~“$label_value”} |= `` [$__interval])), you are calculating the stddev over the labels produced by {$label_name=~“$label_value”} |= `` , and if there is only one set of label it would of course be 0.

Judging from your original post, you are looking for a sudden spike of error logs (please correct me if I am wrong). Naturally you’d probably want to do stddev_over_time( count_over_time ( {Select for error logs} [range]) [range]), but I don’t think you can chain two unwrapped-range functions together in LogQL, hence I recommended trying out rate so you can calculate rate of change for the number of error logs, and see if that would be sufficient.

Ok, thanks. The issue we are having with rate only we are getting too many false positives. Rhe errors are frequent so we need a way to filter those with low errors and those with large errors out. We don’t know the threshold exactly. So, we thought about z-score and standard deviation to filter out those errors outside those “bands”.