Failed to evaluate queries and expressions: input data must be a wide series but got type long (input refid)

[neykuratick]

Hello! I’m trying to setup notifications for loki source and the same query that works in “Explore” window doesn’t work in “Alert rules” window, I couldn’t find any info on this on the internet

So the query is {container="jetton-backend"} | json | status_code > 400

Here it is working in explore

And here is the same query that doesn’t work in alerts

Can anyone help please? Is there something i’m missing?

[victor07be]

I have the same problem with alerting on logs with Loki & Promtail
Working on explore but not in alert.
Query: {namespace=“monitoring”} |= “0”
Response: “Failed to evaluate queries and expressions: input data must be a wide series but got type long (input refid)”

Any help would be apreciated

[sam2b3e]

I was trying to figure this out too. From this Github issue I get the impression that Grafana alerting can only handle numeric data, not log messages.

So you can do a count of log messages matching a query, alert when the count is > 0, and then manually go looking through the logs when you get an alert, but as far as I can see there’s no way to actually get the log message contents.

Which seems nuts, so hopefully I’m missing something and someone else will chime in here with a better solution.

count(rate({container="jetton-backend"}} | json | status_code > 400 [$__interval]))

[georgerobinson]

@sam2b3e is correct. Grafana Alerting works on numeric data, and Loki alerts are expected to use count or count_over_time.

[sam2b3e]

Many thanks for confirming, it’s good to know I’m not missing the obvious

I come from a world that uses Graylog and Elasticsearch for logging, so I’m used to being able to get log message contents in alerts, and it’s one of the first things I set out to do when evaluating Loki. I’m genuinely rather surprised that it can’t be done.

Is there anything in the roadmap that might address this in the future? Some method of having the alert notification template reference the underlying log data? I imagine it’d be a huge enabler for other people in my situation looking to move their logging over Loki (which in every other way is extremely compelling).