Grafana alert manager works but gives error in UI related to loki query?

Hi,

I did manage to get the alert. However, the alert UI doesn’t show the log query output and instead this error message is shown.
Please suggest a way to solve this or a way to debug also would be very appreciated.
Screen Shot 2022-09-01 at 16.52.18

Sincerely

I still have this error. Also alerts won’t dismiss, they stay in “alerting” state forever. Maybe these two issues are related?

This is the error message returned:

{
    "message": "Failed to evaluate queries and expressions: failed to execute conditions: input data must be a wide series but got type long (input refid)",
    "traceID": "00000000000000000000000000000000"
}

And this is the request sent by grafana alerting UI:

Please suggest how to handle?

That popup is often occurs when the alerting platform evaluates data that is not numeric. I’d double check that you are performing evaluations only on numeric data types

it wasn’t obvious to me at first, but when constructing alerts it seems you need to think in terms of three steps:
A. query
B. reduce
C. alert condition

the query will typically be a query you’re using for a dashboard panel.
reduce will take the result of they query, which is typically a time series, and reduce it to a single value – for example sum() or last().
then condition will evaluate the result of that reduction to produce a boolean condition.

here’s an example (albeit using loki data source):

Thanks @nparrish42 @mattabrams ! I made it an aggregation count query and it doesn’t error anymore.
Will check if alerts will work well, now that this issue is understood.

1 Like

HI all, I have this error now for a couple of days and I stil have no clue why it is giving this error. I have the following dashboard where the log files are displayed at the top and a timeseries of that log file where the word “Scheduler” is looked for. I need to build an alert on this time series model, which will give an alert when the amount the word “Scheduler” is below 6 in one hour. Underneath a picture:

At this picture you could see that the data import is working of the job. When i click on the time series picture and try to make an alert. I end up in the alert page. underneath a picture:

When I press the run queries button, will it give the same error above:

Failed to evaluate queries and expressions: failed to execute conditions: failed to query data: Run out of attempts while querying the server

How could i resolve this weird error? Thing i have done:

  • Changed the ruler config file where “http:localhost:9093” to “http:alertmanager:9093” .
  • Changed the job file on promtail.yml on the server.
  • Changed the evaluate function to different time frames.

Hopefully i could work it out in the future, Help is very much appreciated!

Kind regards,

Elias

This happend when i press run queries