Prometheus DS "metric request error" new error on upgrading ubuntu

Grafana Prometheus
1

I’ve had an instance of Graphana running for 2 years on an Ubuntu box using Prometheus as a data source.

After updating ubuntu via apt I noticed my all my dashboards went down.
Going to the Prometheus DataSource settings I see the error
Error reading Prometheus: Metric request error

Looking in my apt upgrade log it looks like things broke with the update batch

Upgrade: libglapi-mesa:amd64 (21.0.3-0ubuntu0.3~20.04.4, 21.0.3-0ubuntu0.3~20.04.5),
libgl1-mesa-dri:amd64 (21.0.3-0ubuntu0.3~20.04.4, 21.0.3-0ubuntu0.3~20.04.5), 
grafana:amd64 (8.2.5, 8.3.0), 
mesa-vulkan-drivers:amd64 (21.0.3-0ubuntu0.3~20.04.4, 21.0.3-0ubuntu0.3~20.04.5), 
libglx-mesa0:amd64 (21.0.3-0ubuntu0.3~20.04.4, 21.0.3-0ubuntu0.3~20.04.5)

If I use curl to access Prometheus from the same machine I get results for endpoints like /metrics or api/v1/query. So it appears that Prometheus is working fine.

Also, googling the error message Error reading Prometheus: Metric request error returns zero results. So, not sure what to do here … roll back the grafana version or contact the Ubuntu apt maintainers?

Post this in the ubuntu stack exchange as well (in case there’s an issue with their grafana package)

2

Downgrading from grafana/stable 8.3.0 to grafana/stable 8.2.5 resolves the issue (until I need to upgrade grafana).

This is using the archive https://packages.grafana.com/oss/deb

3

Tried the ubuntu grafana update grafana/stable 8.3.2 amd64 and it still breaks the Prometheus DataSource.

4

Hello, I had the same issue upgrading grafana from 8.0.4 to 8.3.1:
Error reading Prometheus: Metric request

On some dashboard I had a ‘min step’ defined, removing it (to let it default to auto) solve the issue on my side.

Edit: it seems we used a wrong syntax for “Min step”: 5min ==> 5m solved it

1 Like
5

Same here. Were using an interval variable and something didn’t like the value. Cleared it out and all worked well.