Error setting up alert against Prometheus data source

Grafana Prometheus
1

Hi,

I’m relatively new to Grafanan and I’m trying out the alerting functionality. I have a dashboard with two charts which pulls in metrics from Prometheus. I’m trying to setup an alert on the second chart which has a Prometheus metric query like
sum(rate(MapR{Puts!~“sqgtrhad05.*”}[2m])).

I’m trying to setup an alert for this metric:

WHEN avg () OF query (A, 2m, now) IS BELOW 1000

But when I test this query, I get an error that I’m unable to decipher.

result
firing true
state
conditionEvals = true
timeMs 11.907ms
error **tsdb.HandleRequest() error bad_response: bad response code 503**

I do not understand why I get a 503. The metric query is correct as the graph displays correctly and I couldn’t find any detailed error messages in the grafana logs.

I’m using Grafana 4.4.1.
Thanks for the help

-G

2

Are you using direct or proxy access for datasource? Can grafana server access it?

3

Direct, no proxies. I already have charts against the data source, the issue is only with setting up alerts. Is there a way to get a more detailed error message?

4

And can the url you specified be reached from grafana server?

5

I’m not sure what URL you are referring to. The Prometheus data source is reachable from the grafana server, if it wasn’t it wouldn’t be possible for me create the grafana charts.
Am I missing something?

6

Depends on how you access prometheus, if you use direct or proxy access. Alerting queries are issued by the backend, graph panels (if you setup direct access) are issued to Prometheus directly.

7

Can you shed some light on ‘the backend’? I have a dashboard populating through a connection to Prometheus and I have a notification channel successfully firing to Alertmanager; however, when building an alert threshold in the dashboard, it fails due to a certificate error.

What source is this backend and how can I force Grafana to reach out to Alertmanager through the URL that is already working within the system?