Alertmanager datasource gives 404?

Grafana Alerting
, , ,
1

I added an Alertmanager datasource to Grafana per the instructions here. Configuration is simply:

apiVersion: 1
datasources:
  - name: Alertmanager
    type: alertmanager
    url: http://prometheus-operator-alertmanager:9093
    access: proxy

Grafana shows it added in the logs

logger=provisioning.datasources t=2022-08-02T19:04:32.74628128Z level=info msg="inserting datasource from configuration " name=Alertmanager uid=P7647F508D5F54FCB

However, when I go to /alerting/list, Alertmanager does not appear in the dropdown. When I go over to the Contact points tab (/alerting/notifications), it does appear in the dropdown, but it gives an error:

Error loading Alertmanager config
unexpected content type from upstream. expected YAML, got application/json

When I go to Silences tab (/alerting/silences), it appears in the dropdown but gives a different error:

Error loading silences
404 page not found
Error loading Alertmanager alerts
404 page not found

I don’t have any auth or TLS configured for alertmanager, and it is connecting through local network. What am I doing wrong with this datasource?

2

Hi @jonathanlynch ,

Thanks for providing the details. I will start with the basic debugging steps.

Just wanted to be sure, if you had already installed the alert manager from the official link:

Also, if you have added it inside your prometheus.yml file, it should have a config something like;

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

And, if that is also there then after the restart of the services (i.e. both Prometheus and alertmanager) you can verify it by the above API endpoint link e.g. for localhost will be:

http://localhost:9090/targets

There you should see a section of alertmanager.

I hope this guide helps and let us know so that the community can help you out if still having issues.

3

Hi usmanahmad, thank you for the reply. I am running in kubernetes, so the setup is not exactly as you describe, but similar. My prometheus.yml is using kubernetes discovery like

alerting:
  alertmanagers:
  - path_prefix: /
    scheme: http
    kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
        - prometheus-operator

When I go to http://prometheus-operator-prometheus:9090/targets, I see a section for alertmanager.

When I go to http://prometheus-operator-alertmanager:9093/#/status it says I am running version 0.23.0.

I am already able to use alerting rules to send alerts from Prometheus to Alertmanager. Where I am having trouble is trying to add Alertmanager as a datasource to Grafana.

4

I was able to get it working by telling grafana to use “prometheus” implementation:

apiVersion: 1
datasources:
  - name: Alertmanager
    type: alertmanager
    url: http://prometheus-operator-alertmanager:9093
    access: proxy
    jsonData:
      implementation: prometheus
1 Like