Individual alerts for cloud alerting/alertmanager

Grafana Cloud
,
1

We are using hosted Grafana, synthetic monitoring and alertmanager to send out alerts to both email and slack. However, when receiving the alerts, they are grouped (one email/slack message for all alerts that are firing).

Is it also possible to configure alertmanager so that only individual alerts are sent? Similar to the graph alerts, so that when an alert is OK it is sent specifically for that event and you don’t have to scroll through and compare a list of alerts?

Alertmanager configuration:

global:
  resolve_timeout: 5m
  slack_api_url: https://hooks.slack.com/services/apikey
  http_config:
    follow_redirects: true
  smtp_from: noreply@grafana.net
  smtp_hello: localhost
  smtp_smarthost: smtprelay:2525
  smtp_require_tls: false
  pagerduty_url: https://events.pagerduty.com/v2/enqueue
  opsgenie_api_url: https://api.opsgenie.com/
  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
  victorops_api_url: https://alert.victorops.com/integrations/generic/123/alert/
route:
  receiver: email, 'slack-notifications'
  continue: false
receivers:
  - name: email, 'slack-notifications'
    slack_configs:
      - channel: "#alerts"
        send_resolved: true
        icon_url: https://avatars3.githubusercontent.com/u/3380462
        title: >-
          [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing
          | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{
          .CommonLabels.job }}

          {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
            {{" "}}(
            {{- with .CommonLabels.Remove .GroupLabels.Names }}
              {{- range $index, $label := .SortedPairs -}}
                {{ if $index }}, {{ end }}
                {{- $label.Name }}="{{ $label.Value -}}"
              {{- end }}
            {{- end -}}
            )
          {{- end }}
        text: >-
          {{ range .Alerts -}} *Alert:* {{ .Annotations.title }}{{ if
          .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}

          *Description:* {{ .Annotations.description }}

          *Details:*
            {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
          {{ end }}
    email_configs:
      - send_resolved: true
        to: alert@email.com
        from: noreply@grafana.net
        hello: localhost
        smarthost: smtprelay:2525
        headers:
          From: noreply@grafana.net
          Subject: '{{ template "email.default.subject" . }}'
          To: alert@email.com
        html: '{{ template "email.default.html" . }}'
        require_tls: false
templates: []
2

You can configure alert aggregation by setting group_by in Alertmanager config.

examples:

  1. Group alerts by alertname, cluster and service labels
route:
    group_by: ['alertname', 'cluster', 'service']
  1. Disable alert aggregation and pass all alerts as is (not recommended)
route:
    group_by: [...]

Disabling alert aggregation can result in alert spam, only disable it if you have low alert volume. grouping by alertname and other common labels is recommended

see more:

3

Thanks for your reply, @surajsidh!
As you mentioned that it is not a recommended solution, we have decided to leave it as-is for now…

4

Because we wanted an integration with a max-sized content, we have added the proposed solution. It works great!

1 Like