Grafana alert health is in error state

Grafana Alerting
,
1

Hi There,

Platform: AWS EKS 1.22
Grafana Version: 9.2

I have created an alert to monitor EKS container restarts using Grafana managed alerts with classic conditions using below YAML code but I am getting error “:invalid format of evaluation results for the alert definition A: looks like time series data, only reduced data can be alerted on”.

apiVersion: v1
kind: ConfigMap
metadata:
  name: alerting
  namespace: monitoring
data:
 contactpoints.yaml: |
   apiVersion: 1
   contactPoints:    
      - orgId: 1
        name: GrafanaEKSAlert
        receivers:     
         - uid: ajay-123
           type: email
           disableResolveMessage: false
           settings:
             addresses: ********************** 
             singleEmail: true
             message: ""
   policies: 
       - orgId: 1    
         receiver: GrafanaEKSAlert
         group_by:     [ Labels ]    
         matchers:     
          - Severity = "Critical"
          - ClusterName = "lab-00-us-east-1"          
         group_wait: 60s
         group_interval: 1m    
         repeat_interval: 5m
   groups: 
     - orgId: 1    
       name: my_rule_group
       folder: LabClusterAlerts   
       interval: 10s
       rules:
        - uid: rule2
          title: TraefikRestartMonitor
          condition: A
          data:
           - refId: A
             relativeTimeRange: 
               from: 300
               to: 0
             datasourceUid: "PBFA97CFB590B2093"
             model:
               expr: increase(kube_pod_container_status_restarts_total{namespace="traefik"}[5m])
               hide: false
               intervalMs: 1000
               maxDataPoints: 43200
               range: true
               refId: "A"
           - refId: B
             relativeTimeRange:
                 from: 0
                 to: 0
             datasourceUid: "-100"
             model:
               conditions:
                 - evaluator:
                      params: 
                       - 0
                      type: gt
                   operator:
                       type: and
                   query:
                      params: "A"
                   reducer:
                       type: last
                   type: query
               datasource:
                 type:  "__expr __"
                 uid: "-100"
               expression: "A"
               hide: false
               intervalMs: 1000
               maxDataPoints: 43200
               refId: "B"
               type: classic_conditions
          execErrState: Alerting
          noDataState: Alerting
          for: 60s
          labels:         
               Severity: Critical         
               ClusterName: lab-00-us-east-1

However similar alert is working from UI and I am not getting any error. Also I have taken this code from /var/lib/grafana/grafana.db and converted from json to yaml.

Please suggest !

Logs from Grafana container**
logger=ngalert rule_uid=rule2 org_id=1 version=72 attempt=0 now=2022-11-29T00:18:50Z t=2022-11-29T00:18:50.179308261Z level=error msg=“failed to evaluate alert rule” result
s=“[{Instance: State:Error Error:invalid format of evaluation results for the alert definition A: looks like time series data, only reduced data can be alerted on. Evaluate
dAt:2022-11-29 00:18:50 +0000 UTC EvaluationDuration:179.302436ms EvaluationString: Values:map}]” duration=178.871377ms

2

Hi! You need to change the condition of the alert from “A” to “B” as time series data must be reduced to a single value to be alerted on. “B” here is a classic condition that will do just that.

1 Like
3

Thank you, as suggested by you I made the following changes and it worked.

Thank you once again for your help !

condition: B
          interval: 10s
          execErrState: Alerting
          noDataState: Alerting
          for: 60s
          labels:         
               Severity: Critical         
               ClusterName: ************