Grafana Elasticsearch Alert Rule not Populating Date Histogram Aggregation Field in Query

Grafana Elasticsearch
1

I’ve configured Elasticsearch 7.12.0 as a data source in Grafana 7.5.2.

I can successfully build a dashboard, which shows the data from Elasticsearch that I expect to see. However, when I attempt to configure an alert, no data is returned.

The issue appears to be that the query created for the alert doesn’t populate the value for date_histogram.field. I.e., when I test the alert, the following query is generated:

{"ignore_unavailable":true,"index":"filebeat-*","search_type":"query_then_fetch"}
{"aggs":{"2":{"date_histogram":{"field":"","interval":"1m","min_doc_count":0,"extended_bounds":{"min":"1623161857376","max":"1623162457376"},"format":"epoch_millis"}}},"query":{"bool":{"filter":[{"range":{"@timestamp":{"format":"epoch_millis","gte":"1623161857376","lte":"1623162457376"}}},{"query_string":{"analyze_wildcard":true,"query":"event.dataset:\"foo.bar\""}}]}},"size":0}

As you can see, date_histogram.field: "" is blank. When I run this query from the command line, I get the following data back:

{
  "took" : 54,
  "responses" : [
    {
      "took" : 38,
      "timed_out" : false,
      "_shards" : {
        "total" : 200,
        "successful" : 200,
        "skipped" : 150,
        "failed" : 0
      },
      "hits" : {
        "total" : {
          "value" : 740,
          "relation" : "eq"
        },
        "max_score" : null,
        "hits" : [ ]
      },
      "aggregations" : {
        "2" : {
          "buckets" : [
            {
              "key_as_string" : "1623161280000",
              "key" : 1623161280000,
              "doc_count" : 0
            },
            {
              "key_as_string" : "1623161340000",
              "key" : 1623161340000,
              "doc_count" : 0
            },
            {
              "key_as_string" : "1623161400000",
              "key" : 1623161400000,
              "doc_count" : 0
            },
            {
              "key_as_string" : "1623161460000",
              "key" : 1623161460000,
              "doc_count" : 0
            },
            {
              "key_as_string" : "1623161520000",
              "key" : 1623161520000,
              "doc_count" : 0
            },
            {
              "key_as_string" : "1623161580000",
              "key" : 1623161580000,
              "doc_count" : 0
            },
            {
              "key_as_string" : "1623161640000",
              "key" : 1623161640000,
              "doc_count" : 0
            },
            {
              "key_as_string" : "1623161700000",
              "key" : 1623161700000,
              "doc_count" : 0
            },
            {
              "key_as_string" : "1623161760000",
              "key" : 1623161760000,
              "doc_count" : 0
            },
            {
              "key_as_string" : "1623161820000",
              "key" : 1623161820000,
              "doc_count" : 0
            },
            {
              "key_as_string" : "1623161880000",
              "key" : 1623161880000,
              "doc_count" : 0
            }
          ]
        }
      },
      "status" : 200
    }
  ]
}

The total number of hits shows the expected value, the individual buckets show doc_count: 0. However, if I set date_histogram.field: "@timestamp" while executing the query from the command line as follows, it segregates the count into buckets, as expected.

{"ignore_unavailable":true,"index":"filebeat-*","search_type":"query_then_fetch"}
{"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"1m","min_doc_count":0,"extended_bounds":{"min":"1623161336265","max":"1623161936265"},"format":"epoch_millis"}}},"query":{"bool":{"filter":[{"range":{"@timestamp":{"format":"epoch_millis","gte":"1623161336265","lte":"1623161936265"}}},{"query_string":{"analyze_wildcard":true,"query":"event.dataset:\"foo.bar\""}}]}},"size":0}
2

It looks as if this is the issue resolved by this pull request. I didn’t notice that there was no field selected for the “Group By” in my query, because of the fact that the data showed as expected. Once I did that, the query included the date_histogram.field value and the alert query returned the results I expected.

3

Hi,

You should also check out the new Unified Alerting in Grafana 8, which shipped today. It’s an entirely new alerting platform with much greater power and ease-of-use.

Check it out:
docker run -p 3000:3000 --name=grafana -e "GF_FEATURE_TOGGLES_ENABLE=ngalert" grafana/grafana:8.0.0