Elasticsearch alerts underlying query - missing field value for date_histogram

Grafana Elasticsearch
,
1

Hello.

I have a panel built on top of varnish logs that we keep in ES, that measures the p95 percentile of responseTime of HTTP requests.
So far, everything works fine - the chart correctly displays data.

I’ve tried to create an alert for it, but it’s not working, because it has no data (even though the chart and query do).
After some investigation, it turned out, that the underlying request that grafana queries ES when evaluating alerts is missing the value for the field property of the date_histogram of aggs (as shown in Test rule window). This value is however filled with @timestamp for the data query (as shown in Query Inspector).

Below you can find the underlying queries for both - alert evaluation (1st) and chart data (2nd).
If I manually fill aggs.2.date_histogram field in the former query with “@timestamp” then it returns data as it should.

What’s also interesting is a fact that queries produced when evaluating alerts have string timestamps, while queries produced by Query inspector have them as numbers. Also there’s a difference in indexes - string vs array. This doesn’t affect results, but I guess the queries are produced by different code.

Can this somehow be fixed?

  • Grafana v7.5.13 (7b6cadf646)
  • Elasticsearch v7.14.0 (dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1)
  • Lucene version v8.9.0

Alert query (beautified):

{
    "ignore_unavailable": true,
    "index": "varnish-2022.04.20",
    "search_type": "query_then_fetch"
}
{
    "aggs": {
        "2": {
            "aggs": {
                "1": {
                    "percentiles": {
                        "field": "responseTime",
                        "percents": [
                            "95"
                        ]
                    }
                }
            },
            "date_histogram": {
                "field": "",
                "interval": "10m",
                "min_doc_count": 0,
                "extended_bounds": {
                    "min": "1650472128211",
                    "max": "1650473028211"
                },
                "format": "epoch_millis"
            }
        }
    },
    "query": {
        "bool": {
            "filter": [
                {
                    "range": {
                        "@timestamp": {
                            "format": "epoch_millis",
                            "gte": "1650472128211",
                            "lte": "1650473028211"
                        }
                    }
                },
                {
                    "query_string": {
                        "analyze_wildcard": true,
                        "query": "***"
                    }
                }
            ]
        }
    },
    "size": 0
}

Query ( from query inspector - beautified )

{
    "search_type": "query_then_fetch",
    "ignore_unavailable": true,
    "index": [
        "varnish-2022.04.20"
    ]
}
{
    "size": 0,
    "query": {
        "bool": {
            "filter": [
                {
                    "range": {
                        "@timestamp": {
                            "gte": 1650468289147,
                            "lte": 1650470089147,
                            "format": "epoch_millis"
                        }
                    }
                },
                {
                    "query_string": {
                        "analyze_wildcard": true,
                        "query": "***"
                    }
                }
            ]
        }
    },
    "aggs": {
        "2": {
            "date_histogram": {
                "interval": "1m",
                "field": "@timestamp",
                "min_doc_count": "0",
                "extended_bounds": {
                    "min": 1650468289147,
                    "max": 1650470089147
                },
                "format": "epoch_millis"
            },
            "aggs": {
                "1": {
                    "percentiles": {
                        "field": "responseTime",
                        "percents": [
                            "95"
                        ]
                    }
                }
            }
        }
    }
}