Hi,
I am having troubles in creating Grafana alerts. Raising this topic here to get some help in identifying the issue.
Below are the details.
Grafana version: 7.5.9
Data source: AWS Elasticsearch(Open Distro for Elasticsearch).
Query:
Alert:
Test rule:
{
“firing”: false,
“state”: “no_data”,
“conditionEvals”: “false = false”,
“timeMs”: “46.340ms”,
“logs”: [
{
“message”: “Condition[0]: Query”,
“data”: {
“from”: 1631105903722,
“queries”: [
{
“refId”: “A”,
“model”: {
“alias”: “”,
“bucketAggs”: [
{
“field”: “@timestamp”,
“id”: “1”,
“settings”: {
“interval”: “1m”,
“min_doc_count”: “1”
},
“type”: “date_histogram”
}
],
“format”: “table”,
“metrics”: [
{
“field”: “response_time_ms”,
“id”: “1”,
“meta”: {
“max”: true,
“std_deviation_bounds_lower”: false,
“std_deviation_bounds_upper”: false
},
“type”: “extended_stats”
}
],
“query”: “message.raw:"Request finished" AND app_name.keyword:pharmacy-api”,
“queryType”: “lucene”,
“refId”: “A”,
“timeField”: “@timestamp”
},
“datasource”: {
“id”: 11,
“name”: “Elasticsearch - health-prod”
},
“maxDataPoints”: 0,
“intervalMs”: 0
}
],
“to”: 1631106203722
}
},
{
“message”: “Condition[0]: Query Result”,
“data”: {
“fromDataframe”: true,
“series”:
}
},
{
“message”: “Condition: Eval: false, Query Returned No Series (reduced to null/no value)”,
“data”: null
}
]
}
Debug log related to Alert:
t=2021-09-08T12:55:40+0000 lvl=dbug msg=“Scheduler: Putting job on to exec queue” logger=alerting.scheduler name=“TestAlert alert” id=128
t=2021-09-08T12:55:40+0000 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=126 name=“Pharmacy API response times alert” changing state to=no_data
t=2021-09-08T12:55:40+0000 lvl=info msg=“Alert Rule returned no data” logger=alerting.evalContext ruleId=128 name=“TestAlert alert” changing state to=no_data
t=2021-09-08T12:55:40+0000 lvl=info msg=“New state change” logger=alerting.resultHandler ruleId=128 newState=no_data prev state=unknown
t=2021-09-08T12:55:40+0000 lvl=info msg=“Database locked, sleeping then retrying” logger=sqlstore error=“database is locked” retry=0
t=2021-09-08T12:55:40+0000 lvl=info msg=“Alert already updated” logger=alerting.resultHandler
t=2021-09-08T12:55:40+0000 lvl=dbug msg=“Job Execution completed” logger=alerting.engine timeMs=56.802 alertId=128 name=“TestAlert alert” firing=false attemptID=1
I have tried multiple conditions as well like,
query(A,1m,now-1m),
query(A,1m,now-5m)
I have upgraded to latest version as well but same issue then i reverted the version.
This same alert works perfect when i change the data source to dev index and fails when i change it back to point to prod index.
Please let me know if any further details are needed to provide support on this.