When “If no data or all values are null” is configured with “SET STATE TO” set to either “No Data” or “Alerting” and any of the metrics are seeing no data, the json reply for that alert returns “evalData”: {“noData”: True}. Since it does not return “evalData”: {“evalMatches”: we don’t know which metric is seeing no data.
In addition if one of the metrics is seeing no data and another metrics is alerting, the json still does not include evalMatches so we do not know that there is a metric alerting.
What does your query return when there is no data? Does it return series with null values or no series at all? There can be no series in the eval data if the query returns no series.
When there is no data we get the following (noData seems to replace evalMatches). There nothing indicating which metric is receiving no data. Plus if there was another metric receiving an alert it is not show in the json (in this case the UI Test Rules does show the metric).:
{u’evalData’: {u’noData’: True}, u’name’: u’Overview alert’, u’dashboardUri’: u’db/messaging-overview’, u’newStateDate’: u’2017-11-07T22:38:06Z’, u’state’: u’alerting’, u’executionError’: u’’, u’panelId’: 1, u’dashboardId’: 7, u’message’: u’The messaging tests have fallen below expected number of active connections’, u’id’: 5, u’evalDate’: u’0001-01-01T00:00:00Z’}
When we are not getting any no data failures then we get a list (evalMatches) of metrics:
{u’evalData’: {u’evalMatches’: [{u’metric’: u’staging.Message.failures’, u’value’: 1, u’tags’: None}]}, u’name’: u’Message Low Volume alert’, u’dashboardUri’: u’db/message-overview’, u’newStateDate’: u’2017-11-07T04:53:26Z’, u’state’: u’alerting’, u’executionError’: u’’, u’panelId’: 1, u’dashboardId’: 24, u’message’: u’The Message low volume test has been exceeded failure tolerance thresholds.’, u’id’: 14, u’evalDate’: u’0001-01-01T00:00:00Z’}
Just a little more background. We are running the same test in multiple environments, so we are creating a metric per environment. We are using the API in an automated script to check that each environments is running clean with no alerts. Since no data does not indicate which metric, we are not able to isolate which environment is clean and which is failing.
Under the Alert tab on the Graph panel, the Test Rule seems to report correctly. If any metric is alerting, “firing” is true and “matches” lists the metrics that are alerting. This is true in both cases, another metric has no data and when all metrics have data.
On the Alert List page I think that graph reports “?NO DATA” (but I am going to have to verify this).
