I am defining an alert and I have a query that is returning the number of 500 errors that occurred in a 1 minute interval. We’ll call this Query A.
Let’s say that 5 500 errors were detected in the 5 minute interval. I’d like to have another query (well call this one QueryB) that JOINs the to Query B on the ElasticSearch TraceId fields.
Query B will return all the messages related to the captured 500 errors.
I’d like to get the results from Query B and use add that to an Alert Tag. That way, I can pass over the details of the exceptions to xMatters(an Incidence Management System like PagerDuty).
That way, when on-call staff receives this alert in xMatters and opens it, it will show some diagnostic information about the specific of the errors.
Is it possible to use tags for this? And if I can use Tags, how else can I “decorate” the alert payload so it contains related diagnostic information?