What is best practice to detect a difference in a query result compared to past result

I was wondering how to go about detecting a change. Lets say I want to detect if a vm is turned off. Of course I can make a query that reports on the amount of vm’s and when this number is not equal to a value, make an alert.

But it would be nicer of course if I would not have to choose as specific value to compare to, but calculate this value from past results.
Any advice how to do this? How far back you are looking and how long until you accept the new value as default?

@f1outsourcing I think something like the automatically generated up{} label in Prometheus might be useful here:

from the docs:

up{job="<job-name>", instance="<instance-id>"}: 1 if the instance is healthy, i.e. reachable, or 0 if the scrape failed.

Thanks! I will try this

This topic was automatically closed after 365 days. New replies are no longer allowed.