How to have alerts “unsilence” when they resolve and reopen?

Grafana Alerting
1

Hoping someone can help with this - our operations team wants to acknowledge/silence alerts but have them reopen if the underlying conditions resolve and then break again. Is there a way for silences to be contingent on the alert state change timestamp? Or something like that…

1 Like
2

hi @pvnick :wave:

because the alerts continue to evaluate despite they are silenced, you could use this API endpoint
to GET the silence details (i.e the “state” of the alert), and if its value is “firing”, use the DELETE action to remove the silence. I am not sure this is exactly what you need but it is all I could find or think of.

/api/alertmanager/grafana/api/v2/silence/{silenceId}

https://editor.swagger.io/?url=https://raw.githubusercontent.com/grafana/grafana/main/pkg/services/ngalert/api/tooling/post.json

2 Likes
3

I think this got me in the right direction. Instead of having a separate job running, I am going to try using the api within a dashboard using the JSON API grafana data source and do some fancy filtering that way. Thank you for your help!

1 Like