I am using grafana 10.0.2 and would like to silence alerts and rules using API however unable to achieve the same.
/usr/share/grafana/bin/grafana cli -v
grafana version 10.0.2
Requirement is :
adding the capability to create silence for an alert
adding the capability to create silence for a rule
adding the capability to remove silence for an alert
adding the capability to remove silence for a rule
Our alerts look like below:
{
"status": "success",
"data": {
"alerts": [
{
"labels": {
"alertname": "[prod] MongoDB Replication Lag - Casper",
"cluster": "prodcluster",
"environment": "prod",
"grafana_folder": "Cluster Casper",
"ui_frontend": "mongo replication latency"
},
"annotations": {
"__dashboardUid__": "2nrap_l4z",
"__panelId__": "702"
},
"state": "Error",
"activeAt": "2023-07-17T12:21:20Z",
"value": ""
},
{
"labels": {
"alertname": "[prod] Shallow flows Runtime - Casper",
"cluster": "prodcluster",
"environment": "prod",
"grafana_folder": "Cluster Casper",
"shallow_flows": "runtime"
},
"annotations": {
"__dashboardUid__": "2nrap_l4z",
"__panelId__": "416"
},
"state": "Error",
"activeAt": "2023-07-17T12:24:00Z",
"value": ""
},
{
"labels": {
"alertname": "[prod] Policy Store current connections with leader - Casper",
"cluster": "prodcluster",
"environment": "prod",
--More--
Our Rules look like below:
{
"Cluster Prod": [
{
"name": "Pipelines",
"interval": "1m",
"rules": [
{
"expr": "",
"for": "0s",
"labels": {
"cluster": "casper",
"environment": "Prod",
"inventory": "greater than 1000000 or increased by more than 20%"
},
"annotations": {
"__dashboardUid__": "2nrap_l4z",
"__panelId__": "405"
},
"grafana_alert": {
"id": 1299,
"orgId": 1,
"title": "[Prod] Learned Inventory Items - Prod",
"condition": "C",
"data": [
{
"refId": "A",
"queryType": "",
"relativeTimeRange": {
"from": 900,
"to": 300
},
"datasourceUid": "P6C206B30B775F990",
"model": {
"aggregator": "sum",
"alias": "Learned Inventory Items",
"datasource": {
"type": "opentsdb",
"uid": "P6C206B30B775F990"
},
"downsampleAggregator": "avg",
"downsampleFillPolicy": "none",
--More--
Silence : curl -u xxx:xxx --location --request POST 'http://grafana.staged-by-discourse.com/api/alertmanager/grafana/api/v2/silences' --header 'Content-Type: application/json' -d '{"startsAt":"2022-07-27T07:01:01.783Z","endsAt":"2022-07-27T09:01:01.783Z","comment":"created 2022-07-27 15:01","matchers" [{"name":"alertname","value":"[GCP-DEVPLAYGROUND] Virtual Appliances - Service Process stats memory percent - Gcpsensor","isEqual":true,"isRegex":false}]}'
{“message”:“bad request data”,“traceID”:“”}
I am trying to silence using curl command however it does not work, Once I achieve this manual I will program it using python, Please advise .
Example of existing silence
curl -u XXX:XXXX 'http://grafana.staged-by-discourse.com/api/alertmanager/grafana/api/v2/silences' |jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1148 100 1148 0 0 40433 0 --:--:-- --:--:-- --:--:-- 41000
[
{
"id": "82f6ada4-d994-436a-bb5c-dc6ffb6d26c1",
"status": {
"state": "active"
},
"updatedAt": "2023-07-18T14:24:32.466Z",
"comment": "created 2023-07-18 16:22",
"createdBy": "User",
"endsAt": "2023-07-25T14:22:34.951Z",
"matchers": [
{
"isEqual": true,
"isRegex": false,
"name": "alertname",
"value": "[prod] Services test Up - cluster"
},
{
"isEqual": true,
"isRegex": false,
"name": "cluster",
"value": "cluster"
},
{
"isEqual": true,
"isRegex": false,
"name": "environment",
"value": "prod"
},
{
"isEqual": true,
"isRegex": false,
"name": "service",
"value": "service1"
}
],
"startsAt": "2023-07-18T14:24:32.466Z"
},