I have created an alert.
Grafana fires it and it’s state is pending but the alert never get to the alert state.
Any idea?
Thank you.
Change the now-1m in ther conditions to now, if you want to make the Alert just calculate within 1minute, just open the Query Option and fill the realtive time to 1m and hide time info, also try to change the Evaluation Period to 1m for 5m as default.
THen check the Rule and wait for the alert.
Done, not very hopeful though, I’ve been drifting away from defaults when trying to understand why it didn’t work so all that has been tested already.
Didn’t work. Any tip on how to investigate?
I think you could check the log, either in /var/log/message or in /var/log/gafana/grafana if you set in the grafana.ini
Yes, I did that, I’ve set debug mode level and got this kind of message:
t=2020-10-13T10:00:42+0300 lvl=dbug msg=“Scheduler: Putting job on to exec queue” logger=alerting.scheduler name=“Low disk watermark on elasticsearch” id=25
t=2020-10-13T10:00:42+0300 lvl=dbug msg=“Job Execution completed” logger=alerting.engine timeMs=32.333 alertId=25 name=“Low disk watermark on elasticsearch” firing=false attemptID=1
firing=false is the odd thing since “Test rule” shows firing = true.
DId you already save the dashboard? after change the parameters?
Grafana only use the saved dashboard in the database to run the Alerts…
That was it, wow… It’s misleading.
When testing the rule, changes are taken into account.
Alright, lesson learned. I just didn’t see anything about it in the documentation.
Thank you.