Alerting Mysql Datasource Mute Timing


I am trying to create an alert for no data only using a mysql data source. We have a server that runs specific jobs during business hours only and writes the results to a database. If the jobs fail then no data is written to the db, which is currently what we are looking to get alerted on.

How the Alert is currently set up (Input A):

status AS job_results,
IF(status != NULL, 0, 1) AS fakeStat
FROM table
WHERE jobName like "%JOBNAME_HERE%" AND dateadded > date(now())

Alert Condition: Threshold Input A is above 1

I created a fake metric (fakeStat) and set the threshold to trigger an alert if the value is ever over 1. From there, I added the option to Alert us if no data is received. We should never receive an alert for the fake metric. The database is written to every 10 minutes, so we have an evaluation interval of 10 minutes for 1h. We also added a filter to only pull jobs from today date.

These jobs only run between m-f from 9a to 5p and I have setup mute timings accordingly. There is no data being written to the db outside of these hours. The problem That I am running into is every morning, we are getting an alert that no data has been received because even though we have mute timing setup, its looking at the non business hours as well.

Is there anyway to ignore when no data is received outside of business hours?

The footer of the alert reads: Observed 4h0m35s before this notification was delivered, at 2023-06-28 X:X:X X:X:X MST

Running Grafana v9.5.2

So is the alert triggered on fakeStat?

No, I just want it to alert if no data is received. fakeStat is just an arbitrary metric since we need to define an alert condition.

So it will always have a value of either 0 or 1.
Can you show us the alert config screen shot?

I think the better question is how can I have the rule not be evaluated outside of business hours?

from the grafana documentation found here: Create mute timings | Grafana documentation

Similar to silences, mute timings do not prevent alert rules from being evaluated, nor do they stop alert instances from being shown in the user interface. They only prevent notifications from being created.

hence when the alert comes out of the mute timing in the morning, it fires because we haven’t received data since yesterday at the end of the day.

What I have tried already is moving the mute time up an hour after the start of business hours so it can collect that data and not fire an alert. I just made some changes to that again and will report back with my findings in the morning.

1 Like

see below: