Status visualization and alerting for custom scripts

Grafana Alerting
, ,
1

  • What Grafana version and what operating system are you using?
    Grafana Cloud

  • What are you trying to achieve?
    I’m trying to monitor about 30 Python scripts I’ve made. Some run continuously, others periodically. I need a dashboard displaying their current status (good/bad) and alerts when any of them fail.

  • How are you trying to achieve it?
    I’m creating a new script that will run periodically, get information about the other scripts and insert it into Grafana Cloud.
    I’m not sure about what datasource to use, but since I need to push the data, I discarded Prometheus and I’m trying to push it to the Loki instance Grafana Cloud offers.

I’ve managed to push some test logs into it, but I’m lost now…

  1. Is there a panel I could use to list the status of each script? In the logs I tried using the name of the script as the value for a label “service” (e.g.: service=script1). In the dashboard, I tried adding a table panel, querying for a common label (logger=test), but it didn’t work, it adds a select to change the table for each script, and not all scripts status in the same table. It also displays one row per log entry, but for the status I just need the last log entry of each script. I also tried using vonage-status-panel, but it doesn’t seem to support Loki.
  2. Is there a way to automatically create an alert for each script (each value of the label script)?

Thanks for the attention.

2

Have you explored using Grafana Agent? I think it was designed for use cases just like this…

3

Thanks for the answer!

I hadn’t considered using it before. I’ve read that page now and the section about the agent, but I’m not sure how it would help me.

  1. The reason to use the agent is because it would be a way to insert data to Grafana Cloud Prometheus, assuming it would be easier to visualize instead of Loki data?
  2. Which agent mode would I use?
  3. How would I make my reporting script push data to the agent?
4

Hi! I have the same task!
Did you find a solution for this?

5

Hi! Made my own simple dashboard and never used Grafana anymore. :joy:
My project already includes a Django app, so it was easier to add a dashboard there, with total freedom and using a technology that I already know, than to use Grafana.