Hi All,
i’m quite new with Grafana, hope to find some help here (my very first post too! )
i’ve installed prometheus - alertmanager - grafana (8.0.6) stack on a test EKS cluster (which is the AWS managed version of k8s) with helm using prometheus-community/kube-prometheus-stack chart.
everything is fine with the default install, i can log into grafana admin console (with a port-forward on port 3000), and i can see quite a lot of predefined alert rules already available.
to obtain it, i’ve changed the grafana.ini adding the following snippet to enable the new integrated multialert console from grafana 8.
[feature_toggles] enable = ngalert
(and restarted the pod to enable it)
-
what i can do:
i can create a channel for alert, and i can manually create a firing alert (i can receive it on telegram, tested)
-
what i cannot do:
i cannot receive the alerts from all the preconfigured rules under Cortex / Loki
what i should do to receive these alerts to my configured telegram channel?
thank you for your help,
BR
Alberto
Welcome to the community, Alberto! Thanks for asking your questions here. I’m going to give a little extra context that I hope is useful in explaining what’s going on here ( I suspect).
The alerting engine, under the hood, has two possible code paths that start with the API. One is grafana managed alerting rules that are evaluated by our new alerting engine. The other are the LoTex alerting rules (Loki/Cortex) that are evaluated by the Loki of Cortex datasources you have registered with your grafana instance. You are able to create alerting rules for a LoTex datasource using the UI or the API, but the evaluation and notification for them occurs with that datasource and so the routes and receivers you configure for grafana managed alerts are not available to them, and vice versa.
You have two options here! You can either create a grafana managed alerting rule that will query the datasource at evaluation time and use the alerting configuration for grafana managed alerts or you can create the routes and receivers you’re interested in within the alertmanager used by the datasource. The first option gives you some nice features like dashboard linking and the flexible expression engine used by grafana. The second option is convenient though if you’ve already got the alerting rules you’d like to use, and just need to get the notification piece hooked up.
Hope this helps and please keep asking questions here!
1 Like
hi David!
and thank you very much for your detailed replay!
my “fear” what that the LoTex datasource was still (?) not compatible/integrated with Grafana notification system, and your description confirm it …
indeed, my goal is to have a portable monitoring system ready out of the box, deploying it with just:
helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring -f values.yaml
and be ready to receive alerts starting from consolidated predefined rules. i have to find a way (google help me please!) to integrate in the very simple way an alert method, to leverage from the already available rules.
hopefully in a near future they will integrate it, as i think have it native it’s a must!
thank you again!
BR
Alberto
@alcolombo75 feel free to +1 or create a feature request for this, so that we can gauge interest. I know that we’re doing some brain storming about how this could work but I can’t really speak to if/when/how etc.
1 Like
