I have a cloud based multi-VM setup with multiple docker containers of the same service.
On each VM, an nginx container acts as reverse proxy to load balance across the service containers and utilize the VM resources fully.
This is a node.js service, I have integrated Prometheus metrics in the application am emitting the metrics at container-ip:portXYZ/metrics. If I do a curl on that while sshed on the host, I am able to see the custom metrics I’m emitting.
Used the following packages in the application code to generate custom metrics
I also set up Grafana Agent (Linux Integration) on the VMs and am able to see the host level metrics in Grafana Cloud at the moment. I think the agent makes them available @ host:12345/metrics
Coming back to custom metrics, If query the api enpoint i.e api.org.com/metrics it returns metrics from any one of the container across the multiple VMs.
How to collate these custom metrics to per host level and push to Grafana cloud?
So if I select a VM I should be able to see the collated data across all the containers.
Given that you’re using the Grafana Agent, you have to add jobs to scrape these metrics endpoints you’re exposing. It won’t do this out of the box unless you use one of the available integrations.
I’m not sure what you’re using to manage the containers, but if it’s Docker Swarm, you can check out the configuration reference for dockerswarm_sd_config here: agent/configuration-reference.md at main · grafana/agent · GitHub. Using this you’d be able to use service discovery to get a list of targets/endpoints to scrape.
If you’re not dynamically managing these containers and they will always be available at the same IP/port, then you can use static_config to hardcode the targets to scrape. Unfortunately it appears that docker_sd_config (Configuration | Prometheus) has not been implemented for Agent yet, but if this is something you need, please create an issue on GitHub and our engineers will take a look!
The Agent configuration parameters closely mirror the Prometheus ones, so you can consult the Prometheus docs to learn more and see more examples. The Agent getting started docs may also be useful: agent/getting-started.md at main · grafana/agent · GitHub
In addition to your app metrics, you may also wish to scrape and ship Docker container metrics. You can do this with cAdvisor. To learn more see this doc from the Prometheus docs: Monitoring Docker container metrics using cAdvisor | Prometheus (once again the Agent configuration will be really similar)
The collation/aggregation is done using Prometheus labels, which you set when you define the scrape configuration. Some labels are populated automatically but you can modify, replace, drop, and add labels in these configurations. To learn more about labels, see this blog post: Prometheus Blog Series (Part 1): Metrics and Labels
