Grafana agent scrape config

Grafana Cloud
1

Hi. I am new to the prometheus world and trying to set up the “kube-state-metrics” to gather some more metrics from our cluster.
We have an AKS cluster with the Grafana Agent running and sending metrics, logs and traces to Grafana cloud. I’ve installed the kube-state-metrics which exposes a lot of metrics on :8080/metrics (i’ve confirmed this with a curl on another pod, to this pod).
In the config file for the agent, I’ve added the following to enable the scraping of this new pod:

        scrape_configs:
          - job_name: 'kube-state-metrics'
            static_configs:
            - targets: ['kube-state-metrics.kube-system:8080']

There is however no sign of the metrics data i would expect in grafana (we use the default config file, so we have a lot of other metrics available).
I am a bit unsure how to approach this and how to figure out why it’s not working.
Any help is appreciated

2

Hey @svj!

Thanks for your post. Let’s try to figure this out.

Given that you can curl the endpoint we know that’s working fine… so the issue must be with agent scraping it / processing it, or further down the line, with agent shipping those metrics to Grafana Cloud. If you can see your other metrics in Grafana Cloud that leads me to believe it’s an issue with agent scraping / processing the endpoint.

It’s a little hard to debug without more of your agent configuration and architecture. If you only have one agent scraping stuff in your cluster and have host_filter: true, that could be it, as this parameter ensures the agent only scrapes stuff that is running on the same node. To learn more about this feature see agent/overview.md at main · grafana/agent · GitHub and agent/operation-guide.md at main · grafana/agent · GitHub.

3

If it’s not that, and you’ve rolled out the agent as a DaemonSet so there is an agent on the node running kube-state-metrics (in this case host filtering doesn’t matter), here are some things you can do to debug:

Other potential sources for this issue:

  • K8s RBAC (I think unlikely given other stuff is working fine)
  • K8s DNS (i think also unlikely)
  • If you have any filtering rules in your config after targets, the metrics could be getting dropped there. Hard to tell without sharing more of your config.

Hope this helps you get started resolving your issue. Please post your solution here so others can learn how you fixed this!

4

Also, when you say “default config file,” while file are you referencing?

5

Hi hjet
Thank you for looking into this.

I checked the host filter and it was indeed set to true - switcing this to false seems to have solved the “issue”. I now have all the expected metrics data available in grafana.

Thanks a lot!

1 Like
6

Bumping this post because it’s the only one here talking about agentctl tool.
I am trying to get target-stats using this tool but failing to pass the arguments correctly…
I tried:

./agentctl-linux-amd64 target-stats /tmp/grafana-agent-wal/ kube-state-metrics

./agentctl-linux-amd64 target-stats /tmp/grafana-agent-wal flags=kube-state-metrics

./agentctl-linux-amd64 target-stats /tmp/grafana-agent-wal -j=kube-state-metrics

Nothing seems to be working…
The documentation seems to be non existent…
Please help