Display hostnames instead of IPs of Nodes in Grafana dashboard

1

Hello,

Our current Grafana dashboard layout displays all server nodes as IP addresses which is difficult for us to keep track of and we prefer to have it as a hostname (or FQDN) display in the drop down.
Any suggestions on how to get that done would be very helpful.
While I do believe it has to be done on the Node exporter(Prometheus) config but I am not sure what configuration has to go in to get that done.

Thanks
S

2

I am just starting myself so I am far from expert, but my experience is that yes in almost all cases you need to track the data you want in Prometheus first. Look into the node_uname_info defined query. There is a hostname “label” there that you will then have to assign to a variable in Grafana. There may be another way to work with non-numeric data in Grafana but I don’t know enough to speak to that.

My issue is if you want to then use that string in Grafana you are going to run into some limitations.

3

In your Prometheus scrape_config, do you specify your target hosts by DNS name or by IP? If you specify them with hostname:port (example: “server01:9100”), you will see hostnames in instance label values instead of IP addresses. Otherwise you can add more labels in the Prometheus configuration.

Reference:

4

Thank you!!

So, we have ‘role: endpoints’ defined under scrape_config. Can it remain that way or should role be defined as a ‘node’.

Also, is this similar to what you are suggesting?

relabel_configs:
- source_labels: [address]
regex: ^(.*):\d+$
target_label: address
replacement: $1:9100

5

The role attribute belongs to either kubernetes_sd_config or openstack_sd_config. If you want this configuration to monitor your Kubernetes nodes, yes you need to change the role to “node”.

The other possibilities are “service”, “pod”, “endpoint” and “ingress”, which are all Kubernetes objects hosted on the nodes and you may want to monitor them too eventually.

The relabel config you pasted is a good idea. It creates an “address” label with just the address part from the “instance” label (without the colon and port number).

The Kubernetes service discovery returns this meta-label for nodes: __meta_kubernetes_node_name. You can use it in a relabel_config too.

scrape_configs:
  - kubernetes_sd_configs:
    - role: node
    relabel_configs:
      - source_labels: __meta_kubernetes_node_name
        target_label: hostname

This may also help you: https://github.com/prometheus/prometheus/blob/release-2.4/documentation/examples/prometheus-kubernetes.yml

Reference for kubernetes_sd_config: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#<kubernetes_sd_config>

6

Thanks for the detailed response!
So, I prefer to have role as ‘node’ since we are looking at having hostnames in the Node (server) dropdown.

So can both relabel configs be used together? (one to get the meta_kube_node_name as hostnames and the second one for hostname display without colon and port? ) - Like this??

scrape_configs:

  • kubernetes_sd_configs:
    • role: node
      relabel_configs:
      • source_labels: __meta_kubernetes_node_name
        target_label: hostname
        relabel_configs:
      • source_labels: [address ]
        regex: ^(.*):\d+$
        target_label: address
        replacement: $1:9100

Thanks for your patience :slight_smile:

S

7

Thanks for the response, i am a novice, so just trying to get my hands dirty, appreciate your inputs!

8

Yes, relabel_configs takes a list of configs, you can have as many as you want. I think you are on the right path.

9

Thank you, much appreciated!!!

10

Hello, I have question same as you.
Are you using Prometheus datasource?
I use InfluxDB and I insert a record to database.

insert host,project=T02,component=ElasticSearch,hostname=ElasticSearch-001,ip=172.17.0.1,name=Elastic001 value=11

Then, I create 2 variables with query:

SHOW TAG VALUES WITH KEY = "hostname" WHERE project='T02'
SHOW TAG VALUES WITH KEY = "ip" WHERE hostname=~ /$hostname/

=> I can use hostname and ip.

Good luck ^^