Hi, I am very new to Grafana (and Prometheus) and am having a really hard time working out how to do something which I thought would be very simple.
I’m using node_exporter, prometheus and grafana (v6.3.5). I would like to create a dashboard that has one row per instance, one panel in that row, that panel being a bandwidth graph.
So, first step was to create a bandwidth graph for one instance. A Prometheus query that looks like:
rate(node_network_receive_bytes_total{instance="192.168.80.22:9100",device="bond0"}[1m])*8
(and a matching one for transmit_bytes_total, but I’ll omit those for simplicity)
Great, that works, but I don’t want to manually add a panel for each instance. I’d like it to just get the list of instances from Prometheus.
So, I added a dashboard variable called “instance”:
label_values(node_network_iface_id{device="bond0"}, instance)
idea being, that gives me a list of instances that have a bond0 device. That is set to “Include All option”.
The row is now set to repeat over the “instance” variable, and the query string is set to:
rate(node_network_receive_bytes_total{instance=~"[[instance]]",device="bond0"}[1m])*8
All good so far, three different bandwidth graphs. Although I will say that as a complete newbie it took me quite a while to realise I needed to put =~"[[instance]]" there and not simply ="$instance".
Anyway, now I’d like to have a more descriptive panel title like “somenode - Traffic - bond0”. I have used IP addresses as my instance names in the Prometheus config. Possibly that was an error, which I can still revisit, but conceptually it seemed to me that I should be able to add another dashboard variable that is like:
Name: nodename
Query: label_values(node_uname_info{instance=~"[[instance]]"}, nodename)
as long as it is below the definition of the “instance” dashboard variable. The preview does give me the (currently) three different nodenames as I would expect.
But now when I change the panel title to be:
$nodename - Traffic - bond0
when I am looking at the “All” selection from the dropbox at the top of the dashboard, every row has the same nodename. The graph values are correct for the given instance, that is, these are different graphs. It is just the expansion of the variable “nodename” that seems to remain constant.
If I use the dropdown to select a single instance then the single panel does have the correct title. So it is only in the “All” case that this doesn’t work.
Is this because when the “All” option is selected, my query for a matching nodename does not work properly?
Should I be injecting the nodename in some other way?
Should I be adding a new label at the Prometheus level that is the nodename and inserting that as a variable into Grafana somehow?
I suppose this is a Prometheus-specific question, but is using IP addresses as Prometheus instance names a bad idea all around? I figured that since they would have the ugly “:9100” at the end anyway, I’d be doing a text substitution on them at some point and it didn’t really matter.
Thanks,
Andy