Query with variable does not return data when Multi-Value is enabled

Grafana Prometheus
1

Hi,
i have two smokeping-exporters scraped by prometheus whose values should be displayed as a graph. The prometheus feeds its values to a Victoriametrics Database which can be queried with PromQL and is added as a Prometheus datasource in Grafana.

Each exporter should be defined as an environment, and has several targets. Therefore i have defined two variables in the according grafana dashboard:

  • environment: label_values(ping_rtt_mean_ms, environment)
  • target: label_values(ping_rtt_mean_ms{environment=~"$environment"}, target)

Both variables show preview values, however, the target only shows preview values like it would have if the alphabetically first value of environment would have been chosen.

So i have created a panel with a very simple query:
ping_rtt_mean_ms{environment="$environment", target="$target"} - which is working fine. But now, the problems arise:

When i select one environment, i want to see the results for multiple targets. I have configured the panel to have a repeat by variable “target”, and set the variable target to multi-value. As soon as i enable the multi-value flag for “target”, the panel does not show any value. However, the drop-down value for the variable is populated with the proper values, and if i select more than one target, the panel goes into repeating mode.

So, i think, something with my query inside the panel is not fully correct. Since this is my first time with definint dependent variables, i am seeking for your help, since i cannot solve this on my own. Where is my error?

2

“=” is an exact match in Prometheus.

To do a match on multiple values it would be: “=~” e.g.:

ping_rtt_mean_ms{environment=~"value1|value2|value3"}
or with a variable:
ping_rtt_mean_ms{environment=~"$environment"}

And with variables you may want to select a custom “All” string of .* so it matches on any value. See here for more info:

3

Thank you very much for your suggestions. However, this isn’t working yet. Currently i observe the following:
I change the query for the panel to include the tilde for regex matching, like you suggest. After applying the settings in the panel editor and re-opening the panel editor, the tildes are apparently removed. This is reproducible. Is this something i am using Grafana wrong, or is it a bug?