Panel: Calculate usage from two individual queries (prometheus data source)

Grafana Prometheus
, , ,
1

Hey there,

I am fairly new to grafana and just found something where google couldn’t provide me an outright answer. Probably there is something obvious that I am missing or not understanding right now and hope anyone here can help me with - even if it is just by naming my problem.

  • What Grafana version and what operating system are you using?

  • v8.4.1 (53f5c6a44c) on Kubernetes from kube-prometheus.
    Data source is prometheus

  • What are you trying to achieve?
    I am would like to express a relationship between a current value and the maximum possible value to establish the “usage”.
    Ultimately, I would like to express this in absolute numbers and relative numbers (%).
    To be specific, I would like to set the sum of all k8s pod reservations per k8s node in relationship to the maximum available resources.

So for instance the panel should show something like 1.09 CPU requested of 2 CPU maximum
→ (100/2)*1.09 = 54,5% used
This could be a gauge, a bar or whatever works in the end.

  • How are you trying to achieve it?
    The current value is represented by the query
(sum(kube_pod_container_resource_requests{resource="cpu"}) by (node))

The maximum value is

kube_node_status_allocatable{resource="cpu"}

Both work as individual queries as shown in the screenshot:

image
image1920×1080 130 KB

  • What happened?
  • What did you expect to happen?
    Summarising an answer to both questions:
    When I try to do query with both individual metrics I got “no data”, like:
(100 / kube_node_status_allocatable{resource="cpu"}) * (sum(kube_pod_container_resource_requests{resource="cpu"}) by (node))

or

(100 / kube_node_status_allocatable{resource="cpu"}) * (sum(kube_pod_container_resource_requests{resource="cpu"}))

I guess the issue is the mapping between the two queries but I have no idea how to solve it.

An alternative for me would be to use

(sum(kube_pod_container_resource_requests{resource="cpu"}) by (node))

as the graph (e.g. GAUGE) value and reference

kube_node_status_allocatable{resource="cpu"}

as the maximum value - but I have no idea how to do that either and my google fu is failing, too.
I also tried to a math expression like
$B -$A
but I am getting a lot more graphs than I expected…

image
image1920×1080 84.2 KB

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    No errors except “no data” found.

  • Did you follow any online instructions? If so, what is the URL?
    I am not sure what to look for, hence I have no instructions

TY,

Dominik

2

Hi,
your problem is that the two queries don’t return the same set of labels and therefore cannot be matched/multiplied with each other.

The query should look like this:

(100 / kube_node_status_allocatable{resource="cpu"}) * on(node) (sum(kube_pod_container_resource_requests{resource="cpu"}) by (node))

The on() clause tells Grafana to match the second query only by the node label, and ignore the rest.

Best regards.

1 Like