How can I get CPU usage for multiple servers within a single panel?

Grafana Prometheus
1

Hi,

I’m relatively new to Grafana\Prometheus & I’m trying to build a server monitoring dashboard using the Boom Table Panel with the goal at this very early stage to have a row for each server selected, showing memory & CPU usage. When multiple servers are selected, memory usage works as expected & we get a row for each server, however CPU usage appears to be aggregated & this is also the case if we take the Boom Table out of the equation. I suspect that it’s something to do with the metric query I’m using, which works absolutely fine for a single server, but as soon as I add others, it provides a single aggregated number rather than the usage for each server.

The query I’m using is :

100-(avg(irate(windows_cpu_time_total{instance=~“$instance”,mode=“idle”}[2m])))*100

I’d really appreciate it if someone could point the error of my ways :slight_smile:

Thanks!

2

Hi,

If you want to show the all instance, please use this:

image
image404×752 26.2 KB

Then select the instance, so it’ll show the repeat of all instances…

Regards,
Fadjar

3

Hi Fadjar,

Thanks for replying!

That approach works fine for multiple panels, but I’m looking to have a single panel with a table showing all servers & their stats, so another column here showing CPU usage…

image
image577×557 72.8 KB

Like I say, I’m pretty sure that the query for CPU is taking an average across all servers & I’m trying to achieve something like this…

image
image940×444 64.6 KB

Thanks,

Neil

4

Hi…

I build this panel that suitable with my requirement

image
image1305×364 80.3 KB

This panel just using Table with 3 transform, Merged, Group By and Organized Fields.
The Datasource using Prometheus

Regards,
Fadjar

5

Hello Fadjar,

This panel you made is great! How can I make/download it? Don’t leave us hanging! :slight_smile:

Regards,
Seth

6

Below the panel json if you want to use:

{
  "datasource": {
    "type": "prometheus",
    "uid": "djoI8bg4k"
  },
  "fieldConfig": {
    "defaults": {
      "custom": {
        "align": "center",
        "cellOptions": {
          "type": "auto"
        },
        "inspect": false
      },
      "mappings": [],
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {
            "color": "semi-dark-green",
            "value": null
          },
          {
            "color": "semi-dark-yellow",
            "value": 75
          },
          {
            "color": "semi-dark-red",
            "value": 80
          }
        ]
      },
      "color": {
        "mode": "thresholds"
      },
      "max": 100
    },
    "overrides": [
      {
        "matcher": {
          "id": "byName",
          "options": "CPU Load"
        },
        "properties": [
          {
            "id": "custom.cellOptions",
            "value": {
              "mode": "lcd",
              "type": "gauge"
            }
          },
          {
            "id": "thresholds",
            "value": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "#EAB839",
                  "value": 75
                },
                {
                  "color": "red",
                  "value": 95
                }
              ]
            }
          },
          {
            "id": "custom.width",
            "value": 231
          }
        ]
      },
      {
        "matcher": {
          "id": "byName",
          "options": "RAM"
        },
        "properties": [
          {
            "id": "custom.cellOptions",
            "value": {
              "mode": "basic",
              "type": "gauge"
            }
          },
          {
            "id": "thresholds",
            "value": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "#EAB839",
                  "value": 75
                },
                {
                  "color": "red",
                  "value": 95
                }
              ]
            }
          }
        ]
      },
      {
        "matcher": {
          "id": "byName",
          "options": "Disk Usage"
        },
        "properties": [
          {
            "id": "custom.cellOptions",
            "value": {
              "mode": "lcd",
              "type": "gauge"
            }
          },
          {
            "id": "thresholds",
            "value": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green",
                  "value": null
                },
                {
                  "color": "#EAB839",
                  "value": 80
                },
                {
                  "color": "red",
                  "value": 95
                }
              ]
            }
          }
        ]
      },
      {
        "matcher": {
          "id": "byName",
          "options": "Node"
        },
        "properties": [
          {
            "id": "mappings",
            "value": [
              {
                "options": {
                  "pattern": "(.*):9796",
                  "result": {
                    "index": 0,
                    "text": "$1"
                  }
                },
                "type": "regex"
              }
            ]
          },
          {
            "id": "custom.width",
            "value": 140
          }
        ]
      },
      {
        "matcher": {
          "id": "byName",
          "options": "Cores"
        },
        "properties": [
          {
            "id": "custom.cellOptions",
            "value": {
              "mode": "basic",
              "type": "color-background"
            }
          },
          {
            "id": "color",
            "value": {
              "fixedColor": "semi-dark-blue",
              "mode": "fixed"
            }
          },
          {
            "id": "custom.width",
            "value": 86
          }
        ]
      }
    ]
  },
  "gridPos": {
    "h": 15,
    "w": 19,
    "x": 5,
    "y": 9
  },
  "id": 12,
  "options": {
    "showHeader": true,
    "cellHeight": "sm",
    "footer": {
      "show": false,
      "reducer": [
        "sum"
      ],
      "countRows": false,
      "fields": ""
    },
    "frameIndex": 0,
    "sortBy": [
      {
        "desc": true,
        "displayName": "RAM"
      }
    ]
  },
  "pluginVersion": "9.5.2",
  "targets": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "djoI8bg4k"
      },
      "editorMode": "code",
      "expr": "(sum by (instance) (node_cpu_seconds_total{job=\"linux_vm\",mode!=\"idle\"})) / (sum by (instance) (node_cpu_seconds_total{job=\"linux_vm\"}) ) * 100 ",
      "format": "table",
      "hide": false,
      "legendFormat": "CPU",
      "range": true,
      "refId": "B"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "djoI8bg4k"
      },
      "editorMode": "code",
      "expr": "((((sum by (instance) (node_memory_MemTotal_bytes{job=\"linux_vm\"})) - (sum by (instance)(node_memory_MemFree_bytes{job=\"linux_vm\"}))) / (sum by (instance) (node_memory_MemTotal_bytes{job=\"linux_vm\"}))) * 100)",
      "format": "table",
      "hide": false,
      "legendFormat": "Memory",
      "range": true,
      "refId": "A"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "djoI8bg4k"
      },
      "editorMode": "code",
      "expr": "((((sum by (instance) (node_filesystem_size_bytes{job=\"linux_vm\"})) - (sum by (instance) (node_filesystem_free_bytes{job=\"linux_vm\"}))) / (sum by (instance) (node_filesystem_size_bytes{job=\"linux_vm\"}))) * 100)",
      "format": "table",
      "hide": false,
      "legendFormat": "Disk",
      "range": true,
      "refId": "C"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "djoI8bg4k"
      },
      "editorMode": "code",
      "expr": " sum by (instance)(count without (job,mode) (node_cpu_seconds_total{job=\"linux_vm\",mode=\"idle\"}))",
      "format": "table",
      "hide": false,
      "range": true,
      "refId": "E"
    }
  ],
  "title": "Server Capacity",
  "transformations": [
    {
      "id": "merge",
      "options": {}
    },
    {
      "id": "groupBy",
      "options": {
        "fields": {
          "Value #A": {
            "aggregations": [
              "lastNotNull"
            ],
            "operation": "aggregate"
          },
          "Value #B": {
            "aggregations": [
              "lastNotNull"
            ],
            "operation": "aggregate"
          },
          "Value #C": {
            "aggregations": [
              "lastNotNull"
            ],
            "operation": "aggregate"
          },
          "Value #E": {
            "aggregations": [
              "lastNotNull"
            ],
            "operation": "aggregate"
          },
          "instance": {
            "aggregations": [],
            "operation": "groupby"
          }
        }
      }
    },
    {
      "id": "organize",
      "options": {
        "excludeByName": {},
        "indexByName": {
          "Value #A (lastNotNull)": 3,
          "Value #B (lastNotNull)": 2,
          "Value #C (lastNotNull)": 4,
          "Value #E (lastNotNull)": 1,
          "instance": 0
        },
        "renameByName": {
          "Value #A (lastNotNull)": "RAM",
          "Value #B (lastNotNull)": "CPU Load",
          "Value #C (lastNotNull)": "Disk Usage",
          "Value #E (lastNotNull)": "Cores",
          "instance": "Node"
        }
      }
    }
  ],
  "type": "table"
}

Just tune the query for your prometheus data.

Regards,
Fadjar

2 Likes
7

Hey Fadjar, I finally got a chance to try this out. It worked well once I got the data source right.
Big thanks!

8

Hi There,

I am looking for something similar as well. I tried to copy this Json for 1 of my panel (after changing the datasource uid) but there is no data in the table (the table shows “no data”). I am also not getting any errors and it seems that grafana finds the parameters and the methods (group by, merge) properly. Do you have some idea what could go wrong? I am very new to grafana and prometheus, just trying to figure out basic stuff that I need.

Thank you for the help!

9

First of all, try to remove all the transformation, then check the data in the table view.
Check the query and make sure the data already show up.
Then step by step, put the transformation as per your requirement…
Perhaps the transformation is not suitable for your query…
Please pay attention in the job value… It;s depends on your data value in the Prometheus

Regards,
Fadjar

10

I am afraid that I am a bit of a newbie to properly understand everything that you just typed. I removed the “transformations” part from the json code, but it is still not working and the table view says “no data”. I am using prometheus 2.45.0 and grafana 10.0.1.

What can I do? By the way are you available for hire?

11

Please check the expression of the query:

“((((sum by (instance) (node_memory_MemTotal_bytes{job=“linux_vm”})) - (sum by (instance)(node_memory_MemFree_bytes{job=“linux_vm”}))) / (sum by (instance) (node_memory_MemTotal_bytes{job=“linux_vm”}))) * 100)”,`

At this query, look at the job.
That says: “linux_vm”.
Is your data in the Prometheus have job like that?
If no, change that suitable with your data.

Regards,
Fadjar

PS: message me for the job… :slight_smile:

12

Thank you!

Indeed after I changed the job name it started to work properly. Thank you! I am going to send a PM in the meantime.