Node exporter to show average and max cpu/memory usage in past x days

Grafana Prometheus
, , , ,
1

Basically I want to show a table for all hosts (aws ec2) with ave/max usage of cpu/memory over x days, days being a variable defined.
Using following record rules:

    rules:
      - record: node_cpu_usage_seconds_total:rate1m
        expr: rate(node_cpu_seconds_total{mode="idle"}[1m])

      - record: node_memory_MemNotFree_percent
        expr: 100 - (100 * node_memory_MefFree_bytes / node_memory_memTotal_bytes)

I have written 4 queries, but 1) they are giving misleading data and are very slow…:frowning:
Query to get average cpu isage over x days
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[${days}d])) * 100)
To get max CPU usage
max( 100 - max_over_time(node_cpu_usage_seconds_total:rate1m[${days}d])*100 ) by (instance)
To get Average Memory
100 * (1 - ((avg_over_time(node_memory_MemFree_bytes[${days}d]) + avg_over_time(node_memory_Cached_bytes[${days}d]) + avg_over_time(node_memory_Buffers_bytes[${days}d])) / avg_over_time(node_memory_MemTotal_bytes[${days}d])))
Max memory
max(node_memory_MemTotal_bytes - max_over_time(node_memory_MemFree_bytes[${days}d])) by (instance)/(1024*1024*1024)

What may be wrong? How to improve performance?
Thanks in advance
-S

2

Hi @marathiboy,

Thanks for opening this post.

Well, I can only help very little here as not very strong yet with Prometheus and its queries.

As you need mostly values in terms of min, max or Avg. For that, you can go one step back and define your PromlQL queries based on the x number of days and then get the raw data. After that use the Grafana built-in Transformation function to get those min, max or Avg values (may have to play a little bit with it).

Here is the link to really good documentation which explains the Transformation very well:

I hope this helps and hopefully, other community users can add more input to this post.