How can I group Prometheus metrics based on dynamic time windows?

Grafana Prometheus
1

Hello, I’m new to Grafana so bare with me while I explain this as I think it’s a complicated ask.

  • What Grafana version and what operating system are you using?
    9.2.4
  • What are you trying to achieve?
    I am recording the CPU and memory usage metrics of a Node.js instance to Prometheus and the timestamps of when some events start and end. The timestamp events currently look like this in Prometheus (though they don’t always appear in order):
{node_hook_event_total{eventType="start", instance="localhost:9464", job="NodeJS", type="foo", pid="10292", timestamp="1468003781002"}
{node_hook_event_total{eventType="finish", instance="localhost:9464", job="NodeJS", type="foo", pid="10292", timestamp="1468003799009"}
{node_hook_event_total{eventType="start", instance="localhost:9464", job="NodeJS", type="bar", pid="10292", timestamp="1568003781002"}
{node_hook_event_total{eventType="finish", instance="localhost:9464", job="NodeJS", type="bar", pid="10292", timestamp="1568003799009"}
{node_hook_event_total{eventType="start", instance="localhost:9464", job="NodeJS", type="foo", pid="10292", timestamp="1668003781002"}
{node_hook_event_total{eventType="finish", instance="localhost:9464", job="NodeJS", type="foo", pid="10292", timestamp="1668003799009"}

and examples of the CPU & memory events look like this:

process_cpu_user_seconds_total{instance="localhost:8000", job="NodeJS"}

nodejs_external_memory_bytes{instance="localhost:8000", job="NodeRED"}  | 2022-11-09 20:38:57 | 85170246

nodejs_heap_space_size_used_bytes{instance="localhost:8000", job="NodeJS", space="code"}
nodejs_heap_space_size_used_bytes{instance="localhost:8000", job="NodeJS", space="code_large_object"}
nodejs_heap_space_size_used_bytes{instance="localhost:8000", job="NodeJS", space="large_object"}
nodejs_heap_space_size_used_bytes{instance="localhost:8000", job="NodeJS", space="map"}
nodejs_heap_space_size_used_bytes{instance="localhost:8000", job="NodeJS", space="new"}
nodejs_heap_space_size_used_bytes{instance="localhost:8000", job="NodeJS", space="new_large_object"}
nodejs_heap_space_size_used_bytes{instance="localhost:8000", job="NodeJS", space="old"}
nodejs_heap_space_size_used_bytes{instance="localhost:8000", job="NodeJS", space="read_only"}

I’m looking to create a window of time that I can use in other queries for grouping the CPU and memory metrics, so in the example above, there would be three windows of time to represent three “executions” (1 for bar, 2 for foo), with CPU and memory values occurring within each. My intention is then to be able to extract information about each execution on it’s own, or about all executions for a given eventType.

  • How are you trying to achieve it?

So far, I’m using the ‘Transformations > Group By’ to cluster the timestamp events.

  • What happened?
  • I’m unsure how to bring the heap and CPU measures into the graph.
  • I’ve looked at query variables, and I could get a maximum and minimum for the start and finish timestamps, but I don’t know how to get the next finish event after a given start.
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

No.

  • Did you follow any online instructions? If so, what is the URL?

I’m not expecting anyone to give me the ins and outs of Grafana, but I’ve struggled to find examples of this so anything to nudge me in the right direction would be very much appreciated! Thanks