Step parameter of Prometheus range query and different intervals

Grafana Prometheus
1

Hello,

  • What Grafana version and what operating system are you using?

grafana-9.0.0-1.x86_64 on RHEL 8.4 EUS x86_64 with prometheus 2.32.1 as data source

  • What are you trying to achieve?

Understand concepts behind Min interval and Min step relatively to $__interval and $__rate_interval use in range vector parameter and the step parameter ultimately passed to Prometheus range query

  • Did you follow any online instructions? If so, what is the URL?

I’ve read documentation and This blog post and watched this talk

I’m trying to figure out some experienced behaviors when tweaking Min step and/or Min interval while range vector parameter is $__interval and/or $__rate_interval

My understanding of the $__rate_interval var, defined as max($__interval + Scrape interval, 4 * Scrape interval) is that it solves the problems which can occur when using $__interval as a range vector range parameter which are:

  • no more data when too much zoomed in
  • sudden change of data on interval boundaries

while still having the initial desired effect of $__interval which is not to miss peaks when range vector range parameter is a lot smaller than time range

I understand that Min interval has no effect on $__rate_interval according to the above formula

My understanding of Min step is that it is specific to Prometheus datasource and that it can be used to indicate a specific scrape interval different than the global one and that when set, it will not change $__interval value

my understanding of Min interval is that it is not specific to a particular datasource and that, if set, it will override $__interval value

At first I thought that the step of the range query was $__interval except when Min interval was set (then step would be Min interval) or Min step was set (then step would be Min step). But this does not seem to be always the case so my reasoning must be incorrect.

For instance, when not too much zoomed in (let’s say last 6 hours), the above behavior seems to be true with a metric looking like:

rate(process_cpu_seconds_total{...}[$__rate_interval])

But not anymore when much more zoomed in (let’s say 30 last days)

So basically my question are:

  • what excatly are the conceptual differences between Min interval and Min step ?
  • why Min interval does change interval but Min step does not ?
  • what happens when both Min interval and Min step are sent ?
  • how is the step parameter of the range query sent to Prometheus calculated ?
  • why does documentation states that Min interval changes with resolution (and are we talking here about pixels or chosen time range) ?

Thanks for your help