I’m collecting metrics with Prometheus, specifically histograms of outgoing requests from my service to various routes.
Here’s an metric example:
southbound_request_duration_seconds_bucket{le="0.05",target="api.token-machine.fra.co",method="GET",route="http://api.token-machine.fra.co/states",status_code="200",type="total"} 96
I wish to display in a grafana metrics by route & by latency percentile.
This is the query I wrote:
histogram_quantile(0.90, sum(rate(southbound_request_duration_seconds_bucket{marathon_app_path=~"$instance_path", route=~"$route", env="mars"}[1d])) by (route))
but for some reason i does not generate any data. If I replace
by (route) to by (le)
It generates data but every percentile appears 3 times (one per each route) but it does not display the actual route.
Please advise