Bad Prometheus graphs starting with 4.6.2

Grafana Prometheus
1

4.6.1 on the left, 4.6.2 on the right

2
22537×1443 355 KB

Could it be that this is due to "tsdb.HandleRequest() error invalid value type" · Issue #9777 · grafana/grafana · GitHub ? I believe I have the step interval matched. I’m working on updating Grafana in the Istio project. Our config is here:

https://github.com/istio/istio/tree/master/mixer/deploy/kube/conf
https://github.com/istio/istio/blob/master/install/kubernetes/addons/grafana.yaml

Prom config is here: https://github.com/istio/istio/blob/master/install/kubernetes/addons/prometheus.yaml

Thanks!

2

what is your scrape in interval setting? and what is your grafana step setting and rate scalar?

3

The data source scrape interval is 5s. Min step is ‘auto’. Not sure about rate scalar?
The dashboard json is here: https://github.com/istio/istio/blob/master/mixer/deploy/kube/conf/grafana-dashboard.json

Same dashboard json with 4.6.1 and 4.6.2. Only change is setting the scrape interval to match prometheus, as that setting is new.

I did some searching and am experimenting with the ‘Resolution.’ Changing it from 1/2 to 1/1 seems to make a big improvement.