Grafana 8 variables stopped working due to the HTTP method change

Grafana Configuration
, , ,
1

Background

We are trying to migrate from Grafana 7.5.13 to 8.4.1 in our k8s cluster (deployed via official Grafana helm chart).

We have a bunch of dashboards already defined for cluster monitoring and they are relying on dynamic variables obtained via Prometheus queries.

Problem

Once we swap the image to Grafana 8, dynamic variables are not working due to the lack of response from datasource proxy.

One of the cases - we need to have variable obtained via prom query:

label_values(kube_pod_info,namespace)

In grafana 7 variable queries used GET:

GET https://<grafana7>/api/datasources/proxy/1/api/v1/series?match%5B%5D=kube_pod_info&start=1645620104&end=1645620404

In grafana 8 variable queries seem to be using POST:

POST https://<grafana8>/api/datasources/proxy/1/api/v1/series
form data: match%5B%5D=kube_pod_info&start=1645598928&end=1645620528

For some reason this POST hangs indefintely without respose

Question

  1. Is it possible to configure grafana 8 to use legacy GET when querying variable values?
  2. If #1 is impossible - what can be causing this issue, and how this can be resolved?

Initial debug from our end:

inside promehteus-server container (both curl-s work OK):

curl localhost:9090/api/v1/series?match%5B%5D=kube_pod_info&start=1645620104&end=164

curl -XPOST localhost:9090/api/v1/series --data-raw 'match%5B%5D=kube_pod_info&start=1645601212&end=1645622812'

inside grafana container (all below wget-s work OK, returning valid prometeus API payloads):

# GET full cluster FQDN
wget -O- http://prometheus-server.monitoring.svc.cluster.local/api/v1/series?match%5B%5D=kube_pod_info&start=1645620104&end=164
# GET short cluster FQDN
wget -O- http://prometheus-server.monitoring/series?match%5B%5D=kube_pod_info&start=1645620104&end=164
# GET grafana proxy endpoint
wget -O- --header='X-User-Name: <my email>' http://grafana.staged-by-discourse.com/api/datasources/proxy/1/api/v1/series?match%5B%5D=kube_pod_info&start=1645620104&end=164
# POST full cluster FQDN
wget -O- --post-data='match%5B%5D=kube_pod_info&start=1645601212&end=1645622812' http://prometheus-server.monitoring.svc.cluster.local/api/v1/series
# POST short cluster FQDN
wget -O- --post-data='match%5B%5D=kube_pod_info&start=1645601212&end=1645622812' http://prometheus-server.monitoring/api/v1/series
# POST grafana proxy endpoint
wget -O- --header='X-User-Name: <my email>' --post-data='match%5B%5D=kube_pod_info&start=1645601212&end=1645622812' http://grafana.staged-by-discourse.com/api/datasources/proxy/1/api/v1/series
2

Issue is gaining traction in Grafana 8 variables stopped working due to the HTTP method change · Issue #45817 · grafana/grafana · GitHub and I will continue there. Once we’ll find a way to fix it I’ll post the final solution here as well.