Unable to use HTTP Headers for HTTPS k8s prometheus datasource

Grafana Configuration
1

Hi All,

I hope someone can help me, I’m going a bit mad trying to work this out.

I have a Prometheus in a Kubernetes cluster that I am trying to hook up to our Grafana that runs external to this.

To access it I need to connect via https, but skip tls verify, and pass the correct host header.

If I curl from the Grafana server I get the proper response.
But when I configure the datasource to match these options I get a 503 from the kubernetes ingress, which is the response you get when you don’t pass the right Host header.

I’ve tried configuring the datasource via the GUI and the API but it makes no difference.

I’ve done all sorts of debugging and I am sure the custom header is not being honoured.

Have I missed some obvious step? This feels like it should “just work”.

I am using the latest RPM release of Grafana server - v7.1.0 (8101355285)

Configs below (had to replace some IPs with < my https endpoint > because the forum thinks I’m posting links).

Many thanks,
Mike

Without Host header:

#  curl -k < my https endpoint >
<html>
<head><title>503 Service Temporarily Unavailable</title></head>
<body>
<center><h1>503 Service Temporarily Unavailable</h1></center>
<hr><center>openresty/1.15.8.2</center>
</body>
</html>

With Host header:

#  curl -k -H Host:my.prometheus.host.com < my https endpoint >
<a href="/graph">Found</a>.

Datasource config:

{
  "id": 4,
  "orgId": 1,
  "name": "FOO",
  "type": "prometheus",
  "typeLogoUrl": "",
  "access": "proxy",
  "url": "https://10.0.0.200",
  "password": "",
  "user": "",
  "database": "",
  "basicAuth": false,
  "basicAuthUser": "",
  "basicAuthPassword": "",
  "withCredentials": false,
  "isDefault": false,
  "jsonData": {
    "httpHeaderName1": "Host",
    "tlsSkipVerify": true
  },
  "secureJsonFields": {
    "httpHeaderValue1": true
  },
  "version": 5,
  "readOnly": false
}
2

Hello @biz812g , did you resolve this?
I hit the exact same problem where I need pass a custom header with “Host: prometheus…”.
Thank you!