Elasticsearch data source working partially after migration to HTTPS

Grafana Elasticsearch
1

Hi all,

Elasticsearch servers in my organization have been recently migrated to use https instead of regular http. Unfortunately Grafana seems to have some issues with handling it.

I’m able to add/edit the datasource based on https elastic in the Data Sources management page (screenshot below). I get a green message “Index OK. Time field name OK.”
image

Unfortunately, the dashboards do not display any data. I keep geeting http 403 responses all the time. Previously defined dashboards don’t work, new ones created from scratch give the same result.

image

I’m using the “Server (default)” access mode. The “Browser” mode is not an option for my scenario (and it behaves the same way).
Tested on Grafana v5.3.2 and v5.2.2.

Is there any option to get Elastic working with https? Other client tools are working fine after the migration. Thank you for help!

2

Have you tried using the With CA Cert or Skip TLS verify auth settings?

3

I have the same issue.

I am able to create the datasource successfully using

  • https
  • Basic Auth
  • Skip TLS Verification enabled

I know the datasource is talking to es, because the anonymous user fails, but the user I setup for this succeeds.

Authorized User

image
image1226×770 80.2 KB

Anonymous User

image
image1304×776 99.1 KB

I tested the queries outside of Grafana with curl. They work outside of Grafana.

request

curl -u grafana:grafana --insecure -H 'Content-Type:application/json' "https://<redacted>/.monitoring-es-6-2018.11.19/_mapping"

response

< HTTP/1.1 200 OK
< Date: Mon, 19 Nov 2018 21:23:40 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 10682
< Connection: keep-alive
< access-control-allow-credentials: true
<
{".monitoring-es-6-2018.11.19":{"mappings":{"doc":{"dynamic":"false",
. . .

This is what grafana is showing in the log when I try to use the datasource.

grafana log entry

t=2018-11-19T15:10:12-0600 lvl=info msg="Request Completed" logger=context userId=5 orgId=83 uname=<redacted> method=GET path=/api/datasources/proxy/234/.monitoring-es-6-2018.11.19/_mapping status=403 remote_addr="<redacted>, <redacted>, <redacted>" time_ms=274 size=285 referer=http://<redacted>/grafana/datasources/edit/234
4

I’m afraid you have slightly similar problems. For me to be able to investigate this I would need to know more:

  • Elasticsearch versions?
  • How you have configured https/SSL, kinds of certs etc
  • Use curl on your Grafana server to retrieve the mappings - does that work?
  • Use curl and do a POST for a search to verify if that works as expected from your local machine and/or grafana server

Thanks

5

In my case the issue turned out not to be related to https.

Our team enabled some extended security features in Elastic (including but not limited to https) and the real issue was caused by Grafana proxy passing CORS headers (issue 13328). After upgrading to 5.3.4 it’s working fine. Thank you for your help!

1 Like
6

@mefraimsson

ElasticSearch Version 6.5.0

SSL Private CA, Self-Signed Cert (i.e. Corporate Man-in-the-Middle)

Grafana Server Api works

curl http://localhost:38083/api/frontend/settings
{"alertingEnabled":true,"allowOrgCreate":false,"appSubUrl":"/grafana",
. . .

Search works

curl -u grafana:grafana -v --insecure -H 'Content-Type:application/x-ndjson' -XPOST --data-binary "@requests" "https://<redacted>/.monitoring-es-6-2018.11.19/_msearch?pretty"

< HTTP/1.1 200 OK
< Date: Tue, 20 Nov 2018 13:47:30 GMT
< Content-Type: application/json; charset=UTF-8
< Content-Length: 106409
< Connection: keep-alive
< access-control-allow-credentials: true
<
{
  "responses" : [
    {
      "took" : 2654,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 15872310,
    "max_score" : 0.0,
    "hits" : [ ]
  },
. . .

It looks like I am going to have to upgrade to the latest version. This is not a big deal, it just takes our org a lot of time to get it through the process. I was hoping there might be something that I could do in the interim.

Thanks,
Chris

7

The above search curl command was sent from the Grafana server or from your local computer?

8

sorry. Grafana server.

9

@mefraimsson
I just upgraded to 5.3.4 and the elasticsearch datasource is working.

Thanks,
Chris

1 Like