Hi again
Direct API call:
We tried doing direct api calls, and it seems to work. Only difference in our api calls and the way grafana do them, are a) that we go through an Openshift route and b) that I authenticate using a token while Grafana currently is configured to use authentication with CA cert (see screenshot below). Given that we get the labels. and things work in Grafana 8.5.6, this is what I would have expected. For example we tried:
https://logging-loki-openshift-logging.apps.c03x.paas.corp.jyskebank.net/api/logs/v1/application/loki/api/v1/query_range?query={+log_type%3D"application"+}+|+json&start=1686811511525000000&end=1686815111525000000&limit=100&direction=backward
New datasource
I create the datasource using datasource provisioning (as done in the example in the Loki Operator documentation (see link in post above)). My current datasource definition looks like this:
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
url: https://thanos-querier.openshift-monitoring.svc.cluster.local:9091
access: proxy
basicAuth: false
withCredentials: false
isDefault: true
jsonData:
timeInterval: 5s
tlsSkipVerify: true
httpHeaderName1: "Authorization"
secureJsonData:
httpHeaderValue1: "Bearer ${PROMETHEUS_ACCESS_TOKEN}"
editable: false
- name: Loki - Application
isDefault: false
type: loki
access: proxy
url: https://${GATEWAY_ADDRESS}/api/logs/v1/application/
jsonData:
tlsAuthWithCACert: true
secureJsonData:
tlsCACert: ${GATEWAY_SERVICE_CA}
The environment variable is set in the deployment. Here I made a discovery. At some point in trying to get things to work, I have added an extra certificate to this environment variable. I did this because I got the following error while retrieving data (but not when retrieving labels):
Get “https://oauth-openshift.apps.c03x.paas.corp.jyskebank.net/oauth/authorize?approval_prompt=force&client_id=system%3Aserviceaccount%3Aopenshift-logging%3Alogging-loki-gateway&redirect_uri=https%3A%2F%2Flogging-loki-openshift-logging.apps.c03x.paas.corp.jyskebank.net%2Fopenshift%2Fapplication%2Fcallback%3Froute%3D%2Floki%2Fapi%2Fv1%2Fquery_range&response_type=code&scope=user%3Ainfo+user%3Acheck-access+user%3Alist-projects&state=I+love+Observatorium”: tls: failed to verify certificate: x509: certificate signed by unknown authority
It seems that a oauth flow is triggered when retrieving data. The strange thing is that the CA, for the certificate that fails, is present among the containers CA’s. As I understand it go should use these certificates.
In the 8.5.6 container it hasn’t been necessary to ad the extra certificate. Could my problems be related to this?
To be honest I have had trouble finding good detailed information regarding the different authentication methods in the datasources, so any link would be appreciated.
Proxy
We are accessing loki through a service internally on the Openshift cluster, so there shouldn’t be any proxy. But given the error above I suspect that Red Hat might be using Observatorium.
Screenshot of the datasource:
Do you know if there are any way to configure Grafana to logout all the request that the datasources do? I have already enable datasource logging, and plugin login and changed the loglevel to trace.
