How should you define the Loki URL when it is behind a proxy (Traefik)?
I have https://acme.com/loki/ready health check resolving through curl with no issues.
But when I use https://acme.com/Loki/ as the data source in Grafana I get a 404.
Note I have it working in Grafana with unproxied Loki, eg http://acme.com:3100, but if I curl that I get a 404 (http://acme.com:3100/ready works).
Seems to me that there is some transformation being applied by Grafana when validating data source that doesn’t work out when using an https URL with no port?
Enabling grafana proxy debug logging reveals that the URL path tested by Grafana for loki health is /loki/api/v1/label
. I had also chosen loki as a subpath in the proxy and stripped it before addressing loki server itself. Sadly on re-referral this stripped again resulting in a 404. Adjusting the proxy so that it does not strip the sub-path, or adjusting loki server config to specify a different http_path_prefix
fixes the issue.
Enabling grafana proxy debug to determine how the datasource was operating:
[log]
level = debug
[dataproxy]
logging = true