Is it possible to send traces from the otel collector to the cloud base Tempo service via a reverse proxy

Grafana Tempo
1

Hi guys,

I’m currently trying to send traces to Tempo Grafana Cloud from OpenTelementry. The simple topology will be like this:
Otel-Collector → Reverse Proxy (which currently we use Caddy) → Tempo Grafana Cloud.
Currently we can send to Tempo Cloud directly from Otel-Collector, but when adding the reverse proxy, we got 502 EOF from Server (I think it’s from Grafana Cloud), sometime I got 404 on different Caddy config.
So my question is that possible to send traces from Otel-Collector via Reverse proxy and to Grafana Cloud?
We can do this with Loki, but Tempo endpoint seems different, will it be tempo-us-central1.grafana.net:443 or https://tempo-us-central1.grafana.net:443? (we tried with both grpc and http but not work yet)
Thank you.

2

Hi @nhatsangvn. Are there any differences on how authentication is handled between the OTel collector and with the reverse proxy? Make sure basic auth is correctly configured when using Caddy and that the authorization header is forwarded correctly.

Traces are sent via gRPC (HTTP/2) to Grafana Cloud.

3

Hi Mariorodriguez,

I think the authentication part is good because we still can forward Logs to Loki Grafana Cloud (which uses HTTPs) via the reverse proxy. Maybe the problem relating to the gRPC, I’m also asking on Caddy (the reverse proxy) forum for this issue, not sure if any customization on gRPC request.

We got some logs from the reverse proxy, not sure if it’s related to Grafana Cloud or not:
`{“level”:“error”,“ts”:1656244426.5726256,“logger”:“http.log.error.log2”,“msg”:“net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02F"”,“request”:{“remote_ip”:“172.23.0.3”,“remote_port”:“56704”,“proto”:“HTTP/2.0”,“method”:“PRI”,“host”:“”,“uri”:“*”,“headers”:{}},“duration”:0.10876762,“status”:502,“err_id”:“mqw96e6z4”,“err_trace”:“reverseproxy.statusError (reverseproxy.go:1196)”}

{“level”:“error”,“ts”:1656244426.5728211,“logger”:“http.log.access.log2”,“msg”:“handled request”,“request”:{“remote_ip”:“172.23.0.3”,“remote_port”:“56704”,“proto”:“HTTP/2.0”,“method”:“PRI”,“host”:“”,“uri”:“*”,“headers”:{}},“user_id”:“”,“duration”:0.10876762,“size”:0,“status”:502,“resp_headers”:{“Server”:[“Caddy”]}}’

Thank you,
Sang.

4

Thank you Mariorodriguez,

We found the solution, turns out that some options on our Caddy server need to be “tweaked”.
We added the code (for anyone have a same errror and come to this forum)

{
    servers {
        protocol {
            # We need to enable h2c in order to use grpc without TLS.
            # ref: https://caddy.community/t/caddy-grpc-h2c-passthrough/11780
            allow_h2c
        }
    }
}

this will allow Otel-Collector → Caddy with “plaintext HTTP/2” before forward to Grafana Cloud with https://.

1 Like
5

Awesome! Glad that it finally worked. Thanks for posting the solution.

6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.