Local kubernetes setup of open telemetry & tempo

[satwikareddy007]

Hi,
My aim is to display traces on grafana dashboard. So these are the steps I’m following.

1. An application instrumented for tracing is running on my kubernetes setup
2. Application sends data directly to an OpenTelemetry Collector (agent)
3. The agent then forwards the data to Collector(s) that receive data from multiple
   agents.
4. The Collector then sends the data to the appropriate backend, which is tempo in this
   case
   I’m stuck at 4th step. Collector is unable to push traces to tempo.

The configuration for exporter in collector is -
exporters:
otlp:
endpoint: tempo.default.svc.cluster.local:55680
insecure: true

The logs of collector pod show this -
2021-05-11T10:40:29.776Z info exporterhelper/queued_retry.go:314 Exporting failed. Will retry the request after interval. {“kind”: “exporter”, “name”: “otlp”, “error”: “failed to push trace data via OTLP exporter: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.105.244.36:55680: connect: connection refused"”, “interval”: “5.52330144s”}
Attaching the output screenshot

10.105.244.36 - tempo service cluster IP

I’m deploying tempo through helm chart - helm install tempo grafana/tempo
And I can see that 55680 port is open. Attaching screenshot of services too.

I’m a beginner. I’m not sure where it is going wrong.

[joeelliott]

Connection refused suggests that the port is not open or not accessible to the sender. Can you share your Tempo config?

This will require some basic network debugging like:

• Exec into the Tempo pod and attempt to open the local port 55680
• Exec into the collector pod and attempt to open port tempo.default.svc.cluster.local:55680

[satwikareddy007]

Hi @joeelliott ,

Thank you for the response.
I’m installing tempo through - helm install tempo grafana/tempo
I didn’t pass any configurations using values.html. Can that be a problem?

[satwikareddy007]

I did exec into my tempo pod. It is showing connection refused.

/ # busybox-extras telnet localhost 55680
telnet: can’t connect to remote host (127.0.0.1): Connection refused

[joeelliott]

By default it appears that all trace ingestion is disabled:

grafana/helm-charts/blob/main/charts/tempo-distributed/values.yaml#L318

Perhaps try something like:

helm upgrade --set traces.otlp.grpc=true --install tempo grafana/tempo-distributed

[satwikareddy007]

I tried - helm upgrade --set traces.otlp.grpc=true --install tempo grafana/tempo-distributed

When I give trace id on grafana UI, it returns nothing. I was checking logs for tempo pods.Attached are the logs for tempo-distributed-querier pod

[joeelliott]

Definitely concerned about the errors connecting to from querier → query-frontend. Tempo would be unable to answer any queries if this connection didn’t work.

We recently added the ability to just dump every trace that gets into Tempo. Perhaps this could help:
https://localhost:3000/t/tempo-unable-to-handle-32-digits-long-traceids/47400/8?u=joeelliott

This is the address the field that has to be set correctly for the querier to connect to the frontend. You could double check this as well as exec into your querier pod and see if you can connect to the frontend.

querier:
    frontend_worker:
        frontend_address: ???