Hi,
I have spent days now trying to get a proof of concept with Grafana Tempo and OpenTelemetry running with my .net core application and I have tried everything now but with no progress.
I have followed this guide (Getting started | Grafana Labs) and my docker images is up and running as they should. However it seems that my problem is that no traces whatsoever are exported by my Jaeger exporter to my tempo docker image running on port 6831 as in your guide. As you can see below is my setup of opentelemetry and the AddJaegerExporter() part is using localhost and 6831 as default values so I expected to be all set with that. However, the only thing my tempo docker image logs are showing is startup info and no incoming traces of any kind.
If I try a search in the querycontainer for a traceId I obviously get nothing since nothing seems to be transferred to the tempo container. However, I get an error in the logs saying:
{“level”:“error”,“ts”:1611087947.883616,“caller”:“zap/logger.go:33”,“msg”:“failed to flush Jaeger spans to server: write udp 127.0.0.1:37973->127.0.0.1:6831: write: connection refused”,“stacktrace”:"github.com/uber/jaeger-client-go/log/zap.
Tempo isn’t flushing traces when I search is it? Where 127.0.0.1:37973 comes from I have no Idea. My application is configured to send traces to tempo:6831. What have I missed?
This is the local port your application is using to try and send data to Tempo.
failed to flush Jaeger spans to server: write udp 127.0.0.1:37973->127.0.0.1:6831: write: connection refused
This seems to indicate that your application is not properly configured to send its trace data to Tempo. it is trying to connect to 127.0.0.1 (its own IP). I do not know a lot about configuring the OTEL .net client, but I’d review those documents.
{“level”:“error”,“ts”:1611087947.883616,“caller”:“zap/logger.go:33”,“msg”:“failed to flush Jaeger spans to server: write udp 127.0.0.1:37973->127.0.0.1:6831: write: connection
This log line from the query container is OK. The Query container traces itself and this is reporting that it is unable to send spans anywhere… it should not affect the querying of your application traces.
Yes my application is also running in docker. The address of tempo being used is localhost:6831 as default. Setting samling to 1? Not really sure what you mean there.
If your application is running inside docker the correct address would be tempo:3100. (localhost would refer to the same container which would be wrong).
Hmm tempo:3100 is not valid to use for jaeger exporter configuration in application though. And default port is 6831. But I guess this is where I get it all wrong somehow. Do I need a collector/agent container as well besides the tempo container?
