Hi
I have a client app that sends zipkin traces to Tempo in Kubernetets. The client seems to be sending proper traces to Tempo as the count for tempo_distributor_spans_received_total and tempo_ingester_traces_created_total are greater than zero. The problem comes when trying to view them. The Jaeger UI doesn’t show any tracing or any information, it always shows zero. The tempo-query logs show repeatedly the following message:
{"level":"error","ts":1618781397.1498656,"caller":"zap/logger.go:33","msg":"failed to flush Jaeger spans to server: write udp 127.0.0.1:54969->127.0.0.1:6831: write: connection refused","stacktrace":"github.com/uber/jaeger-client-go/log/zap.(*Logger).Error\n\tgithub.com /uber/jaeger-client-go@v2.23.1+incompatible/log/zap/logger.go:33\ngithub.com/uber/jaeger-client-go.(*remoteReporter).processQueue.func1\n\tgithub.com/uber/jaeger-client-go@v2.23.1+incompatible/reporter.go:288\ngithub.com/uber/jaeger-client-go.(*remoteReporter).processQueue\n\tgithub.com/uber/jaeger-client-go@v2.23.1+incompatible/reporter.go:298"}
There seem to be issues with jaeger-cleint-go and Tempo. I enabled port 6831 and also downgraded my client jager-client-go but the result is the same on binary and distributed.
I am able to see traces using Grafana. However I was hoping to be able to use Jaeger as it provides more information, we are using knative and there is no way to get a traceID and search for it.
Is there any work around for this or it is actually a bug?
That log message “failed to flush Jaeger spans to server” is just an error message regarding Jaeger Query sending its own tracing information somewhere and is not related to your problem.
You’re saying you are capable of finding traces with Grafana, but not Jaeger-UI? Can you share any other error messages are in the tempo-query/Jaeger logs? I’m guessing that this is just a configuration issue.
Hi Joe
Thanks for getting back to me regarding this issue. Yes, that’s right, as you said I am able to get traces using Grafana UI but not from Jaeger UI. I have done the following steps a few times but the results are the same, further more in addition to the error log mentioned in my first post there is no much in the tempo-query logs, except for a 404 error coming from the Jaeger UI on the browser.
1.- Launch tempo container using the tempo-local.yaml file from tempo repo.
docker run -d --rm -p 9411:9411 -p 3100:3100 --name tempo -v $(pwd)/tempo-
local.yaml:/etc/tempo-local.yaml \
--network docker-tempo \
grafana/tempo:latest \
-config.file=/etc/tempo-local.yaml
3.- Launch my application in a container.
4.- Get Trace ID from my app logs
5.- Launch Grafana in a separate container and I am able to see the Trace ID
6. Launch Jaeger UI (localhost:16686), enter same TraceID used in Grafana UI in the Search window, click search, nothing happens but if I hit Enter I get a 404.
As you said it might be a configuration issue but I haven’t found anything that works. I have gone thru the Tempo trouble shooting section and use the suggestions there but nothing.
The only error found on tempo is:
level=error ts=2021-04-27T17:40:51.478839871Z caller=frontend_processor.go:59 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = transport is closing"
As mentioned before my environment is Kubernetes but for the sake of simplicity and POC I am using docker.
It would be interesting to see if tempo_request_duration_seconds_count{route="api_traces_traceid"} is incrementing when you’re querying with Grafana/Tempo-Query.
I’m also suspicious of running the latest tag. Can you change your tags to 0.7.0 and see if we have any luck?
Changing to version 0.7.0 helped, I got the span!.
One thing when using the Jaeger UI is that when you enter the traceID and click the “Search” button nothing happens, but when you press “Enter” the trace shows up.
It would be nice to be able to see and query by services as Jaeger does.
Anyways thank you for your help, after all it was something simple but not obvious.
