Now when I update working Jaeger setup so that agent does not connect to Jaeger collector but to this Tempo service, the Jaeger agent log shows:
{“level”:“info”,“ts”:1632487767.9997396,“caller”:“grpc/builder.go:120”,“msg”:“Agent collector connection state change”,“dialTarget”:“dns:///tempo.jztest:14250”,“status”:“READY”}
But nothing appears in the S3 bucket. How do I verify that the setup is correct? There is nothing regarding S3 in the Tempo log…
And how long has Tempo been running when you checked S3? We keep traces in the ingesters for a while before we flush them to the backend. This is because the ingesters will batch multiple traces into a single block. How quickly blocks are flushed to the backend depends on the amount of traces you are ingesting, but with the default config this can take up to 1h.
I don’t know how without configuring other things. That requires several components configured and interconnected properly.
So I want to go step by step and setup Tempo first before trying to connect Grafana to it (possibly even without trying to connect Jaeger agent to it but since Jaeger agent seems happy I don’t have to worry now). It is always easier to debug the setup of one software than trying to get 2 or 3 programs correct at once.
It is running for several days now and the last thing in the Tempo’s log is still
level=info ts=2021-09-24T12:23:28.268725758Z caller=tempodb.go:359 msg=“compaction and retention enabled.”
S3 still empty (I would expect that it would create some directory structure or at least some empty file to verify the setup).
tempo_distributor_spans_received_total nor tempo_ingester_traces_created_total exist.
I added
distributor:
log_received_traces: true
under tempo: section in grafana-tempo-values.yaml but I don’t know if it was picked up. Should it be visible in some env variable?
There is still nothing in Tempo’s log… (I only update --reporter.grpc.host-port= option of the Jaeger agent and when reset to its default value (jaeger-collector) I can see spans in Jaeger UI, so I don’t think the problem is on Jaeger’s side).
If these metrics don’t exist and you are scraping your pods correctly then you are not receiving traces. Check your jaeger metrics on the agent to see if it thinks it’s successfully sending spans.
