Tempo doesn't take any traces. Nothing in logs

I am trying to run tempo-distributed in kubernetes with minio s3 storage on virtual host is the same cloud.

custom.yaml file to apply

    backend: s3
      access_key: 'acc_key'
      secret_key: 'secret_key'
      bucket: 'echo-test'
      endpoint: 'endpoint with minio on 9000 port (API)'
      insecure: true
      enabled: true
      enabled: true
    enabled: true
      enabled: true
    enabled: true

  enabled: false

  # -- Log level. Can be set to trace, debug, info (default), warn, error, fatal, panic
  logLevel: debug
    # -- Enable to log every received trace id to help debug ingestion
    # -- WARNING: Deprecated. Use log_received_spans instead.
    log_received_traces: true
    # -- Enable to log every received span to help debug ingestion or calculate span error distributions using the logs
      enabled: true
      include_all_attributes: true
      filter_by_status_error: true

So, after deploy in k8s all pods green and seems everything is ok.
But, when i try to push some traces on distributor nothing happens and no single-tenant blocks appear in s3
Curl post send partial success, but also nothing

Hi @abellindsey. Are ingested spans being logged in the distributor? Or not even that?

If nothing is logged in the distributor, it’s likely that your exporter is failing to send traces to Tempo.

You mean that traces dont reach distributor?
I have an example curl request for OTLP http receiver like that

and it shows partial success
Partial success seems to be a distributor response?

Also, logs about accepted traces are not found either in the distributor or in the ingester.

Tempo uses OTel components in the distributor to receive and parse traces. That response is indicating that some data was rejected by the OTel receiver. Given that you’re sending a single trace, it’s rejecting everything.

Verify that the payload adheres to the OTLP spec. For example, the trace ID is not following the spec.

1 Like

I repeat this test scenario for OTLP with traces. Also partial success. I changed nothing in this example.

Could it be misunderstanding between tempo and OTLP/HTTP spec?

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.