Configuring Tempo as Grafana Data source

Grafana Tempo
1

Hello everyone, sorry for the bad english.

I’m trying to configure Tempo as a Grafana data source.

While it says the configuration is successful when I try to query something in “explore” menu it gives me this error.

failed to get trace with id: Status: 404 Not Found Body: 404 page not found

this is my tempo-local.yaml file

server:
 http_listen_port: 3200

# Distributor config block
distributor:

    # receiver configuration for different protocols
    # config is passed down to opentelemetry receivers
    # for a production deployment you should only enable the receivers you need!
    receivers:
        otlp:
            protocols:
                grpc:
                http:
        jaeger:
            protocols:
                thrift_http:
                grpc:
                thrift_binary:
                thrift_compact:

ingester:
  trace_idle_period: 10s               # the length of time after a trace has not received spans to consider it complete and flush it
  max_block_bytes: 1_000_000           # cut the head block when it hits this size or ...
  max_block_duration: 5m               #   this much time passes

compactor:
  compaction:
    compaction_window: 1h              # blocks in this time window will be compacted together
    max_block_bytes: 100_000_000       # maximum size of compacted blocks
    block_retention: 1h
    compacted_block_retention: 10m

storage:
  trace:
    backend: local                     # backend configuration to use
    block:
      bloom_filter_false_positive: .05 # bloom filter false positive rate.  lower values create larger filters but fewer false positives
      index_downsample_bytes: 1000     # number of bytes per index record
      encoding: zstd                   # block encoding/compression.  options: none, gzip, lz4-64k, lz4-256k, lz4-1M, lz4, snappy, zstd, s2
    wal:
      path: /tmp/tempo/wal             # where to store the the wal locally
      encoding: snappy                 # wal encoding/compression.  options: none, gzip, lz4-64k, lz4-256k, lz4-1M, lz4, snappy, zstd, s2
    local:
      path: /tmp/tempo/blocks
    pool:
      max_workers: 100                 # worker pool determines the number of parallel requests to the object store backend
      queue_depth: 10000

when I access the page http://my-tempo-address:3200 it gives me this error:

404 page not found

so… what am I doing wrong?

2

Hi @dvmorais!

Can you share the logs from Tempo? We may find something there that points us to the problem.

Does search work? Does it show any traces?

3 
level=info ts=2022-09-28T15:16:09.89874147Z caller=main.go:200 msg="initialising OpenTracing tracer"
level=info ts=2022-09-28T15:16:09.901877356Z caller=main.go:115 msg="Starting Tempo" version="(version=, branch=main, revision=752d9a90)"
level=info ts=2022-09-28T15:16:09.902855612Z caller=server.go:306 http=[::]:3200 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2022-09-28T15:16:09.907157297Z caller=frontend.go:43 msg="creating middleware in query frontend"
level=warn ts=2022-09-28T15:16:09.908181748Z caller=modules.go:181 msg="Worker address is empty in single binary mode. Attempting automatic worker configuration. If queries are unresponsive consider configuring the worker explicitly." address=127.0.0.1:9095
level=info ts=2022-09-28T15:16:09.908439014Z caller=worker.go:103 msg="Starting querier worker connected to query-frontend" frontend=127.0.0.1:9095
ts=2022-09-28T15:16:09Z level=info msg="OTel Shim Logger Initialized" component=tempo
level=info ts=2022-09-28T15:16:09.911900643Z caller=module_service.go:82 msg=initialising module=server
level=info ts=2022-09-28T15:16:09.912086176Z caller=module_service.go:82 msg=initialising module=store
level=info ts=2022-09-28T15:16:09.912516908Z caller=module_service.go:82 msg=initialising module=memberlist-kv
level=info ts=2022-09-28T15:16:09.912625102Z caller=module_service.go:82 msg=initialising module=usage-report
level=info ts=2022-09-28T15:16:09.912669757Z caller=module_service.go:82 msg=initialising module=ring
level=info ts=2022-09-28T15:16:09.912865341Z caller=ring.go:263 msg="ring doesn't exist in KV store yet"
level=info ts=2022-09-28T15:16:09.913086245Z caller=client.go:255 msg="value is nil" key=collectors/ring index=1
level=info ts=2022-09-28T15:16:09.912612235Z caller=module_service.go:82 msg=initialising module=overrides
level=info ts=2022-09-28T15:16:09.914476236Z caller=module_service.go:82 msg=initialising module=compactor
level=info ts=2022-09-28T15:16:09.91458421Z caller=tempodb.go:417 msg="polling enabled" interval=5m0s concurrency=50
level=info ts=2022-09-28T15:16:09.914865633Z caller=compactor.go:160 msg="enabling compaction"
level=info ts=2022-09-28T15:16:09.914943809Z caller=tempodb.go:391 msg="compaction and retention enabled."
level=info ts=2022-09-28T15:16:09.91505372Z caller=module_service.go:82 msg=initialising module=querier
level=info ts=2022-09-28T15:16:09.915270974Z caller=module_service.go:82 msg=initialising module=query-frontend
level=info ts=2022-09-28T15:16:09.915889764Z caller=module_service.go:82 msg=initialising module=distributor
ts=2022-09-28T15:16:09Z level=info msg="No sampling strategies provided or URL is unavailable, using defaults" component=tempo
level=info ts=2022-09-28T15:16:09.917257406Z caller=module_service.go:82 msg=initialising module=ingester
level=info ts=2022-09-28T15:16:09.917520569Z caller=ingester.go:328 msg="beginning wal replay"
ts=2022-09-28T15:16:09Z level=info msg="Starting GRPC server on endpoint 0.0.0.0:4317" component=tempo
ts=2022-09-28T15:16:09Z level=info msg="Starting HTTP server on endpoint 0.0.0.0:4318" component=tempo
level=warn ts=2022-09-28T15:16:09.917889783Z caller=rescan_blocks.go:23 msg="failed to open search wal directory" err="open /tmp/tempo/wal/search: no such file or directory"
level=info ts=2022-09-28T15:16:09.917966242Z caller=ingester.go:413 msg="wal replay complete"
level=info ts=2022-09-28T15:16:09.918241699Z caller=ingester.go:427 msg="reloading local blocks" tenants=0
level=info ts=2022-09-28T15:16:09.918493024Z caller=worker.go:179 msg="adding connection" addr=127.0.0.1:9095
level=info ts=2022-09-28T15:16:09.918702962Z caller=lifecycler.go:547 msg="not loading tokens from file, tokens file path is empty"
level=info ts=2022-09-28T15:16:09.918891672Z caller=lifecycler.go:576 msg="instance not found in ring, adding with no tokens" ring=ingester
level=info ts=2022-09-28T15:16:09.919361569Z caller=lifecycler.go:416 msg="auto-joining cluster after timeout" ring=ingester
level=info ts=2022-09-28T15:16:09.922098175Z caller=app.go:195 msg="Tempo started"

Seems like everything is working but the errors described above keep showing.

search in…? Grafana is not showing me any traces but if I access the /metrics it shows the tempo metrics as usual.

4

A 404 when accessing http://tempo:3200/ is expected. Have you been able to send traces? With a trace ID in hand, you might try accessing the API directly using that trace to confirm. /api/traces/<traceID> from the the docs might be useful. The logs look sane to me.

5

I… discovered what I was doing wrong and I feel a bit dumb for that…

I was pointing to the wrong port (9095, not 4317 that is the grpc default port).

And I discovered that I needed to enable search in grafana using this flag on tempo.yaml:

# Enable search functionality
search_enabled: true

anyway… thanks for the help guys.

1 Like
6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.