Grafana Tempo ingester error searching block

Grafana Tempo
1

Hi,
When trying to search for a trace in tempo datasource
I see these errors in the ingester.

level=warn ts=2023-03-27T08:56:08.259060424Z caller=grpc_logging.go:43 method=/tempopb.Querier/SearchRecent duration=48.863573ms err="error searching block [fa1b3328-2733-40c4-b38c-aec5 │
│ 76ad0744 112]: searchRaw next failed: join iterator peek failed: join iterator peek failed: [rs.ils.Spans.Name] context done context canceled" msg=gRPC                                   │
│ level=error ts=2023-03-27T08:56:10.821631062Z caller=instance_search.go:202 msg="error searching local block" blockID=fa1b3328-2733-40c4-b38c-aec576ad0744 block_version=vParquet err="er │
│ ror searching block [fa1b3328-2733-40c4-b38c-aec576ad0744 112]: rawToResults next failed: join iterator peek failed: [RootSpanName] context done context canceled"                        │
│ level=warn ts=2023-03-27T08:56:10.821697457Z caller=grpc_logging.go:43 method=/tempopb.Querier/SearchRecent duration=48.306394ms err="error searching block [fa1b3328-2733-40c4-b38c-aec5 │
│ 76ad0744 112]: rawToResults next failed: join iterator peek failed: [RootSpanName] context done context canceled" msg=gRPC                                                                │
│ level=warn ts=2023-03-27T08:56:35.571619864Z caller=grpc_logging.go:43 method=/tempopb.Querier/SearchTagValues duration=37.960649ms err="unexpected error searching head block (fa1b3328- │
│ 2733-40c4-b38c-aec576ad0744): unexpected error searching tag values (status.code): error searching block [fa1b3328-2733-40c4-b38c-aec576ad0744 124]: unexpected error searching special t │
│ ags: iter.Next failed: [rs.ils.Spans.StatusCode] context done context canceled" msg=gRPC                                                                                                  │
│ level=error ts=2023-03-27T08:56:39.145904316Z caller=instance_search.go:202 msg="error searching local block" blockID=fa1b3328-2733-40c4-b38c-aec576ad0744 block_version=vParquet err="er │
│ ror searching block [fa1b3328-2733-40c4-b38c-aec576ad0744 111]: searchRaw next failed: join iterator peek failed: join iterator peek failed: [rs.ils.Spans.Name] context done context can │
│ celed"                                                                                                                                                                                    │
│ level=warn ts=2023-03-27T08:56:39.145970686Z caller=grpc_logging.go:43 method=/tempopb.Querier/SearchRecent duration=51.964143ms err="error searching block [fa1b3328-2733-40c4-b38c-aec5 │
│ 76ad0744 111]: searchRaw next failed: join iterator peek failed: join iterator peek failed: [rs.ils.Spans.Name] context done context canceled" msg=gRPC                                   │
│ level=error ts=2023-03-27T08:56:41.467048545Z caller=instance_search.go:202 msg="error searching local block" blockID=fa1b3328-2733-40c4-b38c-aec576ad0744 block_version=vParquet err="er │
│ ror searching block [fa1b3328-2733-40c4-b38c-aec576ad0744 109]: rawToResults next failed: join iterator peek failed: [RootServiceName] context done context canceled"                     │
│ level=warn ts=2023-03-27T08:56:41.467119288Z caller=grpc_logging.go:43 method=/tempopb.Querier/SearchRecent duration=51.107724ms err="error searching block [fa1b3328-2733-40c4-b38c-aec5 │
│ 76ad0744 109]: rawToResults next failed: join iterator peek failed: [RootServiceName] context done context canceled" msg=gRPC                                                             │
│ level=error ts=2023-03-27T08:56:43.603995624Z caller=instance_search.go:202 msg="error searching local block" blockID=fa1b3328-2733-40c4-b38c-aec576ad0744 block_version=vParquet err="er │
│ ror searching block [fa1b3328-2733-40c4-b38c-aec576ad0744 109]: searchRaw next failed: join iterator peek failed: join iterator peek failed: [rs.ils.Spans.Name] context done context can │
│ celed"                                                                                                                                                                                    │
│ level=warn ts=2023-03-27T08:56:43.60406078Z caller=grpc_logging.go:43 method=/tempopb.Querier/SearchRecent duration=50.458641ms err="error searching block [fa1b3328-2733-40c4-b38c-aec57 │
│ 6ad0744 109]: searchRaw next failed: join iterator peek failed: join iterator peek failed: [rs.ils.Spans.Name] context done context canceled" msg=gRPC                                    │
│ level=error ts=2023-03-27T08:56:44.320512744Z caller=instance_search.go:202 msg="error searching local block" blockID=fa1b3328-2733-40c4-b38c-aec576ad0744 block_version=vParquet err="er │
│ ror searching block [fa1b3328-2733-40c4-b38c-aec576ad0744 109]: searchRaw next failed: join iterator peek failed: join iterator peek failed: [rs.ils.Spans.StatusCode] context done conte │
│ xt canceled"

I manage to lookup traces when i use the exact TraceID
distributed deployment so search query is enabled by default.
without searching traces logs doesn’t show any indication of corrupted block,
What are my possibility to recover from this block and be able to use search for traces?

Thank you

│     app.kubernetes.io/version: 2.0.1                                                                                                                                                      │
│     helm.sh/chart: tempo-distributed-1.2.7
2

The only error I’m seeing is “context cancelled”. This is likely occurring b/c the search limit is being hit in the query frontend and it’s cancelling all other jobs.

Is this query returning successfully?

1 Like
3

Hi, I’ve just discovered that the query is actually working if you use the main “table” panel.
But not when using the “traces” panel.

and since it’s on beta i assume it’s safe to ignore.

thanks anyway :slight_smile:

4

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.