Hi
I am trying to run tempo-distributed in kubernetes but cant find traces in tempo. Tempo search shows me a list on traceIDs, but if I click on any of these - Tempo shows “Query error: Query data error”
I am collecting traces from knative and send them to http://tempo-distributed-distributor.<namespace>.svc.cluster.local:9411/api/v2/spans address, I hope this is a right service. Similar path was working fine when I trying it with single-binary deploy of tempo.
I forward port 3100 and on page http://localhost:3100/memberlist and I see no tempo-distributed-gossip-ring in " Memberlist Cluster Members" list. Can it be a problem? Service is deployed in cluster:
kubectl get svc |grep tempo
tempo-distributed-compactor ClusterIP 10.3.16.180 <none> 3100/TCP 76m
tempo-distributed-distributor ClusterIP 10.3.16.175 <none> 3100/TCP,9095/TCP,9411/TCP 76m
tempo-distributed-gossip-ring ClusterIP None <none> 7946/TCP 76m
tempo-distributed-ingester ClusterIP 10.3.16.243 <none> 3100/TCP,9095/TCP 76m
tempo-distributed-memcached ClusterIP 10.3.16.133 <none> 11211/TCP,9150/TCP 76m
tempo-distributed-querier ClusterIP 10.3.16.9 <none> 3100/TCP,9095/TCP 76m
tempo-distributed-query-frontend ClusterIP 10.3.16.153 <none> 3100/TCP,9095/TCP,16686/TCP,16687/TCP 76m
tempo-distributed-query-frontend-discovery ClusterIP None <none> 3100/TCP,9095/TCP,16686/TCP,16687/TCP 76m
Here it is a log from distributor pod:
level=info ts=2022-07-05T11:04:41.810372356Z caller=main.go:191 msg="initialising OpenTracing tracer"
ts=2022-07-05T11:04:41Z level=info msg="OTel Shim Logger Initialized" component=tempo
level=info ts=2022-07-05T11:04:41.833055484Z caller=main.go:106 msg="Starting Tempo" version="(version=, branch=HEAD, revision=d3880a979)"
level=info ts=2022-07-05T11:04:41.834627113Z caller=server.go:260 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2022-07-05T11:04:41.836991758Z caller=memberlist_client.go:394 msg="Using memberlist cluster node name" name=tempo-distributed-distributor-74ff474ff9-bt88z-fbc02a9a
level=info ts=2022-07-05T11:04:41.838831993Z caller=module_service.go:64 msg=initialising module=server
level=info ts=2022-07-05T11:04:41.839305902Z caller=module_service.go:64 msg=initialising module=overrides
level=info ts=2022-07-05T11:04:41.839403003Z caller=module_service.go:64 msg=initialising module=memberlist-kv
level=info ts=2022-07-05T11:04:41.839504005Z caller=module_service.go:64 msg=initialising module=ring
level=info ts=2022-07-05T11:04:41.840645327Z caller=ring.go:272 msg="ring doesn't exist in KV store yet"
level=info ts=2022-07-05T11:04:41.840781529Z caller=module_service.go:64 msg=initialising module=distributor
level=info ts=2022-07-05T11:04:41.840962333Z caller=app.go:284 msg="Tempo started"
ts=2022-07-05T11:04:41.862999548Z caller=memberlist_logger.go:74 level=warn msg="Failed to resolve tempo-distributed-gossip-ring: lookup tempo-distributed-gossip-ring on 10.3.16.10:53: no such host"
ts=2022-07-05T11:04:43.041632178Z caller=memberlist_logger.go:74 level=warn msg="Failed to resolve tempo-distributed-gossip-ring: lookup tempo-distributed-gossip-ring on 10.3.16.10:53: no such host"
ts=2022-07-05T11:04:46.353917048Z caller=memberlist_logger.go:74 level=warn msg="Failed to resolve tempo-distributed-gossip-ring: lookup tempo-distributed-gossip-ring on 10.3.16.10:53: no such host"
level=info ts=2022-07-05T11:04:52.948855119Z caller=memberlist_client.go:542 msg="joined memberlist cluster" reached_nodes=2