What happened to tempo older helm charts?

Grafana Tempo
1

Hi

I added tracing to our application using opencensus/zipkin and tempo as backend over a month ago. It was working fine with tempo:0.7.0. It is time to merge that code into master, so I tested with the latest helm chart version 0.7.4 (tempo:1.0.1) but doesn’t work anymore. Getting the following error on single binary

connect: connection refused

and on distributed

failed to send the request: Post "http://tempo-distributed-distributor.tempo.svc.cluster.local:9411/api/v2/spans": dial tcp 10.99.163.220:9411: i/o timeout
failed to send the request: Post "http://tempo-distributed-distributor.tempo.svc.cluster.local:9411/api/v2/spans": context deadline exceeded

I tried to go back to tempo:0.7.0 but seems to be gone. The installation goes through but the container doesn’t get created as it fails to pull the image. Not sure which helm chart version installs tempo:0.7.0 but I have tried 0.7.3, 0,7.2… 0.6.0 but all of them fail, only 0.7.4 works which installs tempo:1.0.1

helm install tempo grafana/tempo --version=0.7.0 --set traces.zipkin=true -n tempo

What happened to the older charts?

2

Hi ragamx,

With the single-binary helm chart --set traces.zipkin=true will not do anything, this is not a supported parameter. In fact, the single-binary helm chart, strangely enough, does not have an option to enable the Zipkin receiver: receivers in values.yaml. Did this work for you in the past?
It should be fairly straightforward to add an option to enable the Zipkin receiver as well. It currently just defaults to Jaeger only :man_shrugging:

To run Tempo 0.7.0 I can install version 0.7.1 of the single-binary helm chart, pulling the images works on my end. The image still exists on docker hub.

helm upgrade tempo grafana/tempo --version=0.7.1

But I’d recommend running a recent version as we have been adding a ton of improvements :smiley:

Regarding your error with the distributed helm chart: dial tcp and timeout sounds like your distributor is not accepting push requests. Can you check you have at least one healthy distributor?

3

Hi @koenraad

Thanks for you reply, you’re right the helm chart is still available, for some reason it was failing for me trying to installing it on my k8s. Also yes, it looks like the single binary may not have the receivers: option so all the ports should be available, including 9411, right? I thought my problem was the version but even going back to that one didn’t help. On single binary I keep getting connection refused trying to access port 9411.

kubectl get svc -n tempo
NAME    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                                                                    AGE
tempo   ClusterIP   10.110.221.103   <none>        3100/TCP,16686/TCP,6831/UDP,6832/UDP,14268/TCP,14250/TCP,9411/TCP,55680/TCP,55681/TCP,4317/TCP,55678/TCP   19m

if I do a

curl -v http://10.110.221.103:3100/

I get a response, the same with 16686, 6831, 6832, 14268, 14250. But port 9411 gives a connection refused. I have used tempo in the past and it was very straight forward to use port 9411 to sends spans, no special configuration it really puzzles me, I don’t know what has changed. Also when I do

kubectl edit configmap tempo -n tempo

I only see jaeger, not zipkin, could that be the problem? The single binary doesn’t automatically enables port 9411?

    distributor:
      receivers:
        jaeger:
          protocols:
            grpc:
              endpoint: 0.0.0.0:14250
            thrift_binary:
              endpoint: 0.0.0.0:6832
            thrift_compact:
              endpoint: 0.0.0.0:6831
            thrift_http:
              endpoint: 0.0.0.0:14268

The problem with the distributed version was exactly that, the 9411 was not enabled. I did it thru
--set traces.zipkin=true but for single binary doesn’t seem to work.

Thanks.

4

Yeah, you will have to add Zipkin to the receivers section, otherwise Tempo will not listen on this port.

Using the helm chart you can set this as a value using tempo.receivers, the default is defined here: helm-charts/values.yaml at 160187eed9a5524eac9e73de145bd89a5095f95b · grafana/helm-charts · GitHub
If you are up to creating a PR, I think it would make sense to add it as an option so you could do something like --set tempo.receivers.zipkin=true.

The Tempo stateful set and the service apparently already have port 9411 enabled. Which is a bit weird without the corresponding Zipkin receiver… :thinking:

5

I will try to create the PR although I am not familiar with the code, I’d need to start getting into it, not sure how long it would take me.
Thanks

6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.