Exporting Traces to Tempo | .Net7 + OpenTelemetry + Grafana Agent >> Tempo

[angeloferreira]

Hello community!

I am trying to set up a .Net 7 App that implements OpenTelemetry SDK to export its traces and make them visible on Grafana.

To make this work this is what I have understood so far - and please validate my understanding of it:

1. The .Net7 App generates traces through the use of OpenTelemetry SDK which needs to be configured to point to some Grafana Agent (mine runs on docker)
2. Grafana Agent needs to be configured to post the metrics to Tempo, with the right URL and user and password combination which I got from my Grafana Cloud area by checking on Tempo.
3. Tempo will gather the traces and make them available for consultation through Grafana.

Grafana Agent Docker container is configured this way - please validate
docker-compose.yaml - partial

(...)
  grafana-agent:
    image: grafana/agent:latest
    command: "-config.file=/etc/agent-config.yaml"
    volumes:
      - ./APMConfig/agent-config.yaml:/etc/agent-config.yaml

agent-config.yml

traces:
  configs:
  - name: default
    remote_write:
      - endpoint: tempo-prod-08-prod-eu-west-3.grafana.net:443
        basic_auth:
          username: <myTempoUsername>
          password: <myTempoPassword>
    receivers:
      otlp:
        protocols:
          grpc:

And I have setup OpenTelemetry to use the Grafana Agent from Docker which I would expect with the details I have configured on agent-config.yml would be able to send data to Tempo, but it isn’t doing so. Please consider the Grafana Agent docker container logs attached.

I don’t see any errors on the logs but no traces on Grafana Cloud either!

Any thoughts on what might be missing on this config?
Or have I misunderstood any part of the setup and I am missing something?

Thank you!

[mdisibio]

Hi, thank for you for the thorough information about your configuration and understanding. I have a working example repo that may be helpful. It contains working example for dotnet 6, otel sdk, agent, and grafana cloud. Not sure if there are breaking changes for dotnet 7, hopefully it is close.

The docker-compose and agent config look ok to me. A few things to check:

• The agent is not exposing any host ports - is your app running in the same docker-compose network and can your app reach the agent on port 4317?
• I would check that your app is exporting traces as expected by using the ConsoleExporter, and also the endpoint for the agent. In the example we set it via environment variable OTEL_EXPORTER_OTLP_ENDPOINT=http://agent:4317 for grpc which is the default mode of AddOtlpExporter().
• This may not be relevant, but in my experience I noticed that .StartWithHost() must be called or else the app will silently not export any traces, but also not log any errors.

[angeloferreira]

Hello there @mdisibio and thank you for reaching out on this!

I have seen your repo before, thanks for sharing that work with the community!
Because I have seen so many videos, blogposts and doc about the topic, I ended up changing the initial implementation which was somehow close to what you have on your repo.

I have noticed your OTel config for traces looks slightly different than mine - This is how it looks currently.
You don’t specify your Grafana Agent otlp endpoint on your example - is that on purpose? How does OTel know where to post the traces?

        string serviceName = "my-api-service";
        string otlpEndpoint = "http://grafana-agent:4317";
		var resource = ResourceBuilder.CreateDefault().AddService(serviceName);
        
            //Traces
            builder.Services.AddOpenTelemetry()
                .ConfigureResource(r => r.AddService(serviceName))
                .WithTracing(tracingOptions =>
                {
                    tracingOptions.AddAspNetCoreInstrumentation(); //Monitor incoming http requests
                    tracingOptions.AddHttpClientInstrumentation(); //Monitor outbound http requests
                    tracingOptions.AddOtlpExporter(exp => 
                    {
                        exp.Protocol = OtlpExportProtocol.HttpProtobuf;
                        exp.Endpoint = new Uri(otlpEndpoint);
                    });
                    tracingOptions.AddConsoleExporter();
                })
                .StartWithHost();

The agent is not exposing any host ports - is your app running in the same docker-compose network and can your app reach the agent on port 4317?

Yes, the app is setup to run together with the agent on the same docker-compose file;
How can I make sure the app reaches the agent on port 4317?
Do the logs I have posted on my initial post help check that?

I would check that your app is exporting traces as expected by using the ConsoleExporter, and also the endpoint for the agent. In the example, we set it via environment variable OTEL_EXPORTER_OTLP_ENDPOINT=http://agent:4317 for grpc which is the default mode of AddOtlpExporter().

This is a weird thing: - Traces produced by OpenTelemetry get logged when running locally on VSCode - as per the AddConsoleExporter() code shown above - but I can’t see any when running the app on Docker accessing the Docker Container’s console.
Can it mean they are not being generated for some reason and therefore there is nothing for Grafana Agent to pick and send to Tempo?

This may not be relevant, but in my experience I noticed that .StartWithHost() must be called or else the app will silently not export any traces, but also not log any errors.

Thanks for that, I have it on the end of the OTel config

[mdisibio]

I believe the issue may be this line:

exp.Protocol = OtlpExportProtocol.HttpProtobuf;

This seems to configure http exporter, not grpc. Can you try removing this line? Or we can configure the agent to receive this traffic with port :4318 and config:

    receivers:
      otlp:
        protocols:
          http:

How can I make sure the app reaches the agent

I have not used this, but the otel .net SDK can export logs via an EventSource. See the Troubleshooting section.

but I can’t see any when running the app on Docker accessing the Docker Container’s console

I’m not sure but perhaps the order of exporters and error handling determines why it occurs in one environment and not the other. Try moving the ConsoleExporter ahead so that it always get first chance to export?

[angeloferreira]

@mdisibio thank you for the tips!

I have checked out the code from your implementation and basically configured the Tempo endpoint plus credentials and I got the traces on my Grafana cloud.

This tells me that it can only be related to the fact that my app is not generating logs from OTel once executed on Docker - I will be focusing my investigation on that.

Sorted: despite my code was looking very similar to @mdisibio implementation, it was not working. Reason being I was apparently testing with a non updated version of my app on Docker. After flushing all Docker cache, it started working.

docker container prune
docker image prune -a
docker volume prune
docker compose-up

Important to refer to whoever gets here with the same issue: just make sure your implementation is similar to mdisibio’s one and it should work.

Thank you!