Grafana Explore page does not show the tracing result for a Tempo data source

Grafana
1

So we run a grafana, loki and tempo combination in a kubernetes cluster. The data seems to get ingested just fine everywhere.

So when I put a trace ID that I know exists into the query field for the tempo data source, no tracing view shows up. Now it does do the request to api/datasources/proxy/2/api/traces/<TraceId> and it gets JSON response with essentially the following structure:

{
	"batches": [
		{
			"resource": {
				"attributes": [
                    ... bunch of attributes
				]
			},
			"instrumentationLibrarySpans": [
				{
					"instrumentationLibrary": {
						"name": "OpenTelemetry.Instrumentation.AspNetCore",
						"version": "1.0.0.0"
					},
					"spans": [
                        ... 1 span without a parent
					]
				},
				{
					"instrumentationLibrary": {
						"name": "OpenTelemetry.Instrumentation.Http",
						"version": "1.0.0.0"
					},
					"spans": [
						... 2 spans that have a parent id from the span defined above.
					]
				}
			]
		}
	]
}

The jaeger ui that ships with Tempo does show the timeline view just fine. And there are no errors in the Browser console.

At this point I’m sort of lost on how to get grafana to display me a nice Trace timeline view.

Tried this on both the latest 7.3 and 7.4 image versions.

2

I’ve same issue, (resolved for me, see UPD below)

JSON trace with attribute service.name shows correctly.
But it does not show the trace without it.

I’m not sure if service.name attribute is a reason of fail. (UPD. Yes it is)
So I will try to provide service.name attribute to my traces.
I will let it know if there would be any progress on this issue.

Anyway it is strange if trace document format could not be parsed and there are no errors in Grafana logs.

PS
According to this document service.name is required attribute.

UPD:

Yes, service.name is mandatory attribute to make it possible to view traces in Grafana.

In my case trace starts at Envoy proxy. To make Envoy fill service.name attribute it has to run Envoy with --service-cluster parameter.

When it’s done I had seen traces finally!

It would be nice if Grafana will default service.name with some kind of “UNDEFINED” value, to make it possible to view traces with missing service.name attribute value. Or at least log parsing errors for debug purpose.

I’ve spent half of a day to resolve this issue.

“batches”: [
{
“resource”: {
“attributes”: [
{
“key”: “service.name”,
“value”: {
“stringValue”: “tempo-vulture”
}
}
]
},
“instrumentationLibrarySpans”: [
…

3

This topic was automatically closed after 365 days. New replies are no longer allowed.