Indexing trace_id as a label in Loki

[tzaspelfinatix]

Hey there,

I’m currently trying to index the trace_id from my traces in loki logs. In use is the Grafana agent for logging and tracing in a Kubernetes cluster. I find out that I have to adjust the pipelineStages in the Podslogs (podlogs.yaml) but I’m not familiar with this configuration and don’t know how to configure it right.

There are anyone who is familiar and can give me some advices?

Thanks,
Tom

Blockquote

[b0b]

Hi @tzaspelfinatix ,

you probably do not want to index trace_id. Dynamic labels are bad for Loki performance as is and with trace_id being a unique value per log message will blow up the Loki index (I would assume).

I have spent some time getting the trace_id lookup from logs work to link directly to Tempo traces.

In your Loki data source settings you need to create some Derived fields.

Take this log line as an example

2021-12-03T08:18:53.594Z	INFO	frontend/best_eta.go:106	Dispatch successful	{"service": "frontend", "trace_id": "15129c0846deed11", "span_id": "15129c0846deed11", "driver": "T768375C", "eta": "2m0s"}

This is not a very Loki friendly log format but you can still find the value of trace_id with a pretty simple regex

In Exlore the raw log looks like this

Again, not Loki friendly but not to worry. You need to apply some parsing so the trace_id becomes a Detected field

In the above case this means the query ends up looking like this

Which can then link to Tempo

Hope that helps

[tzaspelfinatix]

Hi @b0b,

thanks for your prompt answer. I’ve already implemented the derived fields, and it works great!

Relying on your first sentence, I prefer not to implement dynamic labels.

Thanks
Tom