Using Grafana, grafana-agent, loki and tempo “latest” docker tags at the time of writing
I’m trying to get the grafana agent to add trace data to loki as per: Automatic logging | Grafana Tempo documentation
There are 2 things I expected to be different:
-
I can see my traces in Grafana and I can see the are logged in Loki as well. But they don’t have the labels like service_name, traceid. Note that svc is visible because I enabled it explicitly in config
-
I only get logs in loki when I enable spans and/or porcesses. But I want just the root span to be indexed.
My agent config
traces:
configs:
- name: default
remote_write:
- endpoint: ${TEMPO_HOST}
insecure: true
automatic_logging:
backend: logs_instance
logs_instance_name: default
spans: true
processes: true
roots: true
receivers:
otlp:
protocols:
grpc:
http:
When setting spans and processes to false, no logs are sent to loki. Also I’d like to have things like svc as a label called service_name so that I can easily query on it as I do with the normal logs.
The end goal is that I can easily query traces for outliers in terms of response time.