Exemplars for Grafana Tempo Traces

Grafana Tempo
1

Hello,
We are using Grafana 9.1.1, Prometheus 2.36.2, Tempo 1.5.0 (with remote-write) and we have setup exemplars (prom data source updated in grafana, exemplar-storage feature enabled in prometheus) but we are still not getting exemplars anywhere. I’ve read the docs but can not see what we are missing, any ideas or pointers pls? Are we supposed to see them when we go to tempo:3100/metrics (we don’t)?

2

Where are you generating metrics from? You will need to change the instrumentation of your code to also create exemplars as you update the metrics. Once your app has been updated, the Prometheus scraping metrics will also capture those exemplars.

3

These are standard metrics generated by tempo, we are trying to setup metrics-to-traces in grafana dashboards so I guess it would be span metrics generated by tempo that would include the exemplars

4

In your remote write config, do you have send_exemplars set to true?

Example config from my local cluster:

metrics_generator:
    # other config omitted
    storage:
        path: /var/tempo/generator_wal
        remote_write:
            - send_exemplars: true
              url: http://prometheus:9090/prometheus/api/v1/write
1 Like
5

no but the name of the option sounds promising :laughing:

6

you are a star, thank you so much @koenraad !

7

I see this isn’t documented anywhere so it’s not very obvious :sweat_smile: I’ll create a ticket so we can improve the docs.

3 Likes
8

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.