Error connecting OTel collector and Tempo

[dvmorais]

Hello, english is not my native one so… sorry for the english

I’m trying to build an environment with open telemetry collector and tempo but I am receiving constantly this error after configuring the collector with the tempo URL:

2022-09-21T21:06:10.579Z error exporterhelper/queued_retry.go:395 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "otlp", "error": "Permanent error: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.metrics.v1.MetricsService", "dropped_items": 64}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector@v0.60.0/exporter/exporterhelper/queued_retry.go:395
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
go.opentelemetry.io/collector@v0.60.0/exporter/exporterhelper/metrics.go:133
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector@v0.60.0/exporter/exporterhelper/queued_retry.go:206
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
go.opentelemetry.io/collector@v0.60.0/exporter/exporterhelper/internal/bounded_memory_queue.go:61

if I understood well tempo is not actually connecting, right?

[joeelliott]

This error means that the collector is successfully connecting to Tempo, but it is failing to send metrics data over gRPC.

It seems like you configured a metrics pipeline and tried to send it to Tempo. Tempo is a distributed tracing backend and only accepts data from trace pipelines.

[dvmorais]

Great, Thanks for the response.

Is it possible to collect metrics in otel format with grafana or using prometheus as a data source is the most viable option?

[joeelliott]

So Grafana itself is for visualization only, but Mimir supports metrics written from the OTel protocol:

Configure the OpenTelemetry Collector to write metrics into Mimir

Note that it’s marked experimental.

[jangaraj]

No, Grafana (as a product for data visualization) doesn’t collect metrics, traces (or logs). First you need to store traces in trace storage (e.g. Tempo) and metrics in metric storage (e.g. Mimir, Prometheus, Cortex, …). And then Grafana can query used metrics/traces storages and visualize data from them.

If you have OTEL metrics (metric pipeline in the OTEL collector) then push them into Mimir (not into Tempo). Mimir is a Prometheus compatible metric backend.