Headers_setter not working to set dynamically X-Scope-OrgID header

Grafana Prometheus
,
1

Hi everyone, I am running Grafana on Kubernetes with also Tempo, Loki and Mimir installed on Kubernetes.
The collect layer is made with OpenTelemetry Collector and everything is fine sending data into datasources (we are using Loki exporter for Loki et otlphttp for both Tempo and Mimir)

The connection is working fine and now we need to enable muti-tenant mode in order to segregate sensible data.

The problem is that till now we didn’t succeed sending metrics to Mimir with OpenTelemetry Collector through otlphttp exporter using header_setter in order to set header X-Scope-OrgID to a dynamic value. When we set a static value we are able to see metrics on Grafana side using the static header, but anytime we use the from_context: keyword, we are getting a HTTP 401 Error (no org id).

In addition, we aleady succeded sending logs to Loki with dynamic header using:
attributes/loki:
actions:
- action: insert
key: loki.tenant
value: service.name

Below is the configuration we are using to get the 401 error:

Configuration file OpentelemetryCollector:

extensions:
  headers_setter:
    headers:
      - action: insert
        key: X-Scope-OrgID
        from_context: service.name

receivers:
  otlp:
    protocols:
      grpc:
      http:
        include_metadata: true

exporters:
  otlphttp/mimir:
    endpoint: https://Our_endpoint/otlp
    timeout: 30s
    tls:
      insecure_skip_verify: true
    auth:
      authenticator: headers_setter

service:
  extensions: [health_check, http_forwarder, zpages, headers_setter]
  pipelines:
    metrics:
      receivers: [otlp]
      processors: []
      exporters: [otlphttp/mimir]

And here is the error on OpenTelemetry Collector side:
error exporterhelper/queued_retry.go:401 Exporting failed. The error is not retryable. Dropping data. {“kind”: “exporter”, “data_type”: “metrics”, “name”: “otlphttp/mimir”, “error”: “Permanent error: error exporting items, request to https://Our_endpoint/otlp/v1/metrics responded with HTTP Status Code 401”, “dropped_items”: 129}

To finish, we tried to replace “from_context: service.name” by “value: test” in headers_setter and we correctly saw metrics into test tenant.

Versions we are using:
OpenTelemetry Collector Contrib: 0.75.0
Mimir: 2.7.1
Grafana: 9.4.7
Loki: 2.6.1

Thanks in advance !

2

Update: After all this time, I finally found out the issue.
An important thing that I didn’t mentioned in my first post: This OpenTelemetry Collector I was talking about is a OTEL Gateway, aggregating all the traffic before sending it into datasources. The problem with that is that I thought service.name parameter was part of metadata and was accessible with from_context parameter (spoiler, I was wrong).

The solution I found to fix this issue is to add the following headers_setter configuration on the local OTEL agent:

extensions:
  headers_setter:
    headers:
      - action: upsert
        key: X-Scope-OrgID
        value: my_tenant

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  otlphttp:
    endpoint: https://My_OTEL_GW
    tls:
      insecure_skip_verify: true
    auth:
      authenticator: headers_setter

service:
  extensions: [headers_setter]
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [otlphttp]

You will notice that we are using otlphttp to send from OTEL Client to OTEL Gateway, this will be changed to otlp in the future.
And on the OTEL Gateway, here is the configuration I have:

extensions:
  headers_setter:
    headers:
      - action: upsert
        key: X-Scope-OrgID
        from_context: X-Scope-OrgID

receivers:
  otlp:
    protocols:
      grpc:
        include_metadata: true
      http:
        include_metadata: true

Rest of the configuration is the same than the one is printed in my first post.
The headers_setters configuration on the Gateway is here working as a passthrough for X-Scope-OrgID parameter, this is the only way I found to ingest data with dynamic tenant with this architecture.