Experimental-prometheus-rw output to a multi-tenant Mimir deployment

Grafana k6 Extensions
1

Hi all,

I am trying to setup K6 output to a multi-tenant Mimir deployment via “experimental-prometheus-rw”

I have deployed multi-tenancy enabled Mimir OpenSource in K8s and grafana-agents running on different clusters can send metrics using X-Scope-OrgID header to different tenants successfully.

I cannot manage to set it up for K6 Prometheus Remote Write with “-o experimental-prometheus-rw” option. I have set following variables:

K6_PROMETHEUS_RW_PASSWORD=<masked_password>
K6_PROMETHEUS_RW_SERVER_URL=https://<masked_fqdn>/api/v1/push
K6_PROMETHEUS_RW_USERNAME=
K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgID=<tenant_id>

and I see the following error at the end of K6 run:

ERRO[0001] Failed to send the time series data to the endpoint error=“got status code: 401 instead expected a 2xx successful status code” output=“Prometheus remote write”

If I unset “K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgID” environment variable, I see the metrics stored as “anonymous” tenant (default tenant on Mimir if no X-Scope-OrgID defined)

Anyone used Prometheus Remote Write with a multi-tenant Mimir deployment to a specific tenant?

Thanks in advance.
Suleyman Kutlu

4

Hi @snk,
welcome to the community forum :tada:

It seems you’re getting an unexpected error because at the moment the scope id header is not supported as you can check from this issue. Better API for HTTP Headers · Issue #101 · grafana/xk6-output-prometheus-remote · GitHub
Please, add a thumbs up reaction to the issue so we can raise the priority.

Are you sure you’re setting it? You should get an error from k6 similar to this:

level=error msg="invalid environment variable name 'K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgId'"
1 Like
6

Hi @codebien

Thanks for the link. There is a slight difference though with the issue:

Documentation says env variable should be K6_PROMETHEUS_RW_HEADERS_=

So I have an env variable as stated in my question:

K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgID=<tenant_id>

But the issue has

K6_PROMETHEUES_RW_HTTP_HEADERS=X-Scope-OrgID:<org>,X-ANOTHER_HEADER:val

So which one should I use? I can try that as well :wink:

Thanks in advance…
-Suleyman

7

Only if I use

K6_PROMETHEUS_RW_HEADERS_X-Scope-OrgID=<tenant_id>

I got that error about status code 401 at the end.

Other combinations I have tested below did not give any error but did not do anything neither

K6_PROMETHEUS_RW_HTTP_HEADERS=X-Scope-OrgID:
K6_PROMETHEUE_RW_HTTP_HEADERS_X-Scope-OrgID=
K6_PROMETHEUS_RW_HEADERS=X-Scope-OrgID:

I never get that invalid environment variable error.

Thanks
-Suleyman