Prometheus Metrics Not displayed in Grafana

[satheeshpandianj]

Hi,
I am trying to visualise the prometheus metrics in Grafana. I was able to push the test metrics to prometheus without any issues. However, I am unable to see them in Grafana.

I am running docker-compose file for Grafana and Prometheus. I was able to spin up them without any issues. I am running the test with the below command

K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true k6 run -o experimental-prometheus-rw --tag testid=100 test.js

I am seeing the test result in the terminal. Also, I am able to see them in the prometheus.

When I try to add the datasource using “http://localhost:9090/api/v1/write”, I throws me an error.

So I changed the container IP instead of localhost and it works fine.

However, it is not showing any metrics in the dashboard.

Refer all the snapshots attached. Kindly help.

Thanks,
Satheesh

[satheeshpandianj]

Hi Team,
Any help is much appreciated.

Thanks,
Satheesh

[eyeveebee]

Hi @satheeshpandianj

I can try and reproduce this locally. Can you share the docker compose you are using, a (sanitized) version of your tests script, the k6 version used, and the version of the dashboard used?

Cheers!

[satheeshpandianj]

Thanks @eyeveebe . Below is the docker-compose file. K6 version is “k6 v0.44.1 ((devel), go1.20.4, darwin/amd64)”. And, I am using sample GET method script available in k6.io site.

version: '3.3'

networks:
  grafana:
  prometheus:

services:
  prometheus:
    image: prom/prometheus:v2.42.0
    command:
      - --web.enable-remote-write-receiver
      - --enable-feature=native-histograms
      - --config.file=/etc/prometheus/prometheus.yml
    networks:
      - grafana
      - prometheus
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:9.4.7
    networks:
      - grafana
      - prometheus
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_BASIC_ENABLED=false
    volumes:
      - ./grafana:/etc/grafana/provisioning/

[eyeveebee]

Hi @shailendrakumaromkar

Thanks for the additional info. Can you share the script or from waht page in the k6 docs you take the full script example? Just to make sure we are testing the same.

And also, what version of the k6 version are you using?

Thanks!

[satheeshpandianj]

K6 version : k6 v0.44.1 ((devel), go1.20.4, darwin/amd64)

Script - JS Code

import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
  http.get('https://test.k6.io');
  sleep(1);
}

Thank you for helping me.

[eyeveebee]

Hi @satheeshpandianj

I’m not sure why it does not work for you, unless the contents of grafana directory are not the same I have (running on Mac OS).

I run the example from https://github.com/grafana/xk6-output-prometheus-remote/blob/main/docker-compose.yml. Using that repo, it loads the dashboard from https://github.com/grafana/xk6-output-prometheus-remote/tree/main/grafana/dashboards, and seems to work well for me. I do not have to change the Prometheus URL, http://localhost:9090/api/v1/write works well.

This is probably what you need to investigate further. Is your local environment working with the URL we expect from inside the container, http://prometheus:9090 (it should probably translate to 172.21.0.1:9090 in your case)?

You can probably check in the grafana container has issues connecting to prometheus:

$ docker exec -it <grafana-container> /bin/bash
2107a0272568:/usr/share/grafana$ wget http://prometheus:9090

And also double-check the Grafana docker logs, to see if anything is suspicious when you are running this dashboard.

In my case, I followed the instructions at Prometheus remote write and used your k6 test script. The only thing I changed was increasing VUs and duration so that I could see a bit more results. With the default script, you will run an interaction without much data to be seen. Though you should still see the one request:

import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
    vus: 10,
    duration: '30s',
};

export default function () {
    http.get('https://test.k6.io');
    sleep(1);
}

I ran the test above with K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true k6 run -o experimental-prometheus-rw --tag testid=test-1 ./samples/test.js and the results show as well, with no need to reconfigure anything as you had to:

You could also load Official k6 Test Result | Grafana Labs with similar results, I imagine. I do not think this will be related or help out.

I am sorry I cannot help more. I would probably investigate how the docker container’s network works in your platform, and if you need to change it to adjust in any way, which is not the usual use case with a docker-compose, but…

Cheers!