K6 testing with InfluxDB/Grafana/Prometheus?

Grafana k6 OSS Support
1

I am relatively new to k6 and definitely new to Grafana/InfluxDB. But thanks to @oleiade Grafana/InfluxDB/Prometheus Docker setup at GitHub - oleiade/k6-support: A full-fledged local k6 ecosystem a docker-compose up away it was very easy to get things setup :slight_smile:

Now I have some general questions with k6 and InfluxDB and Prometheus. I started with only using k6 with InfluxDB 1.8/Grafana and have found that under high concurrency InfluxDB needs tuning parameters like K6_INFLUXDB_PUSH_INTERVAL and K6_INFLUXDB_CONCURRENT_WRITES and see it’s a known issue

I can push to around 2000 VUs with InfluxDB before gets less stable/reliable for metrics recorded in Grafana by tweaking K6_INFLUXDB_PUSH_INTERVAL and K6_INFLUXDB_CONCURRENT_WRITES and stripping some metrics/tags for my scripted tests (thanks for @mstoykov post for tips). I understand it is also dependent on the system and hardware handling things.

So the questions are:

  1. oleiade’s Docker image adds Prometheus to the mix while official K6 Docker compose image doesn’t include Prometheus. Would Prometheus out of the box and potentially after tuning, handle heavier k6 benchmark VUs loads than InfluxDB would handle?
  2. I’ve seen a few mentions for workarounds to InfluxDB to scale with putting Telegraf in front and use that as a middleman.

I’ve modified odeiade’s Docker image to add Telegraf 1.22 into the mix and I can properly setup Telegraf to pull in my Docker/host image stats in a separate Grafana dashboard. But I haven’t found any online guides that outline how I’d go about this for k6 > telegraf > influxdb > grafana workflow ? Any pointers to where to read up on this?

1 Like
2

Hi @eva2000

Glad the existing resources were useful to you :bowing_man:

Regarding Prometheus’ performance versus InfluxDB, I, unfortunately, can’t answer that question in a general manner. The only way to find out would be to run your own investigation on that front. My understanding is that InfluxDB uses a push model (as opposed to Prometheus’ pull model), which might have an impact on performance depending on your workflow/workload.

Regarding using k6 with telegraf, we have an open issue about it. My understanding of its current state is: k6 doesn’t support telegraf, we don’t plan to implement support for it in the future, but if users are interested, and willing to contribute, they could probably put one together via an xk6-extension.

Hope that’s helpful, and let me know if you need anymore assistance :bowing_man:

1 Like
3

Thanks, definitely will have to try it and see :slight_smile:

Cheers. Unfortunately don’t have the coding chops, so hopefully developments eventually come for Telegraf support.

Guess the other way which I read during my research is to run k6 with --out to a json file and parse it for manual insertion into InfluxDB? Maybe k6 can update --out influxdb=filename.log to read as save to filename.log in InfluxDB batch line format for easier insertion?

Something like

pzcat summary-raw.gz | jq -c '. | select(.type=="Point" and .data.tags.expected_response == "true" and .metric == "http_req_duration" and .data.tags.status >= "200")' | tail -3 | tee sample-output-http-duration-log.log

cat sample-output-http-duration-log.log | jq -r
{
  "type": "Point",
  "data": {
    "time": "2022-10-10T13:52:28.232671309Z",
    "value": 0.174614,
    "tags": {
      "name": "https://domain1.com",
      "status": "200",
      "expected_response": "true",
      "stage": "3",
      "scenario": "ramping_vus"
    }
  },
  "metric": "http_req_duration"
}
{
  "type": "Point",
  "data": {
    "time": "2022-10-10T13:52:28.233276495Z",
    "value": 0.185925,
    "tags": {
      "stage": "3",
      "name": "https://domain1.com",
      "status": "200",
      "expected_response": "true",
      "scenario": "ramping_vus"
    }
  },
  "metric": "http_req_duration"
}
{
  "type": "Point",
  "data": {
    "time": "2022-10-10T13:52:28.2338959Z",
    "value": 0.204826,
    "tags": {
      "stage": "3",
      "name": "https://domain1.com",
      "status": "200",
      "expected_response": "true",
      "scenario": "ramping_vus"
    }
  },
  "metric": "http_req_duration"
}

cat sample-output-http-duration-log.log | jq -r '"\(.metric),\(.data.tags|to_entries|map("\(.key)=\(.value|tostring)")|[.[]]|sort|join(",")) value=\(.data.value) \((.data.time[0:19] + "Z"|fromdateiso8601)*1000000000 + (.data.time[20:27]|tonumber))"'

http_req_duration,expected_response=true,name=https://domain1.com,scenario=ramping_vus,stage=3,status=200 value=0.174614 1665409948002326800
http_req_duration,expected_response=true,name=https://domain1.com,scenario=ramping_vus,stage=3,status=200 value=0.185925 1665409948002332700
http_req_duration,expected_response=true,name=https://domain1.com,scenario=ramping_vus,stage=3,status=200 value=0.204826 1665409948002339000
4

I sort of went down a rabbit hole looking at parsing the k6 --output logs into InfluxDB batch line format https://github.com/centminmod/k6-benchmarking/blob/master/logs/k6-log-parsing.md :nerd_face:

But would be nice if k6 natively had a --out influxdb=filename.log option so the file would already be populated in InfluxDB batch line format. Would save a lot of time parsing the logs later.

5

Glad you got it sorted out :tada:

I believe your use case might be too much of an edge case for k6 to integrate it in its core, but it’s probably worth exploring it further! I’d invite you to open a feature-request Github Pull Request describing it, that would be much appreciated and the team would look into it.

Let me know if you need any further support :bowing_man:

1 Like
6

Thanks I’ve posted a feature request at Support native --out influxdb=filename.log InfluxDB batch line format logs · Issue #2721 · grafana/k6 · GitHub :slight_smile:

7

Just an update looks like the conversion of k6 output JSON logs to InfluxDB batch line format worked ok. Demo at https://github.com/centminmod/k6-benchmarking/blob/master/bench-ramping-vus.sh-2.md :slight_smile: