How to minimize the size of the Output data

Grafana k6 Converters & Integrations
1

Hi,

I used the k6 load testing tool for executing the following load.

  • Transactions per second (TPS): 2500
  • Response payload size: 50 KB
  • Duration: 6 hours

I send the data to InfluxDB. After 6 hours the size of my DB was more than 4 GB. Is there any way in which I can reduce the data size?

Thanks,
Dilshan

2

Hi @Dilshan_Fernando, welcome to the community forum :tada:

I don’t have any exact and concrete things you must do to reduce the size but here are some … suggestions that I recommend you try out possibly with 10minute test and see what gives you the best output (you might need to use new influxdb instance as some of those change what things are tags and what are fields):

  1. reduce system-tags you probably don’t care about most of those but they are still both sent and saved and indexed. I would recommend you keep name(not url) , status, expected_response. The following method, error, group and scenario might be useful as well in case you are making use of them as well as check but if you can make it work with just expected_response you can drop your checks as well :wink:
  2. Move as many of the above that you won’t be needing as indexed to fields, this has significant implications for the size and the speed of ingestion although it’s dependant as well as on how much different values there are. You do that through listing tags that you have enabled but want as fields in the K6_INFLUXDB_TAGS_AS_FIELDS env variable.
  3. You can try to use telegraf to aggregate the metrics. This will definitely be the biggest improvement but also requires telegraf and settuping it. I recommend using the statsd input of telegraf as it has pretty good aggregation, but you will likely need to try stuff out. I have a comment with an example config
  4. Using telegraf you can also additionally ignore metrics you don’t care about while this is still not possible with pure k6.

Hope this helps you and please write back how it all went :slight_smile:

1 Like
3

@mstoykov ,

Thanks for the details.

By using systemTags: ['status', 'error', 'error_code'], I was able to reduce some amount of data send by k6.

In the meantime, by sending k6 results to Datadog I was able to resolve this issue. With the Datadog, I didn’t see this issue.

Best Regards,
Dilshan

4

Hi @mstoykov,
I am looking for some guides that outline how I’d go about k6 > telegraf > influxdb > grafana workflow. I saw some feedback in the other community thread (K6 testing with InfluxDB/Grafana/Prometheus?) saying k6 doesn’t support telegraf and doesn’t plan to implement support for it in the future. However, looking at this post and your suggestion to use telegraf to aggregate the metrics for minimizing the size of output data, could you kinda explain your use case in detail and how did you make it happen? What format of k6 test output should be?
My use case is that the current presentation infrastructure uses telegraph agent as a local endpoint to send data to the MSK streaming platform for the downstream Influxdb to consume data from different sources. I was required to go the same route for sending k6 result to InfluxDB.

6

Hi @alex I have answered you in Does k6 support telegraf?, lets move the discussion there.

1 Like