Manage Big JSON size from k6 output

Grafana k6 OSS Support
1

Hi,

I try to do heavy load test with target more than 63000 VU using VM as worker.
I get error about limitation file size JSON from k6 output.

Here example log

Test ✓ [ 100% ] 63000 VUs  3m0s  6299754/6300000 iters, 100 per VU
time="2023-01-10T04:58:00Z" level=error msg="Sample couldn't be marshalled to JSON" error="write target/reports/report.json: file too large" filename=target/reports/report.json output=json
time="2023-01-10T04:58:02Z" level=info msg="[k6-reporter v2.3.0] Generating HTML summary report" source=console
time="2023-01-10T04:58:28Z" level=error msg="Sample couldn't be marshalled to JSON" error="write target/reports/report.json: file too large" filename=target/reports/report.json output=json

Any suggestion for manipulate output?

Thank you.

3

Hi @estu

Welcome to the community forums :wave:

I imagine you are exporting all the metrics, the granular data points, in JSON format with the JSON output. Please clarify if me if I’m wrong. This can create a big file with a large test. Especially when using JSON, since it’s verbose.

One solution is using k6’s built-in support for outputting the results in other formats, like Prometheus, InfluxDB, or maybe even CSV (less verbose than JSON). These formats are more suitable for large data sets. Especially streaming the results to a data store.

If you want to see only the aggregated data, you can export the end-of-test summary to a JSON file. For more details, refer to the handleSummary() topic in the end-of-test summary docs.

There is probably also the option to increase the maximum file size limit allowed by your system. However, this may not be feasible or desirable. Or reducing the amount of data that is being recorded by k6.

I hope this helps.

Cheers!

5

Thank you @eyeveebe for insight.

I will try again, and back with result.

Cheers!

1 Like
6

Hi,

I am back.

So csv still cannot solve my problem.

So. I fix problem to do something in my VM. I set sysctl file size limit to -1 (unlimited).

edit file /etc/security/limits.conf than add this line

k6-user   hard    fsize   -1

Then restart vm.

For 60K VU, k6 generate JSON output with size 26GB.

And then move to solve problem how generate report from that JSON file, work on progress.

Thanks all.

1 Like