Stress-testing file upload functionality

Grafana k6 OSS Support
1

Hello everyone!

I’d like to ‘stress’ test the file upload functionality on my web application. File size uploaded ranges from 50MB to 300MB.

I’ve tried using HTTP batch with multipart requests. But somehow, the upload time takes way longer than when I use browser/Postman. The K6 network throughput was capped at around 550KB/s.

Anyone has experiences on doing this in K6?

Cheers!

2

Hi @cgnwn, welcome to the forum :tada: .

I guess you are using the formdata shown in advance example ?

If so then … I would argue it isn’t really good idea to use exactly that for huge request bodies as it will just recreate the body on all requests, which probably eats a lot of CPU.

prebuilding the body and boundary as such did have around 100x better performance for me:

import http from 'k6/http';
import { check } from 'k6';
import { FormData } from 'https://jslib.k6.io/formdata/0.0.2/index.js';

const img1 = open('./image1.png', 'b');
const img2 = open('./image2.jpg', 'b');
const txt = open('./text.txt');

var b = function() {
    const fd = new FormData();
    fd.append('someTextField', 'someValue');
    fd.append('anotherTextField', 'anotherValue');
    fd.append('images', http.file(img1, 'image1.png', 'image/png'));
    fd.append('images', http.file(img2, 'image2.jpg', 'image/jpeg'));
    fd.append('text', http.file(txt, 'text.txt', 'text/plain'));
    return {body: fd.body(), boundary: fd.boundary};
}()

export default function() {
    const res = http.post('https://httpbin.test.k6.io/post', b.body,
        { headers: { 'Content-Type': 'multipart/form-data; boundary=' + b.boundary }});

    check(res, {
        'is status 200': (r) => r.status === 200,
    });
}

This IMO is still quite bad, as there are some unfortunate design decisions when the current HTTP API was made, in this particular case response.request.body is always populated, which in most cases is probably not an issue, given that the body will be something small, but in my case it is 20mb+ and in yours 50mb+ … additionally it gets converted to string which also doesn’t help at all :(. So you literally get another copy of the request body even if (for most cases IMO) you will never use it.

There is an issue made for read only binary data which could be expanded so it is something similar SharedArray and help with the memory usage of having the body copied for each VU. There are also some more ramblings of mine on the current issues surrounding uploading files and not only.

But regardless, the above should make this a lot better for you, although at least in my testing the memory usage per VU is still really big, and this will require multiple changes to k6 in order for those to work. Hope it helps though :slight_smile:

1 Like