Upload file and be able to determine blob bytesize

Grafana k6 OSS Support
1

Hi K6 team!

I have a question related to upload of files.

Our API for upload has an endpoint that needs to determine the size of the file in bytes.
I used the command as displayed on your documentation to open and read the test.jpg file. This works.
However I run into an issue when I need to finalise the upload.

In the last endpoint called: finaliseResponse, I need a filesize of the file. In the code example, it shows the bytes hardcoded(this works), however I would like to have this dynamically done based on the file.

I tried to use the Blob: size property - Web APIs | MDN method as seen below, but that does not do the trick.

body = {
    "fileName": filename,
    "fileSize": blob.size,      <----
    "chunksCount": 1,
    "sha256": file_sha,
}

I also tried to use

binFile.byteLength
blob.byteLength

What way of determining the size of the file or the blob, would be recommend?

Also tried to use webpack to import fs module, but this generates this issue:

Built at: 01/12/2022 1:47:47 PM
         Asset      Size  Chunks             Chunk Names
main.bundle.js  2.17 KiB       0  [emitted]  main
Entrypoint main = main.bundle.js
[0] ./tests/script.js 3.02 KiB {0} [built]
[1] external "k6/http" 42 bytes {0} [built]
[2] external "k6" 42 bytes {0} [built]
[3] external "k6/crypto" 42 bytes {0} [built]
[5] external "fs" 42 bytes {0} [built]
    + 1 hidden module
➜  Prometheus-original git:(k6_init) ✗ k6 run dist/main.bundle.js

          /\      |‾‾| /‾‾/   /‾‾/
     /\  /  \     |  |/  /   /  /
    /  \/    \    |     (   /   ‾‾\
   /          \   |  |\  \ |  (‾)  |
  / __________ \  |__| \__\ \_____/ .io

WARN[0000] The moduleSpecifier "fs" has no scheme but we will try to resolve it as remote module. This will be deprecated in the future and all remote modules will need to explicitly use "https" as scheme.

Thanks in advance, the last part is the actual code.

import http from 'k6/http';
import {check} from 'k6';
import crypto from 'k6/crypto';
const binFile = open('test.jpg', 'b');

export default function () {
    let body
    let filename = 'test.jpg'
    const f = http.file(binFile, 'image/jpeg');
    var blob = f.data
    let file_sha = crypto.sha256(f.data, 'hex');

    body = {
        "client_id": 'some id,
        "client_secret": 'some secret',
        "grant_type": "client_credentials",
    }
    //Authorise yourself with OAUTH2
    let authenticate_response = http.post(`SOMEURL/v7/authentication/oauth2/token/`, body,
        {
            headers: {'Content-Type': 'application/x-www-form-urlencoded'}
        })

    let token = authenticate_response.json().access_token

    //Prepare a file upload
    const prep_response = http.post('SOMEURL/v7/upload/prepare', {},
        {headers: {"Content-Type": "image/jpeg", Authorization: `Bearer ${token}`}}
    );
    let file_id = prep_response.json().file_id
        check(prep_response, {
            'is status 201': (r) => r.status === 201
        });

    const upload_chunk = http.post(`SOMEURL/v7/upload/${file_id}/chunk/0`, blob,
        {
            headers: {
                "Content-Type": "multipart/form-data",
                "Content-SHA256": file_sha, Authorization: `Bearer ${token}`
            }
        });
        check(upload_chunk, {'is status 201': (r) => r.status === 201});
        
   // Finalise the upload to the API

    body = {
        "fileName": filename,
        "fileSize": 3048897,      <---- HARDCODED VALUE
        "chunksCount": 1,
        "sha256": file_sha,
    }
    const finaliseResponse = http.post(`SOMEURL/v7/upload/${file_id}/finalise_api`, body,
        {headers: {
                "Content-Type": "application/x-www-form-urlencoded",
                "Content-SHA256": file_sha, Authorization : `Bearer ${token}`
        }
    });
        check(finaliseResponse, {
            'is status 201': (r) => r.status === 201});
4

Hi @rodyb,

Thanks for reaching out! I’ll do my best to support you.

I think the solution to your issue comes from a legitimate confusion as to the type of binFile. In the spirit of transparency: we are actively working on improvements on that front at the moment. While it seems that you expected it to be a Blob (part of the FileAPI, and, as far as I know, not currently supported by K6), it returns in fact an ArrayBuffer (part of the ECMAScript specification).

Thus, you should be able to obtain the size of your file using the following operation: binFile.byteLength. Running this in a dummy script against this picture of a very busy cat gave me the proper size indeed.

As I’m at it, I can recommend two small improvements you could bring to the test script you’ve posted:

  • as you upload the same file every time, its SHA256 hash is not expected to change. Thus, you could move its computation into the init context of your script, effectively computing only once for the whole K6 run (as opposed to every iteration). It would need to happen right after the opening of the file:
const binFile = open("busycat.jpg", "b");
const fileSHA256 = crypto.sha256(binFile, 'hex');
  • In the event you’re interested in excluding authentication from the measurement, and unless the authentication tokens provided by the service you interact with are very short-lived, you could consider obtaining it as part of the setup function of the script. That way it will be called less often too:
import http from "k6/http";
import { check, sleep } from "k6";
import crypto from "k6/crypto";

const binFile = open("busycat.jpg", "b");
const fileSHA256 = crypto.sha256(binFile, 'hex');

export setup() {
    let authenticate_response = http.post(`SOMEURL/v7/authentication/oauth2/token/`, body, {
        headers: {'Content-Type': 'application/x-www-form-urlencoded'}
    });

    return {'token': authenticate_response.json().access_token};
}

export default function(data) {
  // from here on, whenever you need to use the token, access it via data.token
}

Let me know if that was helpful, and if you need any more support from me :slight_smile:

5

@oleiade

Thanks for the quick reply and I appreciate the improvements! Will definitely implement them.
I do however want to see what byteLength returns, but when I log it, it just returns.

const binFile = open('test.jpg', 'b');

export default function () {
    let body
    let filename = 'test.jpg'
    const f = http.file(binFile, 'image/jpeg');
    var blob = f.data
    let file_sha = crypto.sha256(f.data, 'hex');
    console.log(binFile.byteLength)

But it returns


INFO[0001] undefined                                     source=console

Can you tell me what I am missing? And can you share how that bytes size looks like when you log it?

Thanks in advance.

6

Hey @rodyb

Sorry to read that, didn’t solve your issue. I guess I wasn’t thorough enough in my experimentation indeed, sorry for that.

I have an intuition as to what could be happening, but I need to verify it. Furthermore, I’ll come back to you as soon as I know more. As I’m at it, could you also give me more information about your test setup please: K6 version?

In the context of a very simple test script:

const binFile = open("busycat.jpg", "b");

export default function () {
  console.log(binFile.byteLength);
  sleep(10);
}

Given busycat.jpg is the file I refered to earlier, k6 run logs: INFO[0000] 114228 source=console, and du -b busycat.jpg reports 114228 too.

7

@oleiade

No problem, I appreciate the help :slight_smile:

My K6 version is : k6 v0.31.1 ((devel), go1.16.2, darwin/amd64)
I tried also with your busycat example, but also get undefined.

Let me know if any more info is required.

And, thank again!

8

@oleiade It was the version. I was under the assumption I was on the latest version. Now when updating it works perfectly!

Thanks for all the help and recommendations!

9

Hey @rodyb, glad you found out the issue, and that the solution turned out to be so easy :innocent:

Don’t hesitate to reach out again if you need any further help!

1 Like