Issue while parallel data generation using K6 [ Data generation terminating abruptly ]

gopinath · November 3, 2020, 1:12pm

Hello, We are currently performing load testing of graphQL with K6. We are using graphQL API services to get the data into our pre-created hbase tables … Below is our API call and graphQL mutation payload passed while calling below API from K6. Here FT7001 is the hbase table name into which we are trying to load test data into using K6. The table has 30 columns FLAT_ATTR_1 … till FLAT_ATTR_30… Similarly there are 9 other tables from FT7002…till FT7010…

ENDPOINT_URL=http://p2unit-rt.epsilonagilityloyalty.com/execute

Payload :

mutation {
newFT7001(input: {
  FLAT_ATTR_5 : "2015-06-30 05:07:00",
  FLAT_ATTR_1: "${randomNumber}",
  FLAT_ATTR_2: "StringOne",
  FLAT_ATTR_3: "StringTwo",
  FLAT_ATTR_4: "StringThree",
  FLAT_ATTR_6: "fef"
  FLAT_ATTR_7: "fref"
  FLAT_ATTR_8: "fdsf"
  FLAT_ATTR_9: "fdfsf"
  FLAT_ATTR_10: "dsf"
  FLAT_ATTR_11: "dsf"
  FLAT_ATTR_12: "fdsf"
  FLAT_ATTR_13: "sfde"
  FLAT_ATTR_14: "fdsf"
  FLAT_ATTR_15: "sdgf"
  FLAT_ATTR_16: "sdf"
  FLAT_ATTR_17: "dfdds"
  FLAT_ATTR_18: "sdfds"
  FLAT_ATTR_19: "fds"
  FLAT_ATTR_20: "sfad"
  FLAT_ATTR_21: "fsds"
  FLAT_ATTR_22: "sfa"
  FLAT_ATTR_23: "sfds"
  FLAT_ATTR_24: "dsf"
  FLAT_ATTR_25: "fds"
  FLAT_ATTR_26: "dsfs"
  FLAT_ATTR_27: "sdfds"
  FLAT_ATTR_28: "dsf"
  FLAT_ATTR_29: "sfds"
  FLAT_ATTR_30: "sdfdsd"
  
}){id FLAT_ATTR_5 FLAT_ATTR_1 FLAT_ATTR_2 FLAT_ATTR_3 FLAT_ATTR_4 FLAT_ATTR_6 FLAT_ATTR_7 FLAT_ATTR_8 FLAT_ATTR_9 FLAT_ATTR_10 FLAT_ATTR_11 FLAT_ATTR_12 FLAT_ATTR_13 FLAT_ATTR_14 FLAT_ATTR_15 FLAT_ATTR_16 FLAT_ATTR_17 FLAT_ATTR_18 FLAT_ATTR_19 FLAT_ATTR_20 FLAT_ATTR_21 FLAT_ATTR_22 FLAT_ATTR_23 FLAT_ATTR_24 FLAT_ATTR_25 FLAT_ATTR_26 FLAT_ATTR_27 FLAT_ATTR_28 FLAT_ATTR_29 FLAT_ATTR_30}}

So we are running below K6 command [ totally 10 instances of below command in parallel ] to get the data generated into each of above 10 hbase tables. I am also capturing the K6 logs for each of these 1o instances.

k6 run --vus 100 --duration 2880m --rps 500 --iterations 10000 -e ENDPOINT_URL=http://p2unit-rt.epsilonagilityloyalty.com/execute -e MIN=1 -e MAX=100000000 ./specPerformance/specDynamicEntityForwardSync/specPerfForwardSyncK6JSFiles/PerfTestFlatK6JSFile1.js >> ./specPerformance/specDynamicEntityForwardSync/K6LOGS/FT7001.txt &

Below is the Java Script file passed to above command. [ that is the content of the file “PerfTestFlatK6JSFile1.js” in above command ]. Similar JS file content is used across all 10 K6 run instances run in parallel as quoted above except that we are generating data for different hbase tables
[ that mean only “newFT7001(input…” content changes in the JS file…here hbase table name is FT7001…similarly other JS files have like “newFT7002(input…” till “newFT7010(input”…

As per above command we are generating 10000 records per each hbase table [ single K6 instance run ]… and there are 10 K6 instances triggered in parallel…

But we see that out of 10 only for 6-7 tables data of 10000 gets generated and for other 3 tables the data generation terminates abruptly after same 4000 plus records or 6000 plus records and there is not much information in the K6 logs for these runs saying why it terminated.

ATTACHING K6 logs as below for success case [ all 10000 records generated ] and failure case [ only 4000 plus records generated ] as below…

Please request to have a look here and let us know the reason for such terminations and how it can be addressed so that all 10 K6 instances generate expected 10000 records.

JS FILE CONTENT:

@@@@@@@@ START OF FILE @@@@@@@@@@@@@@

import http from "k6/http";
import { check } from "k6";
import { Rate, Trend } from "k6/metrics";
import faker from "cdnjs.com/libraries/Faker";

let createErrorRate = new Rate("New Entity errors");
let CreateTrend = new Trend("New Entity");
let createResultErrorRate = new Rate("New Entity Result errors");

export default function () {
    var endpointUrl = `${__ENV.ENDPOINT_URL}`;
    var i = 0;
    console.log("endpointUrl::" +endpointUrl);

var randomNumber = randomNumberGeneratorEntID();
console.log("random number value " +randomNumber);
var entityName = "FT7003";
console.log("entityName "+entityName);
const mutation = `mutation {
newFT7001(input: {
  FLAT_ATTR_5 : "2015-06-30 05:07:00",
  FLAT_ATTR_1: "${randomNumber}",
  FLAT_ATTR_2: "StringOne",
  FLAT_ATTR_3: "StringTwo",
  FLAT_ATTR_4: "StringThree",
  FLAT_ATTR_6: "fef"
  FLAT_ATTR_7: "fref"
  FLAT_ATTR_8: "fdsf"
  FLAT_ATTR_9: "fdfsf"
  FLAT_ATTR_10: "dsf"
  FLAT_ATTR_11: "dsf"
  FLAT_ATTR_12: "fdsf"
  FLAT_ATTR_13: "sfde"
  FLAT_ATTR_14: "fdsf"
  FLAT_ATTR_15: "sdgf"
  FLAT_ATTR_16: "sdf"
  FLAT_ATTR_17: "dfdds"
  FLAT_ATTR_18: "sdfds"
  FLAT_ATTR_19: "fds"
  FLAT_ATTR_20: "sfad"
  FLAT_ATTR_21: "fsds"
  FLAT_ATTR_22: "sfa"
  FLAT_ATTR_23: "sfds"
  FLAT_ATTR_24: "dsf"
  FLAT_ATTR_25: "fds"
  FLAT_ATTR_26: "dsfs"
  FLAT_ATTR_27: "sdfds"
  FLAT_ATTR_28: "dsf"
  FLAT_ATTR_29: "sfds"
  FLAT_ATTR_30: "sdfdsd"
  
}){id FLAT_ATTR_5 FLAT_ATTR_1 FLAT_ATTR_2 FLAT_ATTR_3 FLAT_ATTR_4 FLAT_ATTR_6 FLAT_ATTR_7 FLAT_ATTR_8 FLAT_ATTR_9 FLAT_ATTR_10 FLAT_ATTR_11 FLAT_ATTR_12 FLAT_ATTR_13 FLAT_ATTR_14 FLAT_ATTR_15 FLAT_ATTR_16 FLAT_ATTR_17 FLAT_ATTR_18 FLAT_ATTR_19 FLAT_ATTR_20 FLAT_ATTR_21 FLAT_ATTR_22 FLAT_ATTR_23 FLAT_ATTR_24 FLAT_ATTR_25 FLAT_ATTR_26 FLAT_ATTR_27 FLAT_ATTR_28 FLAT_ATTR_29 FLAT_ATTR_30}}`;
    let params = {
        headers: {
            "Content-Type": "application/json",
        },
    };
    // Data for the POST request
    let createOrgData = JSON.stringify({ query: mutation });
    let requests = {
        "New Entity": {
            method: "POST",
            url: endpointUrl,
            params: params,
            body: createOrgData,
        },
    };

    let responses = http.batch(requests);
    let createResp = responses["New Entity"];

    check(createResp, {
        "status is 200": (r) => r.status === 200,
    }) || createErrorRate.add(1);

    CreateTrend.add(createResp.timings.duration);

    check(createResp, {
        "resultCode is 200": (r) => JSON.parse(createResp.body).resultCode === 200,
    }) || createResultErrorRate.add(1);
};

function randomNumberGeneratorEntID(){
console.log("generate a randomn number");
var min = `${__ENV.MIN}`;
var max = `${__ENV.MAX}`;

min = Math.ceil(min);
max = Math.floor(max);
return Math.floor(Math.random() * (max - min + 1)) + min;
};

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
END OF FILE @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

K6 SUCCESS CASE LOG BELOW :


          /\      |‾‾|  /‾‾/  /‾/   
     /\  /  \     |  |_/  /  / /    
    /  \/    \    |      |  /  ‾‾\  
   /          \   |  |‾\  \ | (_) | 
  / __________ \  |__|  \__\ \___/ .io

 Init   [--------------------------------------] runner
 Init   [--------------------------------------] options
 Init   [--------------------------------------] execution scheduler
Init   [--------------------------------------] Init engine
Init   [--------------------------------------] Init metric outputs
Init   [--------------------------------------] Init API server
  execution: local
     script: ./specPerformance/specDynamicEntityForwardSync/specPerfForwardSyncK6JSFiles/PerfTestFlatK6JSFile10.js
     output: -

  scenarios: (100.00%) 1 executors, 100 max VUs, 48h0m30s max duration (incl. graceful stop):
           * default: 10000 iterations shared among 100 VUs (maxDuration: 48h0m0s, gracefulStop: 30s)

Init   [--------------------------------------] Init VUs

Init      [   0% ] 000/100 VUs initialized
default   [   0% ]


Init      [   2% ] 002/100 VUs initialized
default   [   0% ]


Init      [   5% ] 005/100 VUs initialized
default   [   0% ]



Init      [  98% ] 098/100 VUs initialized
default   [   0% ]

Init   [   0% ] Start test

running (0d00h00m00.8s), 100/100 VUs, 0 complete and 0 interrupted iterations
default   [   0% ] 100 VUs  0d00h00m00.4s/48h0m0s  00000/10000 shared iters


running (0d00h00m01.8s), 100/100 VUs, 28 complete and 0 interrupted iterations
default   [   0% ] 100 VUs  0d00h00m01.4s/48h0m0s  00028/10000 shared iters


running (0d00h00m02.8s), 100/100 VUs, 52 complete and 0 interrupted iterations
default   [   1% ] 100 VUs  0d00h00m02.4s/48h0m0s  00052/10000 shared iters

,........

running (0d00h12m56.6s), 000/100 VUs, 10000 complete and 0 interrupted iterations
default ✓ [ 100% ] 100 VUs  0d00h12m55.0s/48h0m0s  10000/10000 shared iters


    ✓ status is 200
    ✗ resultCode is 200
     ↳  0% — ✓ 0 / ✗ 10000

    New Entity.................: avg=1338.975452 min=1.62302 med=681.384226 max=41799.694019 p(90)=3193.63414 p(95)=4970.335521
    New Entity Result errors...: 100.00% ✓ 10000 ✗ 0    
    checks.....................: 50.00%  ✓ 10000 ✗ 10000
    data_received..............: 4.9 MB  6.3 kB/s
    data_sent..................: 14 MB   18 kB/s
    http_req_blocked...........: avg=14.2ms      min=1.08µs  med=4.11µs     max=4.46s        p(90)=10.87µs    p(95)=19.02µs    
    http_req_connecting........: avg=804.99µs    min=0s      med=0s         max=552.92ms     p(90)=0s         p(95)=0s         
    http_req_duration..........: avg=1.33s       min=1.62ms  med=681.38ms   max=41.79s       p(90)=3.19s      p(95)=4.97s      
    http_req_receiving.........: avg=104.32ms    min=18.38µs med=62.54µs    max=35.37s       p(90)=6.49ms     p(95)=426.3ms    
    http_req_sending...........: avg=21.27ms     min=8.84µs  med=24.82µs    max=8.54s        p(90)=48.66µs    p(95)=239.91µs   
    http_req_tls_handshaking...: avg=0s          min=0s      med=0s         max=0s           p(90)=0s         p(95)=0s         
    http_req_waiting...........: avg=1.21s       min=1.52ms  med=625.71ms   max=35.47s       p(90)=2.95s      p(95)=4.23s      
    http_reqs..................: 10000   12.877316/s
    iteration_duration.........: avg=7.68s       min=2.43ms  med=2.53s      max=1m26s        p(90)=23.54s     p(95)=32.88s     
    iterations.................: 10000   12.877316/s
    vus........................: 1       min=0   max=100
    vus_max....................: 100     min=100 max=100

@@@@@@@@@@ FAILURE CASE LOG @@@@@@@@@@@@@@@@@@@@@@@



          /\      |‾‾|  /‾‾/  /‾/   
     /\  /  \     |  |_/  /  / /    
    /  \/    \    |      |  /  ‾‾\  
   /          \   |  |‾\  \ | (_) | 
  / __________ \  |__|  \__\ \___/ .io

 Init   [--------------------------------------] runner
 Init   [--------------------------------------] options
 Init   [--------------------------------------] execution scheduler
Init   [--------------------------------------] Init engine
Init   [--------------------------------------] Init metric outputs
Init   [--------------------------------------] Init API server
  execution: local
     script: ./specPerformance/specDynamicEntityForwardSync/specPerfForwardSyncK6JSFiles/PerfTestFlatK6JSFile1.js
     output: -

  scenarios: (100.00%) 1 executors, 100 max VUs, 48h0m30s max duration (incl. graceful stop):
           * default: 10000 iterations shared among 100 VUs (maxDuration: 48h0m0s, gracefulStop: 30s)

Init   [--------------------------------------] Init VUs

Init      [  10% ] 010/100 VUs initialized
default   [   0% ]


Init      [  20% ] 020/100 VUs initialized
default   [   0% ]


Init      [  30% ] 030/100 VUs initialized
default   [   0% ]


Init      [  42% ] 042/100 VUs initialized
default   [   0% ]


Init      [  52% ] 052/100 VUs initialized
default   [   0% ]


Init      [  58% ] 058/100 VUs initialized
default   [   0% ]


Init      [  74% ] 074/100 VUs initialized
default   [   0% ]


Init      [  83% ] 083/100 VUs initialized
default   [   0% ]


Init      [  87% ] 087/100 VUs initialized
default   [   0% ]


Init      [  91% ] 091/100 VUs initialized
default   [   0% ]


Init      [  95% ] 095/100 VUs initialized
default   [   0% ]


Init      [  98% ] 098/100 VUs initialized
default   [   0% ]

Init   [   0% ] Start test

running (0d00h00m00.8s), 100/100 VUs, 0 complete and 0 interrupted iterations
default   [   0% ] 100 VUs  0d00h00m00.4s/48h0m0s  00000/10000 shared iters


running (0d00h00m01.8s), 100/100 VUs, 28 complete and 0 interrupted iterations
default   [   0% ] 100 VUs  0d00h00m01.4s/48h0m0s  00028/10000 shared iters


running (0d00h00m02.8s), 100/100 VUs, 52 complete and 0 interrupted iterations
default   [   1% ] 100 VUs  0d00h00m02.4s/48h0m0s  00052/10000 shared iters


running (0d00h00m03.8s), 100/100 VUs, 73 complete and 0 interrupted iterations
default   [   1% ] 100 VUs  0d00h00m03.4s/48h0m0s  00073/10000 shared iters


running (0d00h00m04.8s), 100/100 VUs, 96 complete and 0 interrupted iterations
default   [   1% ] 100 VUs  0d00h00m04.4s/48h0m0s  00096/10000 shared iters


running (0d00h00m05.8s), 100/100 VUs, 197 complete and 0 interrupted iterations
default   [   2% ] 100 VUs  0d00h00m05.4s/48h0m0s  00197/10000 shared iters


running (0d00h00m06.8s), 100/100 VUs, 287 complete and 0 interrupted iterations
default   [   3% ] 100 VUs  0d00h00m06.4s/48h0m0s  00287/10000 shared iters


running (0d00h00m07.8s), 100/100 VUs, 375 complete and 0 interrupted iterations
default   [   4% ] 100 VUs  0d00h00m07.4s/48h0m0s  00375/10000 shared iters


running (0d00h00m08.8s), 100/100 VUs, 489 complete and 0 interrupted iterations
default   [   5% ] 100 VUs  0d00h00m08.4s/48h0m0s  00489/10000 shared iters


running (0d00h00m09.8s), 100/100 VUs, 631 complete and 0 interrupted iterations
default   [   6% ] 100 VUs  0d00h00m09.4s/48h0m0s  00631/10000 shared iters


running (0d00h00m10.8s), 100/100 VUs, 782 complete and 0 interrupted iterations
default   [   8% ] 100 VUs  0d00h00m10.4s/48h0m0s  00782/10000 shared iters


running (0d00h00m11.8s), 100/100 VUs, 871 complete and 0 interrupted iterations
default   [   9% ] 100 VUs  0d00h00m11.4s/48h0m0s  00871/10000 shared iters


running (0d00h00m12.8s), 100/100 VUs, 1004 complete and 0 interrupted iterations
default   [  10% ] 100 VUs  0d00h00m12.4s/48h0m0s  01004/10000 shared iters


running (0d00h00m13.8s), 100/100 VUs, 1117 complete and 0 interrupted iterations
default   [  11% ] 100 VUs  0d00h00m13.4s/48h0m0s  01117/10000 shared iters


running (0d00h00m14.8s), 100/100 VUs, 1207 complete and 0 interrupted iterations
default   [  12% ] 100 VUs  0d00h00m14.4s/48h0m0s  01207/10000 shared iters


running (0d00h00m15.8s), 100/100 VUs, 1310 complete and 0 interrupted iterations
default   [  13% ] 100 VUs  0d00h00m15.4s/48h0m0s  01310/10000 shared iters


running (0d00h00m16.8s), 100/100 VUs, 1389 complete and 0 interrupted iterations
default   [  14% ] 100 VUs  0d00h00m16.4s/48h0m0s  01389/10000 shared iters


running (0d00h00m17.8s), 100/100 VUs, 1472 complete and 0 interrupted iterations
default   [  15% ] 100 VUs  0d00h00m17.4s/48h0m0s  01473/10000 shared iters


running (0d00h00m18.8s), 100/100 VUs, 1535 complete and 0 interrupted iterations
default   [  15% ] 100 VUs  0d00h00m18.4s/48h0m0s  01535/10000 shared iters


running (0d00h00m19.8s), 100/100 VUs, 1608 complete and 0 interrupted iterations
default   [  16% ] 100 VUs  0d00h00m19.4s/48h0m0s  01608/10000 shared iters


running (0d00h00m20.8s), 100/100 VUs, 1693 complete and 0 interrupted iterations
default   [  17% ] 100 VUs  0d00h00m20.4s/48h0m0s  01693/10000 shared iters


running (0d00h00m21.8s), 100/100 VUs, 1750 complete and 0 interrupted iterations
default   [  18% ] 100 VUs  0d00h00m21.4s/48h0m0s  01750/10000 shared iters


running (0d00h00m22.8s), 100/100 VUs, 1835 complete and 0 interrupted iterations
default   [  18% ] 100 VUs  0d00h00m22.4s/48h0m0s  01835/10000 shared iters


running (0d00h00m23.8s), 100/100 VUs, 1900 complete and 0 interrupted iterations
default   [  19% ] 100 VUs  0d00h00m23.4s/48h0m0s  01900/10000 shared iters


running (0d00h00m24.8s), 100/100 VUs, 1984 complete and 0 interrupted iterations
default   [  20% ] 100 VUs  0d00h00m24.4s/48h0m0s  01984/10000 shared iters


running (0d00h00m25.8s), 100/100 VUs, 2058 complete and 0 interrupted iterations
default   [  21% ] 100 VUs  0d00h00m25.4s/48h0m0s  02058/10000 shared iters


running (0d00h00m26.8s), 100/100 VUs, 2131 complete and 0 interrupted iterations
default   [  21% ] 100 VUs  0d00h00m26.4s/48h0m0s  02131/10000 shared iters


running (0d00h00m27.8s), 100/100 VUs, 2201 complete and 0 interrupted iterations
default   [  22% ] 100 VUs  0d00h00m27.4s/48h0m0s  02201/10000 shared iters


running (0d00h00m28.8s), 100/100 VUs, 2265 complete and 0 interrupted iterations
default   [  23% ] 100 VUs  0d00h00m28.4s/48h0m0s  02265/10000 shared iters


running (0d00h00m29.8s), 100/100 VUs, 2353 complete and 0 interrupted iterations
default   [  24% ] 100 VUs  0d00h00m29.4s/48h0m0s  02353/10000 shared iters


running (0d00h00m30.8s), 100/100 VUs, 2406 complete and 0 interrupted iterations
default   [  24% ] 100 VUs  0d00h00m30.4s/48h0m0s  02406/10000 shared iters


running (0d00h00m31.8s), 100/100 VUs, 2464 complete and 0 interrupted iterations
default   [  25% ] 100 VUs  0d00h00m31.4s/48h0m0s  02464/10000 shared iters


running (0d00h00m32.8s), 100/100 VUs, 2514 complete and 0 interrupted iterations
default   [  25% ] 100 VUs  0d00h00m32.4s/48h0m0s  02514/10000 shared iters


running (0d00h00m33.8s), 100/100 VUs, 2577 complete and 0 interrupted iterations
default   [  26% ] 100 VUs  0d00h00m33.4s/48h0m0s  02577/10000 shared iters


running (0d00h00m34.8s), 100/100 VUs, 2624 complete and 0 interrupted iterations
default   [  26% ] 100 VUs  0d00h00m34.4s/48h0m0s  02624/10000 shared iters


running (0d00h00m35.8s), 100/100 VUs, 2656 complete and 0 interrupted iterations
default   [  27% ] 100 VUs  0d00h00m35.4s/48h0m0s  02656/10000 shared iters


running (0d00h00m36.8s), 100/100 VUs, 2720 complete and 0 interrupted iterations
default   [  27% ] 100 VUs  0d00h00m36.4s/48h0m0s  02720/10000 shared iters


running (0d00h00m37.8s), 100/100 VUs, 2742 complete and 0 interrupted iterations
default   [  27% ] 100 VUs  0d00h00m37.4s/48h0m0s  02742/10000 shared iters


running (0d00h00m38.8s), 100/100 VUs, 2799 complete and 0 interrupted iterations
default   [  28% ] 100 VUs  0d00h00m38.4s/48h0m0s  02799/10000 shared iters


running (0d00h00m39.8s), 100/100 VUs, 2843 complete and 0 interrupted iterations
default   [  28% ] 100 VUs  0d00h00m39.4s/48h0m0s  02843/10000 shared iters


running (0d00h00m40.8s), 100/100 VUs, 2873 complete and 0 interrupted iterations
default   [  29% ] 100 VUs  0d00h00m40.4s/48h0m0s  02873/10000 shared iters


running (0d00h00m41.8s), 100/100 VUs, 2932 complete and 0 interrupted iterations
default   [  29% ] 100 VUs  0d00h00m41.4s/48h0m0s  02932/10000 shared iters


running (0d00h00m42.8s), 100/100 VUs, 2971 complete and 0 interrupted iterations
default   [  30% ] 100 VUs  0d00h00m42.4s/48h0m0s  02971/10000 shared iters


running (0d00h00m43.8s), 100/100 VUs, 2979 complete and 0 interrupted iterations
default   [  30% ] 100 VUs  0d00h00m43.4s/48h0m0s  02979/10000 shared iters


running (0d00h00m44.8s), 100/100 VUs, 3069 complete and 0 interrupted iterations
default   [  31% ] 100 VUs  0d00h00m44.4s/48h0m0s  03069/10000 shared iters


running (0d00h00m45.8s), 100/100 VUs, 3084 complete and 0 interrupted iterations
default   [  31% ] 100 VUs  0d00h00m45.4s/48h0m0s  03084/10000 shared iters


running (0d00h00m46.8s), 100/100 VUs, 3113 complete and 0 interrupted iterations
default   [  31% ] 100 VUs  0d00h00m46.4s/48h0m0s  03113/10000 shared iters


running (0d00h00m47.8s), 100/100 VUs, 3158 complete and 0 interrupted iterations
default   [  32% ] 100 VUs  0d00h00m47.4s/48h0m0s  03158/10000 shared iters


running (0d00h00m48.8s), 100/100 VUs, 3181 complete and 0 interrupted iterations
default   [  32% ] 100 VUs  0d00h00m48.4s/48h0m0s  03181/10000 shared iters


running (0d00h00m49.8s), 100/100 VUs, 3201 complete and 0 interrupted iterations
default   [  32% ] 100 VUs  0d00h00m49.4s/48h0m0s  03201/10000 shared iters


running (0d00h00m50.8s), 100/100 VUs, 3244 complete and 0 interrupted iterations
default   [  32% ] 100 VUs  0d00h00m50.4s/48h0m0s  03244/10000 shared iters


running (0d00h00m51.8s), 100/100 VUs, 3275 complete and 0 interrupted iterations
default   [  33% ] 100 VUs  0d00h00m51.4s/48h0m0s  03275/10000 shared iters


running (0d00h00m52.8s), 100/100 VUs, 3307 complete and 0 interrupted iterations
default   [  33% ] 100 VUs  0d00h00m52.4s/48h0m0s  03307/10000 shared iters


running (0d00h00m53.8s), 100/100 VUs, 3340 complete and 0 interrupted iterations
default   [  33% ] 100 VUs  0d00h00m53.4s/48h0m0s  03340/10000 shared iters


running (0d00h00m54.8s), 100/100 VUs, 3364 complete and 0 interrupted iterations
default   [  34% ] 100 VUs  0d00h00m54.4s/48h0m0s  03364/10000 shared iters


running (0d00h00m55.8s), 100/100 VUs, 3409 complete and 0 interrupted iterations
default   [  34% ] 100 VUs  0d00h00m55.4s/48h0m0s  03409/10000 shared iters


running (0d00h00m56.8s), 100/100 VUs, 3447 complete and 0 interrupted iterations
default   [  34% ] 100 VUs  0d00h00m56.4s/48h0m0s  03447/10000 shared iters


running (0d00h00m57.8s), 100/100 VUs, 3470 complete and 0 interrupted iterations
default   [  35% ] 100 VUs  0d00h00m57.4s/48h0m0s  03470/10000 shared iters


running (0d00h00m58.8s), 100/100 VUs, 3484 complete and 0 interrupted iterations
default   [  35% ] 100 VUs  0d00h00m58.4s/48h0m0s  03484/10000 shared iters


running (0d00h00m59.8s), 100/100 VUs, 3504 complete and 0 interrupted iterations
default   [  35% ] 100 VUs  0d00h00m59.4s/48h0m0s  03504/10000 shared iters


running (0d00h01m00.8s), 100/100 VUs, 3562 complete and 0 interrupted iterations
default   [  36% ] 100 VUs  0d00h01m00.4s/48h0m0s  03562/10000 shared iters


running (0d00h01m01.8s), 100/100 VUs, 3581 complete and 0 interrupted iterations
default   [  36% ] 100 VUs  0d00h01m01.4s/48h0m0s  03581/10000 shared iters


running (0d00h01m02.8s), 100/100 VUs, 3599 complete and 0 interrupted iterations
default   [  36% ] 100 VUs  0d00h01m02.4s/48h0m0s  03599/10000 shared iters


running (0d00h01m03.8s), 100/100 VUs, 3628 complete and 0 interrupted iterations
default   [  36% ] 100 VUs  0d00h01m03.4s/48h0m0s  03628/10000 shared iters


running (0d00h01m04.8s), 100/100 VUs, 3657 complete and 0 interrupted iterations
default   [  37% ] 100 VUs  0d00h01m04.4s/48h0m0s  03657/10000 shared iters


running (0d00h01m05.8s), 100/100 VUs, 3688 complete and 0 interrupted iterations
default   [  37% ] 100 VUs  0d00h01m05.4s/48h0m0s  03688/10000 shared iters


running (0d00h01m06.8s), 100/100 VUs, 3710 complete and 0 interrupted iterations
default   [  37% ] 100 VUs  0d00h01m06.4s/48h0m0s  03710/10000 shared iters


running (0d00h01m07.8s), 100/100 VUs, 3735 complete and 0 interrupted iterations
default   [  37% ] 100 VUs  0d00h01m07.4s/48h0m0s  03735/10000 shared iters


running (0d00h01m08.8s), 100/100 VUs, 3759 complete and 0 interrupted iterations
default   [  38% ] 100 VUs  0d00h01m08.4s/48h0m0s  03759/10000 shared iters


running (0d00h01m10.3s), 100/100 VUs, 3761 complete and 0 interrupted iterations
default   [  38% ] 100 VUs  0d00h01m09.9s/48h0m0s  03761/10000 shared iters


running (0d00h01m12.0s), 100/100 VUs, 3763 complete and 0 interrupted iterations
default   [  38% ] 100 VUs  0d00h01m11.6s/48h0m0s  03763/10000 shared iters


running (0d00h01m14.4s), 100/100 VUs, 3767 complete and 0 interrupted iterations
default   [  38% ] 100 VUs  0d00h01m14.0s/48h0m0s  03767/10000 shared iters


running (0d00h01m16.3s), 100/100 VUs, 3773 complete and 0 interrupted iterations
default   [  38% ] 100 VUs  0d00h01m15.9s/48h0m0s  03773/10000 shared iters


running (0d00h01m20.9s), 100/100 VUs, 3789 complete and 0 interrupted iterations
default   [  38% ] 100 VUs  0d00h01m20.5s/48h0m0s  03789/10000 shared iters


running (0d00h01m30.6s), 100/100 VUs, 3832 complete and 0 interrupted iterations
default   [  38% ] 100 VUs  0d00h01m30.2s/48h0m0s  03832/10000 shared iters


running (0d00h01m35.0s), 100/100 VUs, 3854 complete and 0 interrupted iterations
default   [  39% ] 100 VUs  0d00h01m34.5s/48h0m0s  03854/10000 shared iters


running (0d00h01m35.5s), 100/100 VUs, 3895 complete and 0 interrupted iterations
default   [  39% ] 100 VUs  0d00h01m35.0s/48h0m0s  03895/10000 shared iters


running (0d00h01m35.8s), 100/100 VUs, 3897 complete and 0 interrupted iterations
default   [  39% ] 100 VUs  0d00h01m35.4s/48h0m0s  03897/10000 shared iters


running (0d00h01m36.8s), 100/100 VUs, 3908 complete and 0 interrupted iterations
default   [  39% ] 100 VUs  0d00h01m36.4s/48h0m0s  03908/10000 shared iters


running (0d00h01m37.8s), 100/100 VUs, 3914 complete and 0 interrupted iterations
default   [  39% ] 100 VUs  0d00h01m37.4s/48h0m0s  03914/10000 shared iters


running (0d00h01m40.4s), 100/100 VUs, 3966 complete and 0 interrupted iterations
default   [  40% ] 100 VUs  0d00h01m40.0s/48h0m0s  03966/10000 shared iters


running (0d00h01m40.8s), 100/100 VUs, 3984 complete and 0 interrupted iterations
default   [  40% ] 100 VUs  0d00h01m40.4s/48h0m0s  03984/10000 shared iters


running (0d00h01m41.8s), 100/100 VUs, 4015 complete and 0 interrupted iterations
default   [  40% ] 100 VUs  0d00h01m41.4s/48h0m0s  04015/10000 shared iters


running (0d00h01m42.8s), 100/100 VUs, 4025 complete and 0 interrupted iterations
default   [  40% ] 100 VUs  0d00h01m42.4s/48h0m0s  04025/10000 shared iters


running (0d00h01m43.8s), 100/100 VUs, 4039 complete and 0 interrupted iterations
default   [  40% ] 100 VUs  0d00h01m43.4s/48h0m0s  04039/10000 shared iters


running (0d00h01m44.8s), 100/100 VUs, 4055 complete and 0 interrupted iterations
default   [  41% ] 100 VUs  0d00h01m44.4s/48h0m0s  04055/10000 shared iters


running (0d00h01m45.8s), 100/100 VUs, 4063 complete and 0 interrupted iterations
default   [  41% ] 100 VUs  0d00h01m45.4s/48h0m0s  04063/10000 shared iters


running (0d00h01m46.8s), 100/100 VUs, 4112 complete and 0 interrupted iterations
default   [  41% ] 100 VUs  0d00h01m46.4s/48h0m0s  04112/10000 shared iters


running (0d00h01m47.8s), 100/100 VUs, 4133 complete and 0 interrupted iterations
default   [  41% ] 100 VUs  0d00h01m47.4s/48h0m0s  04133/10000 shared iters


running (0d00h01m48.8s), 100/100 VUs, 4140 complete and 0 interrupted iterations
default   [  41% ] 100 VUs  0d00h01m48.4s/48h0m0s  04140/10000 shared iters


running (0d00h01m49.8s), 100/100 VUs, 4146 complete and 0 interrupted iterations
default   [  41% ] 100 VUs  0d00h01m49.4s/48h0m0s  04146/10000 shared iters


running (0d00h01m50.8s), 100/100 VUs, 4185 complete and 0 interrupted iterations
default   [  42% ] 100 VUs  0d00h01m50.4s/48h0m0s  04185/10000 shared iters


running (0d00h01m51.8s), 100/100 VUs, 4213 complete and 0 interrupted iterations
default   [  42% ] 100 VUs  0d00h01m51.4s/48h0m0s  04213/10000 shared iters


running (0d00h01m52.8s), 100/100 VUs, 4230 complete and 0 interrupted iterations
default   [  42% ] 100 VUs  0d00h01m52.4s/48h0m0s  04230/10000 shared iters


running (0d00h01m53.8s), 100/100 VUs, 4257 complete and 0 interrupted iterations
default   [  43% ] 100 VUs  0d00h01m53.4s/48h0m0s  04257/10000 shared iters


running (0d00h01m54.8s), 100/100 VUs, 4279 complete and 0 interrupted iterations
default   [  43% ] 100 VUs  0d00h01m54.4s/48h0m0s  04279/10000 shared iters


running (0d00h01m55.9s), 100/100 VUs, 4282 complete and 0 interrupted iterations
default   [  43% ] 100 VUs  0d00h01m55.4s/48h0m0s  04282/10000 shared iters


running (0d00h01m56.8s), 100/100 VUs, 4303 complete and 0 interrupted iterations
default   [  43% ] 100 VUs  0d00h01m56.4s/48h0m0s  04303/10000 shared iters


running (0d00h01m57.8s), 100/100 VUs, 4331 complete and 0 interrupted iterations
default   [  43% ] 100 VUs  0d00h01m57.4s/48h0m0s  04331/10000 shared iters


running (0d00h01m58.8s), 100/100 VUs, 4370 complete and 0 interrupted iterations
default   [  44% ] 100 VUs  0d00h01m58.4s/48h0m0s  04370/10000 shared iters


running (0d00h02m01.6s), 100/100 VUs, 4375 complete and 0 interrupted iterations
default   [  44% ] 100 VUs  0d00h02m01.2s/48h0m0s  04375/10000 shared iters


running (0d00h02m04.8s), 100/100 VUs, 4375 complete and 0 interrupted iterations
default   [  44% ] 100 VUs  0d00h02m04.4s/48h0m0s  04375/10000 shared iters


running (0d00h02m07.0s), 100/100 VUs, 4381 complete and 0 interrupted iterations
default   [  44% ] 100 VUs  0d00h02m06.6s/48h0m0s  04381/10000 shared iters

@@@@@@@@@ END OF TOPIC @@@@@@@@@@@@@

mstoykov · November 3, 2020, 4:15pm

HI @gopinath, welcome to the forum,

Looking at the way you’ve posted the logs it seems like this is only the stdout not the stderr where things such as errors will be printed. From what you’ve written I would expect that you either redirect the output to some file or something like that.

Please either redirect the stderr as well or better try to run it locally and see what happens, without all the other parallel k6 runs (as that is unlikely to has anything to do with it).

I would expect that you might be running out of memory so looking into how much memory it uses on the machines will probably be beneficial.

gopinath · November 24, 2020, 7:35am

Thanks @mstoykov for your response on this… First thing here is , i could not find a way to capture std_err logs for my above runs and neither i could see in K6 documentation on how to specifically capture std_err logs as part of any K6 runs…

Secondly , with regards to going out of memory, we could try with an linux box which is having better memory resource [ 8GB ] , but ran into same issue…Please note that default min and max JVM memory setting was not changed… [ java -Xms256m -Xmx512m] …

Can you please confirm us how does the memory allocation happen when we are simultaneously running multiple instances of k6 data generation instances… [ We have a requirement of order or 30 to 50 K6 instances running in parallel ]… Is it because we did not change the above min and max value to higher order [ say java -Xms256m -Xmx4096m ] , we ran into same issue though this time though we had a linux box with better memory resource… Do you thing if we set higher min and max value our issue of K6 run instances abruptly terminating should be resolved ? Is there any other options or configurations in K6 to handle above issue…Any inputs here would be of great help and highly appreciated …Thanks …Cheers.

mstoykov · November 24, 2020, 9:17am

Hi @gopinath,

stderr is captured as in any other case, here some explanation written by someone else, you can arguably google for more if you want to. You can also check that the kernel didn’t kill it through something like dmesg

K6 is not … based on Java, and Javascript (not with a space between it) has nothing to with Java … or just as much as both have to do with C++ (for example). As such the reason that you ran out of memory is that k6 tried to use some and your OS couldn’t give it any - because it ran out.

It could also be something else, but until you can confirm what is happening, running out of memory seems like the most expected thing …

I suspect you want to run 30-50 instances on different machines, in order to scale better? If so I will recommend:

going with 1 big machine instead - 50 x 100 = 5000 VUs which is perfectly fine for one machine with enough ram. You can read this guide on running large tests.
going with k6 cloud - it will save you the next big issues that you are going to have “how to visualize the results” and would probably be cheaper once you take into account the human hours required. For 48h scripts I would expect that we will give you some custom pricing depending on a lot of additional stuff, so it might be even cheaper than you think. Disclaimer: I do work for the company, and I am not a sales rep, so I might be wrong, but with a couple of emails you will probably know what the price will be.

Topic		Replies	Views
[fatal error] k6 memory leak OSS Support	2	1502	August 6, 2019
After running 600 RPS in linux, after 15 mins it is getting killed (out of memory) OSS Support	4	750	December 11, 2021
Modify run max duration OSS Support	3	2591	May 20, 2021
K6 is memory hungry when using several modules OSS Support	39	1745	June 19, 2023
Out of memory K6 OSS Support	8	148	June 21, 2023

Issue while parallel data generation using K6 [ Data generation terminating abruptly ]

Related topics