How to Understand: Sleep and Ramping Up VUs

import http from 'k6/http';
import { check, group, sleep, fail } from 'k6';
export let options = {
  vus: 1000,
iterations: 1000,
 // duration: '3m',
        //stages: [{ target: 1000, duration: '5m'  }],
  //thresholds: {
  //http_req_duration: ['p(95)<500', 'p(99)<1500'],
  //'http_req_duration{name:PublicCrocs}': ['avg<400'],
  //'http_req_duration{name:Create}': ['avg<600', 'max<1000'],
  //},
};

// TM - global vars object for storing authToken (and whatever else you want)
const vars = {};
const PASSWORD = 'test123';
const BASE_URL = 'https://dev.target.org';

export default function main() {
  //let uniqueNumber = __VU * 20 + __ITER; //setting up unique VUs
  //This flow is intended that unique VUs each carry an email with same Password
  //1) Register
  //2) Login with email and Password
  //3) Token is grabbed
  //4) Joining the group with the token no need user credentials here,
  //  since the token from the logged in user itself is enough to login
  //I am looping here so that I create individual email IDs as usernames
  //and a standard pass to register
  //for (var id = 1; id <= 20; id++) {
  //const USERNAME = `ithayam_${id}@target.org`//[uniqueNumber]; //user_id_${id}@heartmath.com
  registerAndLogin();
  sleep(4);
 joinGroup();
  sleep(7);
}

function registerAndLogin() {
  const USERNAME = `ithayam_${__VU}${__ITER}${Date.now()}@target.org`;
  //const PASSWORD = 'test123';
  //vars["BASE_URL"] = 'https://dev.target.org';
  let data = {
    "email": `${USERNAME}`,
    "password": `${PASSWORD}`,
    "confirm_password": `${PASSWORD}`,
    "first_name": "Real",
    "last_name": "Original",
    "coherenceGoal": "300",
    "isUnder16": "false",
    "opt_in": "true",
    "is_admin": "false",
    "inspiration": "false",
    "offers": "false",
    "birthday": "2000-11-26",
    "parental_email": "test@realoriginal.in"
  };
  //console.log(` checking for USERNAME ${USERNAME}`);
  //console.log(` checking for PASSWORD ${PASSWORD}`);
  //console.log(` checking for BASE_URL ${BASE_URL}`);
  // register a new user
  /*let handleResponse = function(resObj) {
    console.log("HAndle response")
    console.log(resObj)
  }*/

  let res;
  res = http.post(`${BASE_URL}/api/v1/register`, JSON.stringify(data),
    {
      headers: {
        "Content-Type": "application/json",
        Accept: "*/*",
        Host: "dev.target.org",
        Connection: "keep-alive",
      },
     //responseCallback: handleResponse
    }
  );
  //console.log(` Request-body: ${res.request.body}`);
  //console.log(` checking response code/token after reg: ${res.status} ${res.body}`);
  //check(res, { 'auth_token': (r) => r.status === 200 }); //201 ==> 200, there's no success message so I am using the response token variable to check the success
  sleep(5);
//Login and grab token
  let loginRes;
  loginRes = http.post(`${BASE_URL}/api/v1/login`,
    JSON.stringify({
      "email": `${USERNAME}`,
      "password": `${PASSWORD}`
    }),
    {
      headers: {
       Accept: "*/*",
       "Content-Type": "application/json",
       Host: "dev.target.org",
        Connection: "keep-alive",
       "User-Agent": "Mozilla/5.0",
     },
   }
  );

  //console.log(` Request-body: ${loginRes.request.body}`);
  //console.log(` checking for auth_token after login ${loginRes.status} ${loginRes.body}`);
  //let authToken = loginRes.json('data');
  let authToken = loginRes.json('data').auth_token;
  check(authToken, { 'auth_token': () => authToken !== '' });


  // TM - no need to return authToken; just set it in a global object
  //return authToken;
  vars["authToken"] = authToken;
 // console.log(` AT : ${JSON.stringify(vars["authToken"])}`);
}

function joinGroup() {
  let joinRes = http.get(`${BASE_URL}/api/v1/auth/group/join/1991`,
    //null, // empty response body?  -- I commented out this
  {
    headers: {
      Authorization: `Token ${vars["authToken"]}`,
    },
   }
  );

  //console.log(` Join Response-headers: ${JSON.stringify(joinRes.headers)}`);
 // console.log(` checking after Join: ${joinRes.status} ${joinRes.body}`);
  console.log(` JoinGroup-stats : ${joinRes.status}`);
}

have couple of questions regarding the above script

  1. To simulate a realistic approach of 1000 users registering at the same time, login in etc(the overall flow): how must I setup/ramp-up the VUs.
  2. The sleep commands that I have used, do they actually sleep because I dont see that they are causing actual sleep when running the program. The reason I have implemented longer sleep because I often get timeouts from the server.
  1. I guess what you’ve done now is actually exactly this - 1000 VUs doing in total 1000 iterations (in your case this likely will mean everyone will do 1 iteration, but I would recommend using per-vu-iterations executor in most cases :wink: ). I wouldn’t consider this to be interesting case for most users as likely you will never see 1k real users doing one thing once with absolutely nothing else but if this is your usecase go for it ;). I would probably do a longer load with an arrival rate and decided that every second 50 users will start registering , so I will see 1000 users start registering in 1000/50 = 200seconds or 3 minutes 20 seconds and I will run it for 10 minutes or something like that. Again you might really have a case that once in a while 1k users will start exactly at the same time, but IMO this is fairly rare pattern :wink:
  2. sleep does block the execution of the particular VU for the duration of the sleep. Also if you are getting timeouts likely your users will get them as well or at least end up waiting a very long time

Thank you for the prompt response @mstoykov: I followed the per-vu-iterations links you suggested and I used the above example into my script. FYI the flow is like this, the VUs register, login, grab the token from login and use it to join the group url. What happens is that 1k users got registered and nothing else happened.

But when I use the setup

export let options = {
  vus: 1000,
iterations: 1000,
};

all the users got registered, login and around 700+ users join the group. But I expect 1k users to join the group. I think I can tinker that on the server side.

But my question is why didnt the per-vu-iteration didnt go beyond the registration flow?
And
Could you please show me how the code would be when you meant this

I wouldn’t consider this to be interesting case for most users as likely you will never see 1k real users doing one thing once with absolutely nothing else but if this is your usecase go for it ;). I would probably do a longer load with an arrival rate and decided that every second 50 users will start registering , so I will see 1000 users start registering in 1000/50 = 200seconds or 3 minutes 20 seconds and I will run it for 10 minutes or something like that

But my question is why didnt the per-vu-iteration didnt go beyond the registration flow?

Without more information don’t think I can answer that. Maybe:

  1. you ran out of the default maxDuration of 10 minutes, at which point I would consider this pretty bad
  2. there was some exception which will be seen in the terminal
  3. Maybe the calls were made but you got not positive responses, at which point there should be some indication of that as well.

Could you please show me how the code would be when you meant this

I would recommend reading the arrival rate documentation for more understanding of the concepts and what is going on, but something like the below configuration is what I meant.

export let options =  {
  scenarios: {
    "somename": {
      executor: "constant-arrival-rate",
      rate: "50", // how many iterations per second are started
      preallocatedVUs: 1000, // this should be smaller if your iterations are faster, it is just how many JS VMs will use
      duration: "3m20s" // how long to test, this should give you 1000 iterations by my calculations
    }
  }
}

@mstoykov thanks for the prompt response, I applied your sample code and also applied based on the arrival rate documentation, but it executes only one function which is registerAndLogin only, instead it should be sequential: the main function should first call registerAndLogin() - once the VUs finish this function and they then go this function - joinGroup()

I tried running your original script(with changed URL and deleted the auth_token part) and it does work as expected, but if I didn’t delete the auth_token I would get an error that there is no such thing in the response, as I am not hitting an endpoint returning such a response ;), and that exception will bubble up and stop the given iteration.

Do you get exceptions thrown when you run it? Is there anything on the terminal output and do you change any of the logging configurations?

@mstoykov Thanks again for the prompt response. Please find attached the errors(3 screenshots in the order of arrival) I receive at run time. This is my modified code so far

import http from 'k6/http';
import { check, group, sleep, fail } from 'k6';
export let options = {
 discardResponseBodies: true,
  scenarios: {
    contacts: {
      executor: 'shared-iterations',
      maxDuration: '5m20s',
      iterations: 1000,
      vus: 1000
    },
  },
 //vus: 1000,
//iterations: 1000,
 // duration: '3m',
        //stages: [{ target: 1000, duration: '5m'  }],
  //thresholds: {
  //http_req_duration: ['p(95)<500', 'p(99)<1500'],
  //'http_req_duration{name:PublicCrocs}': ['avg<400'],
  //'http_req_duration{name:Create}': ['avg<600', 'max<1000'],
  //},
};

// TM - global vars object for storing authToken (and whatever else you want)
const vars = {};
const PASSWORD = 'test123';
const BASE_URL = 'https://target.org';

export default function main() {
  //let uniqueNumber = __VU * 20 + __ITER; //setting up unique VUs
  //This flow is intended that unique VUs each carry an email with same Password
  //1) Register
  //2) Login with email and Password
  //3) Token is grabbed
  //4) Joining the group with the token no need user credentials here,
  //  since the token from the logged in user itself is enough to login
  //I am looping here so that I create individual email IDs as usernames
  //and a standard pass to register
  //for (var id = 1; id <= 20; id++) {
  //const USERNAME = `ithayam_${id}@target.com`//[uniqueNumber]; //user_id_${id}@target.com
  registerAndLogin();
  //sleep(1);
 joinGroup();
  //sleep(1);
}

function registerAndLogin() {
  const USERNAME = `ithayam_${__VU}${__ITER}${Date.now()}@target.com`;
  //const PASSWORD = 'test123';
  //vars["BASE_URL"] = 'https://target.org';
  let data = {
    "email": `${USERNAME}`,
    "password": `${PASSWORD}`,
    "confirm_password": `${PASSWORD}`,
    "first_name": "Real",
    "last_name": "Original",
    "coherenceGoal": "300",
    "isUnder16": "false",
    "opt_in": "true",
    "is_admin": "false",
    "inspiration": "false",
    "offers": "false",
    "birthday": "2000-11-26",
    "parental_email": "test@realoriginal.in"
  };
  //console.log(` checking for USERNAME ${USERNAME}`);
  //console.log(` checking for PASSWORD ${PASSWORD}`);
  //console.log(` checking for BASE_URL ${BASE_URL}`);
  // register a new user
  /*let handleResponse = function(resObj) {
    console.log("HAndle response")
    console.log(resObj)
  }*/

  let res;
  res = http.post(`${BASE_URL}/api/v1/register`, JSON.stringify(data),
    {
      headers: {
        "Content-Type": "application/json",
        Accept: "*/*",
        Host: "target.org",
        Connection: "keep-alive",
      },
     //responseCallback: handleResponse
    }
  );
  //console.log(` Request-body: ${res.request.body}`);
  //console.log(` checking response code/token after reg: ${res.status} ${res.body}`);
  //check(res, { 'auth_token': (r) => r.status === 200 }); //201 ==> 200, there's no success message so I am using the response token variable to check the success
  sleep(5);
//Login and grab token
  let loginRes;
  loginRes = http.post(`${BASE_URL}/api/v1/login`,
    JSON.stringify({
      "email": `${USERNAME}`,
      "password": `${PASSWORD}`
    }),
    {
      headers: {
       Accept: "*/*",
       "Content-Type": "application/json",
       Host: "gca-dev.heartmath.org",
        Connection: "keep-alive",
       "User-Agent": "Mozilla/5.0",
     },
   }
  );

  //console.log(` Request-body: ${loginRes.request.body}`);
  //console.log(` checking for auth_token after login ${loginRes.status} ${loginRes.body}`);
  //let authToken = loginRes.json('data');
  let authToken = loginRes.json('data').auth_token;
  check(authToken, { 'auth_token': () => authToken !== '' });


  // TM - no need to return authToken; just set it in a global object
  //return authToken;
  vars["authToken"] = authToken;
 // console.log(` AT : ${JSON.stringify(vars["authToken"])}`);
}

function joinGroup() {
  let joinRes = http.get(`${BASE_URL}/api/v1/auth/group/join/1991`,
    //null, // empty response body?  -- I commented out this
  {
    headers: {
      Authorization: `Token ${vars["authToken"]}`,
    },
   }
  );

  //console.log(` Join Response-headers: ${JSON.stringify(joinRes.headers)}`);
 // console.log(` checking after Join: ${joinRes.status} ${joinRes.body}`);
  console.log(` JoinGroup-stats : ${joinRes.status}`);
}



This confirms my suspicion that one of the previous requests fails and then the code throws an exception as a response is expected.

In your case the exception is being thrown on

  let authToken = loginRes.json('data').auth_token;

As the login requests timeouts and the response body is null but you try to use it either way. The message can definitely do with some work though :thinking: .

Given this I would argue that you have a result : your API is so slow that multiple users will be timing out on registering/loging so at this point this needs some optimization :). You can also check that you got a workable result before trying to use it, but in this case this will just hide that your tests is failing.

@mstoykov This is where my confusion begins,
when I use (executor: "constant-arrival-rate") as seen below,

export let options =  {
  scenarios: {
    "somename": {
      executor: "constant-arrival-rate",
      rate: "50", // how many iterations per second are started
      preallocatedVUs: 1000, // this should be smaller if your iterations are faster, it is just how many JS VMs will use
      duration: "3m20s" // how long to test, this should give you 1000 iterations by my calculations
    }
  }
}

I get the below result. Some users joined the group BUT if you see the iterations part, it went to 1080, what happens here is it started creating 80 more new users on top of 1000. My target is 1 full iteration per user meaning that every user must register, login and join group. Lets say some users failed to join group then how can I increase the “re-trial” attempt for those particular users to re-join the group

And,
when I use this VU setup

export let options = {
  vus: 1000,
iterations: 1000,
};

I get all the users successfully(99%) registered, logged in and joined the group and also 1 user performed 1 full iteration of (register>login>join-group)


In the above pic what does “vus 15 min=15 max=200” what does this mean. I understand that the max is 200 VUs but what about min=15?
also what does the below mean

_ auth_token
 _  99% _ _ 1197 / _ 1

and
http requests 1798 ?

I also remember you mentioned to me that the above approach is not feasible and you suggested me to use the approach with constant-arrival-rate but its only when I use it gives a different result which I have posted. Kindly help me understand the underlying logic.

Sorry for all the questions but I want to understand it like the back of my hand so I can perform tests better.

BUT if you see the iterations part, it went to 1080,

Looking at the screenshot provided you are clearly having the duration at 10m20s and you end at 3m35s so I would expect that you just copied the options wrongly or are running a different script with 9m50s as a duration (I expect 30s of added gracefulStop) and just stopped the test manually to get the screenshot.

. Lets say some users failed to join group then how can I increase the “re-trial” attempt for those particular users to re-join the group

You can always just check the response and whether its status is 200/201 or w/e you expect + some other checks and do the request again, until you actually succeed.

I would argue that given the current results the system under test has performance problems and it’s likely better to work on those, not on this test, after all it’s in general the system under test that has to be improved :slight_smile:

I get all the users successfully(99%) registered, logged in and joined the group and also 1 user performed 1 full iteration of (register>login>join-group)

Do you mean that only 1 iteration got through the whole iteration or that 1 didn’t?

In the above pic what does “vus 15 min=15 max=200” what does this mean. I understand that the max is 200 VUs but what about min=15?

vus is a Counter and as every counter in k6 it gets to be shown the latest value for it as well as the minimal and the maximum value it had for the run. Also important is the the vus value (as reported as a metric) is sampled every 2 seconds - k6 checks what the value is every 2 seconds and report that. So given that the minimal and maximum values for that … might be off (although with the current way things are working, just the minimal can be … I think). I would expect that the 15 is because while vus were finishing up their last iterations at some near the end point there were 15 vus still running and 2s later the test has ended, so there was not another sample with 1 for example.

also what does the below mean

_ auth_token
 _  99% _ _ 1197 / _ 1

and
http requests 1798 ?

I would expect that you had 1198 checks call for auth_token one of which was a failure (I don’t know why for you the ticks and crosses aren’t working, are you using a windows terminal?) and there were 1798 - 1198 other requests, which given the script should be a lot more, so I would guess you also dropped the registration requests in that run, and those are just the group joining ones.

I also remember you mentioned to me that the above approach is not feasible and you suggested me to use the approach with constant-arrival-rate but its only when I use it gives a different result which I have posted. Kindly help me understand the underlying logic.

I don’t think I have said it’s not feasible, just that IMO it’s not a thing that will emulate a real situation, but you can definitely do it.

About the different results - it’s generally doing a different thing, but in either way from at least this results, again, I would argue the system under test is in a very bad spot performance-wise, and should be optimized. The k6 test testing can be improved as well, but I would argue you don’t need anything better than the current one to show that the system under test can’t handle registration+login+joining a group without a user experiencing upwards of 60 seconds response times, which at least IMO will be more than a couple of times more time than any user will wait.

Yes you’re right, I just ran the script to send you the screenshot of the error I am facing and also show the result at the moment it was stopped

Yes thats also something we are working on but I had to make sure the script was 110% ready because the script is as strong as the user coding it(which is my understanding on how k6 works) thats why came up with this support thread.

I meant here that I was happy with the result even if it failed with 1 user, which I can assume that its something to do on the server side and update accordingly.

I am using Window10 powershell themed terminal, usually it shows but I dont know why it didnt show this time.

Ok then we shall continue optimizing the backend/API and see how the test fairs in the future runs.

I truly appreciate every bit of information you have share here @mstoykov