I am evaluating k6 for automated performance regression testing of a product I work on. It looks pretty good, but I ran into some quirks.
The first problem that I ran into is that we have dozens of tests that need to be run sequentially, with detailed histogram metrics aggregated for each individual test.
Scenarios are mentioned as a solution, but they have two issues:
- Scenarios don’t seem adequately cover this type of execution right now, not just because it’s impossible to specify sequential ordering (without using
startTime) - Also, the cloud UI doesn’t appear to not distinguish between scenarios at all. In other words, scenarios aren’t “test runs”, they’re just phases of a test.
So the only option appears to be to run one k6 test per test.
The second problem that I ran into is that the pricing model appears to assume you run relatively few test runs. With dozens of tests that each run on every CI push, we would exceed the limit very quickly. We could reasonably compromise here and run the tests just once a day per active branch, but with dozens of tests, we’d still exceed that limit. For example, assume 20 tests and that our product is being actively worked on daily. That’s perhaps 5,000 runs per year, more than twice the limit for Team/Pro plans. (Custom/Enterprise is not on the table, sorry!) Maybe we could dial it down further and only run tests for each pull request and mainline merge, but it seems a bit silly to be so arbitrarily restrictive.
Ideally, I’d like to go into the performance results for any given GitHub commit and compare it with a different GitHub commit.
Another thing I am curious about is that the only result view offered is a timeline. Seeing request rates over time can be useful, I guess, and we do want to store “trends” like GC pause times where a timeline can be a nice visualization. But a 1-100th percentile latency histogram would be much more useful, and tells you real things about your data.
