Percentile statistics precision for >1rps metrics when using Cloudwatch

Grafana k6 OSS Support
1

Context:
we are in the very beginning of transitioning to k6 from Python performance tests at our company. This means we will use Python for a while, but going to start to collect metrics in CloudWatch(we HAVE to use AWS services). I know it’s a k6 forum and i will need this knowledge later when we can use k6 to run performance tests so please bear with me.
So continuing, some facts about CloudWatch custom metrics:

  • CloudWatch stores the metric with sub-minute resolution down to one second
  • if we have multiple data points per second we can publish statistic sets instead of single metrics, BUT it comes with limitations: besides the average, Cloudwatch can only store minimum and maximum for the pre-aggregated, 1 second based data sets

Question:
do i understand well, that if we choose metric collection for k6 in Cloudwatch(to me it’s not optional), we lose precision on some higher percentiles(p90, p95) of >1rps endpoint metrics? Because i presume that these higher percentiles will look more like p99, since as a result of the aggregation, there is a higher chance that the maximum of the 1 second based datasets will be more similar.
Is there any trick with the config params described in the k6 docs here to get as realistic results as possible?
I’m not a statistical expert, but i presume that the outcome very much depends on the number of samples(duration of the test) and the actual value of the number of (higher than 1 second) requests as well.

2

Hi there,

I’m not very familiar with CloudWatch, nor a statistics expert either, but here’s my understanding of it.

do i understand well, that if we choose metric collection for k6 in Cloudwatch(to me it’s not optional), we lose precision on some higher percentiles(p90, p95) of >1rps endpoint metrics?

Yes, I believe so, since even if k6 sends raw data points with no aggregation, CloudWatch would still aggregate it per second at the highest resolution and you’d lose the sub-second precision.

Is there any trick with the config params described in the k6 docs here to get as realistic results as possible?

I don’t think so, see the Agent documentation. The suggested values in the k6 docs disable aggregation on the statsd side, but there’s no way to control how the data is aggregated in CloudWatch itself.

Maybe someone more familiar with CloudWatch could provide more details. Good luck!

1 Like
3

Thanks Ivan, really appreciate the reply!