Hi Max,
as you’ve seen from the discussion in Http_req_duration wrong datadog, this is related to how metrics are aggregated for Datadog. Specifically, see this explanation.
The problem is that Datadog already receives aggregated metrics from the DogStatsD agent, which creates the 95percentile metric, and then to visualize them in a Datadog graph, you aggregate them again by choosing e.g. “avg by”. This is why they don’t match up with the aggregation that k6 shows in the end-of-test summary, which is the more reliable source.
I wasn’t able to reproduce your exact issue, though, and in my test I get different values for both, though 95percentile is still inconsistent with the k6 summary:
Reading the Datadog documentation about percentiles, it seems it would be more flexible to send some metrics to DogStatsD as distribution values. These would apparently skip the aggregation on the client/agent side, and allow you to compute any percentile in the Datadog query.
This will need more evaluation to see how well it’s supported by DogStatsD, as well as any other backends using StatsD, but it looks like it would be an improvement over the currently limited and confusing situation. I created the GitHub issue #2819, so feel free to subscribe to it for updates.