Dear community member,
We’d love your feedback on the following feature request: https://github.com/gatling/gatling/issues/2394
This request is about removing the mean and standard deviation columns in Gatling reports, and use this extra room to display more percentiles.
However, I’d love to get some feedback from the community first.
Please retain “Mean” and Std Deviation can be removed. A column exclusively for “Throughput” (under the same name) would be very helpful!
I really don't see the need to introduce these many percentiles. At least load tests that I have done, mangers/users are usually happy with 90th or 95th percentiles.
What we do need is a throughput graph. I keep getting that request.
I generally now look at the 95% line, but clients constantly describe aspirational responses in average time. I think it is still worth providing a mean time and standard deviation. I wouldn’t take much notice of the 50%, 70% line (maybe, I should). But if I am analysing the wrong metrics, others are too.
Mean is probably less meaningful than median, so having it turned off by default is not unreasonable. Std. Deviation is important if you show the mean. But the histogram of response times conveys what people really need in a way that they can intuitively understand it.
That said, taking functionality away will almost always result in squawks. My advice is to think of a way to enable people to configure which columns are displayed in the reports. And if you can, make the report capable of being full-width, so that if people choose to display more columns, they can actually see them.
Mean and standard deviation go hand-in-hand. If you provide one, you should also have the other. Individually, it’s hard to make sense of either.
When comparing two test runs, I like look at the mean response time as a single, easily understood metric.
But, when trying to understand the response of a system, I look at both mean and std dev.
The two percentile columns can be changed in the configuration. I don’t need anymore than two percentiles.
My vote is to keep both mean and std dev and two percentiles.
A better feature is ability to add/remove metrics to the report, so people can choose what they want to see.
2.0.0M3a has a column for throughput. Has this been removed in the released version?
Personally I don’t use these measurements because the business people are not very interested in this aspect of the system. But from I’ve learned of performance engineering thus far, it looks like there are occasions where these measurements are the better ones to use for tuning and calibration.
I would vote for adding the other columns, but give users the option to select the columns that matter most for their use case.
Thank you all for your feedback.
As a conclusion, the combo (mean + standard deviation) is an interesting metric so we have to keep it.
I was able to add 2 more percentiles: https://github.com/gatling/gatling/issues/2394
Default values are 50, 75, 95 and 99. There’s no more room for adding more.
The consequence is that Graphite plugin users will have to adapt, as default 95 and 99 were respectively shifted from percentile1 to percentile3 and percentile2 to percentile4.
Thanks again for your help!