Possible discrepancy between generated and computed percentile99 values

[Michael_Absin]

Hello all,

This is my first time to post here in this group and hope you guys are well.

I found a discrepancy (diff of 51ms) between the generated percentile99 value in the gatling generated static html and in grafana (given received metric values and shown as percentile99). Maybe due to computation/conversion with round-offs, but I need to verify, since 51ms in multi-user sampling will be hundreds or thousands of ms diff when going to more user sampling.

• Summary report https://drive.google.com/open?id=1DaVRgm0xPrjLB3jxcZh4CerpjqwrpeEe - 99th percentile = 3500.

• The 99th plot in summary report (all others are excluded) - https://drive.google.com/open?id=1_nFY-lesz6zZH7s9u1zbeLpavTibC_Vt

• From that summary, it has 7 raw values captured by influxdb and shown in local grafana ( .727, 1.559, 1.823, 1.967, 2.095, 3.055, 3.551 ) - I believe the frequency used on gatling summary graph is lower than what was configured in gatling for graphite plugin - that’s why it has more values

• When we plot the final resulting percentile99 in grafana, it appears to be 3551ms (not matching with the summary report of 3500ms) - see https://drive.google.com/open?id=14NRZhXmOpqopHHov5LITTDPsbCmksAgw

• I roamed around gatling codes in github and found this https://drive.google.com/open?id=1CZW6gfrBqIccPaAUeDuaBTWrSY2wXR1F - this can be found on this URL https://github.com/gatling/gatling/blob/df34c96fabad3e1a3d21026dc3a6053e7729dd84/gatling-charts/src/main/scala/io/gatling/charts/stats/buffers/GeneralStatsBuffers.scala

• I was expecting the formula will be like below link A, but while understanding the AVLTreeDigest class it looks like they are following the below link B. Only that the formula to get the index is like this ( p * (n - 1)), minus 1 because the source of the values are in array and need to coincide with the nth element starting at zero.

link A: https://www.emathhelp.net/calculators/probability-statistics/percentile-calculator/?i=.727%2C1.559%2C1.823%2C1.967%2C2.095%2C3.055%2C3.551&p=99&steps=on
link B: http://www.statisticshowto.com/quantile-definition-find-easy-steps#text-14

• Either formula is supposed to return 3551ms even if it uses math.round() function.

My questions are:

• Is 3551ms the better value to present than the ones showed in static html summary report?
• Above are all on how I understand the flow, please do correct me if I’m wrong. Will that 51ms difference is due to conversion from sec to ms and back from ms to sec for presentation?
• I’m still not convinced regarding the math.round() function as I believe it has to declare some format at what point decimal it should start rounding off, would somebody point me the specific code block that does the rounding off which leads to slight discrepancy?

Gatling highcharts v=2.3.0
Gatling maven plugin v=2.2.3

Any reply would be much appreciated!

Thank you much,
Mike

[slandelle]

There are many differences between HTML reports and the Graphite integration:

• the former uses t-digest while the latter uses HdrHistogram
• the former split the total time window in 1000 buckets by default, with min 1s, while the latter uses constant 1s buckets, so grouping can be different
• InfluxDB is very ill suited for distributions as it can only store numbers, hence percentiles, so you’ll end up doing plain wrong maths as soon as you need to aggregate values (eg averages of percentiles don’t make any sense).
  In any case, you’ll get approximated values. Computing exact values would require retain all values in memory and sort them, which is simply not feasible as Gatling needs to support use cases with 100,000 rps.

That being said, FrontLine comes with great real time distributions computation without the Graphite/influxDB flaw.

[Michael_Absin]

Hello Stephane,

Thank you very much for the explanation, this enlightens me and thank you for suggesting Frontline.

Regards,
Mike