Gatling got biased performance result?

Gatling (Open-Source)
1

Hi, I found that Gatling got a far way better result than other load testing tools.

I got two machine (2core 4g), one of them run tomcat, another one run the load testing tools.:

docker run -p 8080:8080
-p 1099:1099
-p 1100:1100
-d
–name tomcat
-e HEAP_SIZE=“1G”
-e CONNECTOR_MAX_THREADS=“1000”
-e CONNECTOR_MIN_SPARE_THREADS=“1000”
-e EANBLE_ACCESS_LOG_VALVE=“false”
chanjarster/api-gateway-comp-tomcat

The test url is: http://tomcat:8080/examples/jsp/jsp2/simpletag/book.jsp

The Gatling got 19131.647 mean req/sec, which is pretty high.

$ mvn gatling:test -Dgatling.simulationClass=Tomcat

tomcat.jmx (6.42 KB)

gatling.zip (16.3 KB)

2

No, it’s not biased, you most likely don’t have the same default settings for the 3 different tools you’re comparing.

By default, Gatling tries to emulate a web browser’s behavior, meaning that it has caching enabled.Your test is having the same virtual users loop and hit the same url over and over again (which BTW has zero value in terms of load testing, think about your Tomcat’s JIT) and I’m pretty sure the response contains some caching headers.

If you want Gatling to have the same behavior as the other tools you’re comparing to, you have to use disableCaching, cf https://gatling.io/docs/current/http/http_protocol/.

3

Thanks for the quick reply :slight_smile:

I tried disable the cache you mentioned, and the result doesn’t change.

Besides, the result doesn’t contains any cache headers:

$ curl -v http://tomcat:8080/examples/jsp/jsp2/simpletag/book.jsp

  • Trying 192.168.101.11…
  • TCP_NODELAY set
  • Connected to tomcat (192.168.101.11) port 8080 (#0)

GET /examples/jsp/jsp2/simpletag/book.jsp HTTP/1.1
Host: tomcat:8080
User-Agent: curl/7.58.0
Accept: /

< HTTP/1.1 200
< Set-Cookie: JSESSIONID=0DD629B87A653D199001AB47C1FF1298; Path=/examples; HttpOnly
< Content-Type: text/html;charset=ISO-8859-1
< Content-Length: 951
< Date: Fri, 10 Apr 2020 06:52:51 GMT
<

About the different settings of 3 tools issue you mentioned, em…, since they are different tools, so only I can do is to make them as same as possible.

Gatling max vusers is 2250 (below is gatling outpu):

[--------------------------------------------------------------------------] 0%
waiting: 0 / active: 2250 / done: 0

Gatling vuser rampup config:

object BenchmarkConfig {

val rampFrom = 50;
val rampTo = 100;
val rampDuring = 30;
val totalDuring = 120;
val uri = “/examples/jsp/jsp2/simpletag/book.jsp”;

}

setUp(
scnTomcat.inject(
rampUsersPerSec(BenchmarkConfig.rampFrom) to BenchmarkConfig.rampTo during (BenchmarkConfig.rampDuring seconds),
).protocols(tomcatConf),
)

The Jmeter script number of threads is also 2250, and ramp-up period 15 seconds.

Since wrk2 doesn’t support ramp up, I can only set connection to 2250.

As you can see, settings of 3 tools are quite similar, it doesn’t make sense to get so different result: jmeter and wrk2 got only 1/3 throughput of gatling.

在 2020年4月10日星期五 UTC+8下午2:38:36，Stéphane Landelle写道：

4

I see your Tomcat replies with Set-Cookie header, meaning it creates HttpSessions.
Gatling comes with cookies support enabled by default. I’m sure JMeter doesn’t and you have to enable it.
It’s possible that for the other tools, you’re creating a new HttpSession for each request and clog your Tomcat instance with unused allocated HttpSessions that will only be garbage collected on session idle timeout.
I recommend you disable HttpSessions on your tomcat.

5

You are the life saver! I’ve been stuck on this problem the whole week, thanks a lot.

在 2020年4月10日星期五 UTC+8下午3:40:57，Stéphane Landelle写道：