Hi guys,
here is an interesting and recent dissertion about performance tools (Gatling is among them):
Hi Rafael,
Some of the results look weird, like Gatling performing way better for REST than SOAP.
Either a bug in AHC multipart, or a bug in the tests.
I couldn’t find any reference to where the sources are published, so I tweeted to the author.
Let’s hope he replies 
Thanks for sharing.
I didn’d read the hole dissertion but i can’t agree on usability matrix (IMO gatling is a big win here) and that jmeter changes can be tracked on a VCS but overall thesis seams great, I’ll careful read.
Very well-written indeed.
Well…
I’d love the author would reply to me and provide the code he used.
I suspect his methodology is wrong and he’s comparing apples and oranges.
As he only had to increase the number of file descriptors for Gatling, I suspect he used a closed model for all the tools but Gatling where he used the default open model behavior.
And if you want to run 40.000 rps with each virtual user having its own connections, you definitively have to tune.
So if I’m right, the other tools just used as many keep-alive connections as there’s concurrent users, and would reuse them between one loop iterations.
For example, if response time is 50ms, you actually run 2.000 concurrent connections (each connection being used to send 20 rps).
But if you’re to use an open load model, and probably just start 40.000 new virtual users per second, you’re opening and closing as many new connections per second. And then, of course you have to increase file descriptors and tune ephemeral ports. And this takes quite an extra burden on the OS.
A perfectly valid use case, but completely different.
Finally, I think the test cases really don’t match the test tools. This is just url bashing, and wrk is definitively the tool you want for this.
The tested tools come with way more features than send static requests, and all those features come of course with a performance price.
This is just url bashing, and wrk is definitively the tool you want for this
Why would you suggest wrk for URL bashing as opposed to other command line tools, such as AB or Siege?
Aidy
It’s supposed to be the more url basher out there.
Hi Stephane,
I’m the author of the thesis. I just found this thread. I published most of the relevant sources to Github, here’s the gatling.conf: https://github.com/rsmeral/modern-performance-tools-applied/blob/master/test-configurations/gatling/gatling.conf
With regard to the methodology (“comparing apples to oranges”), I admit you could be right. Another thing I have to admit is, I did not dedicate as much time to the thesis as it deserved.
The thesis had two goals: compare overall usability/user-friendliness and compare “raw” performance – how many RPS the tool is capable of doing in trivial scenarios.
Things I did right: consistency, fairness. I made extra sure the environment was stable for all the runs, it was always the same (48 GB RAM, 8-core, 1Gb/s NIC) machines, always restarted the app server between runs, cleaned the processes, checked the network throughput, etc. The results are always averages of 3 runs.
Things I could have done a lot better: look more deeply into how the individual tools work and tweak them properly. I spent some time with each tool trying to get the highest possible throughput, though I have to admit I kind of struggled with Gatling. This is understandable since Gatling was the only tool which does not have a thread-per-user model, so it admittedly has its specifics (AHC…). But overall, I think Gatling performed rather well in most tests. I also considered it very promising and chose it for further testing in the last chapter.
Though, I was definitely puzzled by the difference between HTTP vs. SOAP results, since it’s basically the same scenario, the only difference being GET vs. POST (with a XML body). I was curious and wanted to consult this, but I just didn’t find the time. Hopefully, you can shed some light on this now
Thanks for taking the time to read it, guys!
Hi Ron,
Thanks a lot for getting in touch, and for providing your material, which is extremely valuable for us.
First, you properly used a closed model for Gatling, so your tests are right, and you’re not comparing apples and oranges as I suspected.
My apologies.
Then, I’m not sure why you needed to increase the files limit for such use cases. Was it only for the SOAP test?
Regarding the poor performance under heavy load for the SOAP test , the culprit is RawFileBody. It currently uses AHC’s zero-copy file upload (over HTTP only, not HTTPS of course) and it seems it’s efficient from a memory point of view, but not from throughput one. I’ve fixed this in 2.2: https://github.com/gatling/gatling/issues/2733. This would also explain why you needed more file descriptors here because it opens and closes a file socket for every upload.
As a workaround in 2.1, one can use ELFileBody that would load the bytes in memory.
Thanks again,
Regards,
Hi Ron,
Thanks a lot for getting in touch, and for providing your material,
which is extremely valuable for us.First, you properly used a closed model for Gatling, so your tests are
right, and you're not comparing apples and oranges as I suspected.
My apologies.Then, I'm not sure why you needed to increase the files limit for such
use cases. Was it only for the SOAP test?
If I remember correctly, I hit upon the file limit in other tests as
well, not just SOAP, but I'm not 100% sure. I found the documentation
(http://gatling.io/docs/2.1.2/general/operations.html#os-tuning),
followed it thinking it's just a tweak that's required for proper
operation of Gatling. I didn't play with the file limit any further
after first increasing it to 300k.
Regarding the poor performance under heavy load for the SOAP test , the
culprit is RawFileBody. It currently uses AHC's zero-copy file upload
(over HTTP only, not HTTPS of course) and it seems it's efficient from a
memory point of view, but not from throughput one. I've fixed this in
2.2: HTTP: Don't use zero-copy for uploading files · Issue #2733 · gatling/gatling · GitHub. This would also
explain why you needed more file descriptors here because it opens and
closes a file socket for every upload.
So NOT using zero-copy in the SOAP test (or any using RawFileBody)
should result in better throughput? Why is that? I thought the whole
zero-copy concept is useful exactly in increasing throughput by reducing
strain on the CPU and RAM. Is that because of the excessive FS
operations? I'm not sure I understand. Sadly, I don't have the resources
to test this out now.
So NOT using zero-copy in the SOAP test (or any using RawFileBody)
should result in better throughput?
Yep
Why is that? I thought the whole
zero-copy concept is useful exactly in increasing throughput by reducing
strain on the CPU and RAM.
Probably mostly when you can't afford loading the whole payload in memory,
like upload a 100 Mb file.
If zero-copy was a throughput winner, I guess caches such as memcached
would serve this way instead of from memory 
Is that because of the excessive FS operations?
That's what it seems. On my set up, I got throughput x2 just by replacing
with ELFileBody.