Benchmarking Gatling

Gatling (Open-Source)
1

I am attempting to benchmark the 2.2.0-SNAPSHOT version of Gatling, running a web server and Gatling on the same Red Hat GNU/Linux box (4 CPUs, 15 GB of RAM) to avoid any network latency. I am also using htop and jvmtop.

I am using Go’s Revel web framework which claims ‘to serve three to ten times as many requests as Rails’ (which doesn’t exactly aspire much confidence in me). Revel seems to cave in after 2000 r/s using the default “Hello World” application and this sim:
https://github.com/bbc/gatling-load-tests/blob/master/src/test/scala/bbc/benchmark/Benchmark.scala

I looked at Rust’s, Iron web framework which claims to have been benchmarked at “84,000+ requests per second for hello world” using wrk (though not sure where that figure comes from (https://github.com/iron/iron/wiki/How-to-Benchmark-hello.rs-Example)). However, I am receiving a Java connection refused error on localhost even though I can curl the sample webpage. I tried Lua’s Lapis which has nginx thrown in, but it has been a pain to install.

I wonder if anyone could suggest a web server/framework I could use, and if it would be worth standardising these benchmarks which can be run pre-release (I don’t mind doing this)?

https://revel.github.io/
https://github.com/iron/iron
http://leafo.net/lapis/

Many Thanks

Aidy

2

I am attempting to benchmark the 2.2.0-SNAPSHOT version of Gatling,
running a web server and Gatling on the same Red Hat GNU/Linux box (4 CPUs,
15 GB of RAM) to avoid any network latency. I am also using htop and
jvmtop.

I don't expect way better numbers on this kind of "url basher" benchmark
until when support native transport
<http://netty.io/wiki/native-transports.html&gt; (planned for 2.2, not
implemented yet).

I am using Go's Revel web framework which claims 'to serve three to ten
times as many requests as Rails' (which doesn't exactly aspire much
confidence in me). Revel seems to cave in after 2000 r/s using the default
"Hello World" application and this sim:

https://github.com/bbc/gatling-load-tests/blob/master/src/test/scala/bbc/benchmark/Benchmark.scala

Your limit is the connection opening/port recycling speed, as your
injection profile is rampUsersPerSec(100) to 30000 during(10 minutes) with
a one single request scenario. There's no way you could open 30.000 new
HTTP connections per second.

I looked at Rust's, Iron web framework which claims to have been
benchmarked at "84,000+ requests per second for hello world" using wrk
(though not sure where that figure comes from (
How to Benchmark hello.rs Example · iron/iron Wiki · GitHub)).

Check out the wrk test config: 900 connections being kept alive + threads
looping for 10s, very different from what you do.

However, I am receiving a Java connection refused error on localhost even

though I can curl the sample webpage.

Yep, connection allocation limit.
If you were to only inject 1000 users and loop, you'd probably get
something like 50.000 rps. Gatling is not an url basher.

Honestly, those benchmarks might make sense internally for those web
frameworks/HTTP stacks developers to keep performance regression in check.
But IMO they don't make any sense compared to real work use cases: how does
it fare when dealing with a normal size payload? With a huge payload? How's
the multipart support? etc