We’re having throughput issues when using SSL and our theory right now is that the Gatling client is the issue.
Our setup:
- we have Apache in front of a SOAP web service
- Apache handles SSL and forwards HTTP requests to the SOAP web service
- gatling version 2.0.0-RC2
Simulation: Running with no SSL directly against the SOAP web service
- simulation that runs for 1 minute with 10 users: 10000 requests per minute
Simulation: Running against Apache with SSL
- a simulation that runs for 1 minute with 1 user: 1000 requests per minute
- a simulation that runs for 1 minute with 10 users : 14 requests per minute
When we only get 14 requests per minute; Gatling reports that the max response time was 1 second. So it looks like Gatling hangs for a long time between making requests.
We ran the same SSL simulation with JMeter with 10 concurrent users and got 10000 requests per minute, so we are pretty confident that Apache isn’t the bottleneck.
Does anyone have suggestions for configurations that we can tweak or how we can go about debugging this issue?
Simulation: Running with no SSL directly against the SOAP web service
- simulation that runs for 1 minute with 10 users: 10000 requests per minute
It seems that you are recycling the users, that isn’t the Gatling way of doing things.
So, really, without seeing the scenario, we can’t really help here.
I’d also want to see
3 thread dumps 10 seconds apart
3x netstat -na dumped to file
Thanks
Alex
I failed to say that we ran the client on Windows 7. We just ran the same simulation on Linux and the SSL throughput was 10000 requests per minute.
Is there some windows based configuration we could tweak perhaps?
By the way, impressed with having two responses in such a short amount of time 
Sincerely
Odin
thanks, Like Nicolas said, it would be useful to see a minimal test case (likely mainly the http protocol options and inject config)
and for my part the thread dumps and netstat
there may be other ways but a thread dump is usually a good starting point to provide some visibility on where the delay might be.
thanks
Alex
This is the test case we use (can’t share the actual code due to company policy):
package sim
import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._
class MySimulation extends Simulation {
val myData = csv(“some.csv”).circular
val httpProtocol = http
.baseURL(“https://some-url”)
.inferHtmlResources()
.acceptEncodingHeader(“gzip,deflate”)
.authorizationHeader(“Basic casdkajslkqwe=”)
.contentTypeHeader(“text/xml;charset=UTF-8”)
.userAgentHeader(“someclient”)
val headers = Map(“SOAPAction” → “”)
val scn = scenario(“My Scenario”)
.feed(myData)
.during(1 minutes) {
exec(http(“req_0”)
.post("/some/path")
.headers(headers)
.body(StringBody(""""
…valid soap request …
“”".stripMargin.trim()))
.basicAuth(“abc”, “123”))
}
setUp(scn.inject(atOnceUsers(10))).protocols(httpProtocol)
}
Additionally, we use the default gatling.conf settings.
Hi Odin,
What Java version are you using? ensure the latest JDK(not jre) is installed and on the path.
I can’t see anything particular from the script - did you manage to get a thread dump?
You can use the jstack command in the java jdk bin directory to get a thread dump from the pid of gatling.
when running netstat … can you look for any lines with “SYN”??
I understand if you don’t want to send the whole output but connections in SYN* state would indicate a “hangs for a long time between making requests”. Currently gatling doesn’t include the connection time in the latency figures.
just to simplify the numbers and scope, can you run just a plain get on a static file on the apache server (you could copy a file to the document root for diagnostic purposes), and only inject 1 user, and report the throughput and latency? thanks
Thanks
Alex
Hi Alex,
We’re using Java version 1.8.0_11-b12. When we ran with only one user instead of 10, we got 1000 requests per minute, so the problem is concurrency related.
netstat -an shows:
- 10 TCP-connections to the server when running without SSL
- 1 (sometimes 2) TCP connections when running with SSL
I’ll try to get the thread dumps and post the details here.
In the meantime we’ve solved the problem by running the tests on Linux so this isn’t a showstopper for our Gatling proof of concept any more. Thankfully I should add, because Gatling is so much better to work with than JMeter
Sincerely
Odin
Hi Alex,
Here are the thread dumps, see https://dl.dropboxusercontent.com/u/296943/gatling_thread_dump.zip
My gut feeling is that there is something with my Windows setup that is causing the issue.
Sincerely
Odin
looks like you may have been hitting this:
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6450279
$ grep Inet4AddressImpl.getHostByAddr thread_dump*
thread_dump1.txt: at java.net.Inet4AddressImpl.getHostByAddr(byte[ ])
thread_dump2.txt: at java.net.Inet4AddressImpl.getHostByAddr(byte[ ])
thread_dump3.txt: at java.net.Inet4AddressImpl.getHostByAddr(byte[ ])
Thread “GatlingSystem-akka.actor.default-dispatcher-19”:
at java.net.Inet4AddressImpl.getHostByAddr(byte[ ])
at java.net.InetAddress$2.getHostByAddr(byte[ ])
at java.net.InetAddress.getHostFromNameService(java.net.InetAddress, boolean)
at java.net.InetAddress.getHostName(boolean)
at java.net.InetAddress.getHostName()
at java.net.InetSocketAddress$InetSocketAddressHolder.getHostName()
at java.net.InetSocketAddress$InetSocketAddressHolder.access$600(java.net.InetSocketAddress$InetSocketAddressHolder)
at java.net.InetSocketAddress.getHostName()
at com.ning.http.client.providers.netty.channel.SslInitializer.connectRequested(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelStateEvent)
…
Hi Alex,
I put a DNS record in my hosts file and ran the simulation again; the problem is gone.
Thank you so much for your help!
Sincerely
Odin
Hi,
The problem was that some part of AsyncHttpClient was using InetAddress.getHostName, which can cause a reverse DNS lookup.
I changed targeted JDK to JDK7 so we can use getHostString that doesn’t perform this lookup.
Next Gatling RC3 will ship the fix.
Thanks for reporting!
Stéphane