Load test is getting crashed with gatling version 3.1.1

Gatling (Open-Source)
1

HI,

Recently we migrated gatling from version 2.3 to 3.1.1. While doing the load test with 3.1.1 Gatling getting crashed and test is getting stopped automatically after 15 to 20 Min of test. Attached is the configuration file for your reference.
Same load test scripts are working fine on Gtaling 2.3 version and test is not getting crashed.
Observed below errors in run.log. Please help how to fix this issue.

06:06:35.179 [WARN ] i.g.h.e.GatlingHttpListener - Request ‘Main_Request’ failed for user 226
java.lang.OutOfMemoryError: Java heap space
at java.util.LinkedHashMap.newNode(LinkedHashMap.java:256)
at java.util.HashMap.putVal(HashMap.java:631)
at java.util.HashMap.put(HashMap.java:612)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapObject(UntypedObjectDeserializer.java:892)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:654)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapArray(UntypedObjectDeserializer.java:831)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:668)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapObject(UntypedObjectDeserializer.java:895)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:654)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapArray(UntypedObjectDeserializer.java:811)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:668)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapObject(UntypedObjectDeserializer.java:895)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:654)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4013)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3070)
at io.gatling.core.json.Jackson.parse(Jackson.scala:42)
at io.gatling.core.json.JsonParsers.$anonfun$safeParseJackson$2(JsonParsers.scala:42)
at io.gatling.core.json.JsonParsers$$Lambda$1159/323891134.apply(Unknown Source)
at io.gatling.commons.validation.package$.safely(package.scala:34)
at io.gatling.core.json.JsonParsers.safeParseJackson(JsonParsers.scala:42)
at io.gatling.http.check.body.HttpBodyJsonPathCheckMaterializer$.$anonfun$jsonPathPreparer$1(HttpBodyJsonPathCheckMaterializer.scala:46)
at io.gatling.http.check.body.HttpBodyJsonPathCheckMaterializer$$$Lambda$385/593447952.apply(Unknown Source)
at io.gatling.core.check.CheckBase.memoizedPrepared$1(Check.scala:74)
at io.gatling.core.check.CheckBase.$anonfun$check$6(Check.scala:88)
at io.gatling.core.check.CheckBase$$Lambda$1046/347780414.apply(Unknown Source)
at io.gatling.commons.validation.Success.flatMap(Validation.scala:43)
at io.gatling.core.check.CheckBase.$anonfun$check$4(Check.scala:87)
at io.gatling.core.check.CheckBase$$Lambda$1044/523876086.apply(Unknown Source)
at io.gatling.commons.validation.Success.flatMap(Validation.scala:43)
at io.gatling.core.check.CheckBase.check(Check.scala:86)
at io.gatling.http.check.HttpCheck.check(HttpCheck.scala:36)
at io.gatling.http.check.HttpCheck.check(HttpCheck.scala:33)
06:06:35.180 [WARN ] i.g.h.e.r.DefaultStatsProcessor - Request ‘Main_Request’ failed for user 226: j.l.OutOfMemoryError: Java heap space
06:06:35.180 [ERROR] i.g.h.c.i.HttpAppHandler - Fatal error
java.lang.OutOfMemoryError: Java heap space
at java.util.LinkedHashMap.newNode(LinkedHashMap.java:256)
at java.util.HashMap.putVal(HashMap.java:631)
at java.util.HashMap.put(HashMap.java:612)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapObject(UntypedObjectDeserializer.java:892)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:654)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapArray(UntypedObjectDeserializer.java:831)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:668)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapObject(UntypedObjectDeserializer.java:895)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:654)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapArray(UntypedObjectDeserializer.java:811)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:668)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.mapObject(UntypedObjectDeserializer.java:895)
at com.fasterxml.jackson.databind.deser.std.UntypedObjectDeserializer$Vanilla.deserialize(UntypedObjectDeserializer.java:654)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4013)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3070)
at io.gatling.core.json.Jackson.parse(Jackson.scala:42)
at io.gatling.core.json.JsonParsers.$anonfun$safeParseJackson$2(JsonParsers.scala:42)
at io.gatling.core.json.JsonParsers$$Lambda$1159/323891134.apply(Unknown Source)
at io.gatling.commons.validation.package$.safely(package.scala:34)
at io.gatling.core.json.JsonParsers.safeParseJackson(JsonParsers.scala:42)
at io.gatling.http.check.body.HttpBodyJsonPathCheckMaterializer$.$anonfun$jsonPathPreparer$1(HttpBodyJsonPathCheckMaterializer.scala:46)
at io.gatling.http.check.body.HttpBodyJsonPathCheckMaterializer$$$Lambda$385/593447952.apply(Unknown Source)
at io.gatling.core.check.CheckBase.memoizedPrepared$1(Check.scala:74)
at io.gatling.core.check.CheckBase.$anonfun$check$6(Check.scala:88)
at io.gatling.core.check.CheckBase$$Lambda$1046/347780414.apply(Unknown Source)
at io.gatling.commons.validation.Success.flatMap(Validation.scala:43)
at io.gatling.core.check.CheckBase.$anonfun$check$4(Check.scala:87)
at io.gatling.core.check.CheckBase$$Lambda$1044/523876086.apply(Unknown Source)
at io.gatling.commons.validation.Success.flatMap(Validation.scala:43)
at io.gatling.core.check.CheckBase.check(Check.scala:86)
at io.gatling.http.check.HttpCheck.check(HttpCheck.scala:36)
at io.gatling.http.check.HttpCheck.check(HttpCheck.scala:33)
06:06:35.233 [WARN ] i.g.h.e.GatlingHttpListener - Request ‘10-healthy-midweek-meals.html?icid=’ failed for user 720
javax.net.ssl.SSLException: handshake timed out
at io.netty.handler.ssl.SslHandler.handshake(…)(Unknown Source)
06:06:35.237 [WARN ] i.g.h.e.GatlingHttpListener - Request ‘10-healthy-midweek-meals.html?icid=’ failed for user 1472
io.netty.channel.ConnectTimeoutException: connection timed out: XXXXX
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:495)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

Thanks in advance,
Swetha. G

gatling.conf (8.53 KB)

2

Please provide a sample we can run on our side and that reproduces your problem.
We can’t investigate without one.

3

HI,

We cannot provide sample scripts as it contains confidential information. Please let me know if you need any other information to resolve this issue.

4

Please provide a heap dump of when the OOME occurs.
If it doesn’t help, please build a sample app that exhibits the same behavior as your real one.
If you can’t, you can consider contracting with us and have us investigate on your real app under NDA.
Otherwise, I’m afraid you’re on your own.

5

We can share Heap dump. Please provide the location, as it is very heavy we couldn’t attach here.

6

There are tons of file sharing platforms: wetransfer, google drive, dropbox…

7

Please find Hep dump in the below location…

https://drive.google.com/drive/folders/1-cuiL8h4JT9IJqc_Fd_kTe6KCigvKDF7?usp=sharing

Thanks,
Swetha. G

8

I don’t know how you’ve set up your heap size, but you have ~50 JSON or HTML pages with size between 500kB and 2MB in memory.
You might want to either increase heap size, or work on the size of your JSON payloads (IMHO, 2.3 MB JSON payloads are not great for a website).

9

Heap dump is generated at the time of jvm crash.

The same scripts work fine in older version whereas, in gatling 3.1.1 version Gatling is crashed after few mins into the test.

As you said, we have multiple JSON Payload pages in our scripts, even after increasing the JVM heap size, it is not worked for us.

We get “javax.net.ssl.SSLException: handshake timed out” & io.netty.channel.ConnectTimeoutException: connection timed out and then we get “java.lang.OutOfMemoryError: Java heap space”.

Please advice.

10

Check if your server can really handle the load you’re injecting.
It could be that it can’t, hence the connect and ssl handshake timeouts, so your users/requests pile up in memory and cause a crash.

Note that with Gatling 3, we’ve fixed the way we’re handling SSLSessions. We now have per virtual user SSLSessions, unless you use shareConnections.

11

HI Stephane,

Issue got resolve by increasing the heap size for running the gatling process.

DEFAULT_JAVA_OPTS="${DEFAULT_JAVA_OPTS} -Xmx1****0G -XX:+HeapDumpOnOutOfMemoryError"

Location: /home/ec2-user/gatling/bin/gatling.sh

Thanks,
Swetha. G