Gatling Out Of Memory Issue

Gatling (Open-Source)
1

Hi,

I am new to Gatling. I am using Gatling 3.5.1 to run 1000 users in a windows LG. I have assigned 15G Heap and the machine has 27G RAM 6 Processors and 64bit.

Now after the injection starts when the user count reaches ~110 users the entire Heap memory is getting filled and I don’t see any garbage collection taking place. The Gatling’s JVM Opts is as (using Java 9) : JAVA_OPTS=-Xms15g -Xmx15g -XX:+UseG1GC -XX:MaxGCPauseMillis=30 -XX:G1HeapRegionSize=16m -XX:InitiatingHeapOccupancyPercent=75 -XX:+ParallelRefProcEnabled -XX:+PerfDisableSharedMem -XX:+HeapDumpOnOutOfMemoryError -XX:MaxInlineLevel=20 -XX:MaxTrivialSize=12 -XX:-UseBiasedLocking

I have even tried session.remove and session.reset and clear() all Mutable Buffers. But still within 20 mins into the injection Heap is getting full.

Below is my simulation class:

class Try extends Simulation {

val httpProtocol = http
.baseUrl(GlobalVariableConstants.baseUrl)
.inferHtmlResources()
.header(“Connection”,“keep-alive”)
.acceptEncodingHeader(“gzip, deflate, br”)
.acceptLanguageHeader(“en-GB,en;q=0.9,en-US;q=0.8”)
.userAgentHeader(“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36 Edg/89.0.774.57”)

val feeder_081 = separatedValues(s"C:\081_SystemUserDetails.dat", ‘\t’).batch(100).random

val feeder_082 = separatedValues(s"C:\082_SystemUserDetails.dat", ‘\t’).batch(100).random

val scn1 = scenario(“Portlet”)
.forever(){
feed(feeder_081)
.exec(G_081_Portlet.script)
.exec(flushSessionCookies)
.exec(session =>{
session.reset
})

}

val scn2 = scenario(“MyPages”)
.forever(){
feed(feeder_082)
.exec(G_082_MyPages.script)
.exec(flushSessionCookies)
.exec(session =>{
session.reset
})

}

setUp(
scn1.inject(
rampConcurrentUsers(0) to(500) during(7200 seconds),
constantConcurrentUsers(500) during(120 minutes)),
scn2.inject(
rampConcurrentUsers(0) to(500) during(7200 seconds),
constantConcurrentUsers(500) during(120 minutes))
)
.protocols(httpProtocol)
.maxDuration(120 minutes)

}

I have writers = [console, file, graphite] on to write data to InfluxDB. In a heap dump I found scala.collection.immutable to be taking 92% of the memory. Attached is the Jconsole monitoring of the Gatling’s Java process.

Please let me know how can I resolve this? I was doing a PoC on Gatling to move from JMeter. JMeter was able to scale up to 1000 users with 15G Heap in this same machine.

Thanks & Regards,
Uddipan

Gat_Jconsole.jpg

2

Impossible to tell without knowing what G_081_Portlet.script and G_082_MyPages.script.
Also, please provide the heap dump caused by your OutOfMemoryError.

3

Here are the masked script files that are used in the Try simulation.

Script.zip (8.25 KB)

4

Thanks, but please also provide the stacktrace with the OOME and the heap dump.

5

I had to explicitly do the Heap dump as Gatling didn’t do it itself and just hanged with no request being sent.

This heap dump I did for a 5G heap size just to replicate the issue.
heap1.zip

6

jvm parameters has to be set to generate heapdump on out of memory. Set these in pom.xml if it’s a maven project.

JVM parameters: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=<PATH_FOLDER><file_name>. hprof.

And also make sure there is enough disk space available. Usually at least double the size of jvm max heap size

Best /Mohan

7

-XX:+HeapDumpOnOutOfMemoryError is already in the JVM OPTS. I have attached the heap dump in the earlier email where Heap was set to 5G.
Not sure why Gatling is consuming the entire memory.

8

The OOME is 100% your fault.
funEPortalLogOn#GArrBuff#req and req1 and mutable global share state where every virtual user keeps on piling up new entries over and over again.

9

Thanks for pointing out the vars - I’ll have a look into it. But not sure what you meant by mutable Global share also I am calling the clear() on these Buffers.

Thanks & Regards,
Uddipan

10

Check where you're "clearing" them: not inside any scenario or piece of
scenario. This has absolutely no effect.
If your intent is not to share global state,* don't use global
references*, store state
in the user's Session.

[image: Logo] <https://gatling.io>
*Stéphane Landelle**Chief Technical Officer*
twitter: @slandelle <https://twitter.com/slandelle>
site: gatling.io

11

Totally my fault. I had a version in my local with this inside a exec block:
println(“BEFORRRREEEE CLEAAAAAAAAAAAAAAR” + req)

req.clear()
println(“AFTEEERRRRRR CLEAAAAAAAAAAAAAAR” + req)

And I was using a different one to test, that’s why I was sure that I was clearing Buffers. Thanks a lot Stephane for pointing that out. Issue is now resolved!! Max Heap usage now is 2.6G.

12

Still, you really shouldn’t use a global reference for this.
Your code is super racy, with multiple users modifying the collection concurrently.

13

Okk. Are you suggesting I move the collections inside the exec block to make it local. Will that make each virtual user modify them individually owing to the exec blocks?