# 2.0.0-RC2 Increasing heap size issue

**URL:** <https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304>\
**Category:** Gatling (Open-Source)\
**Created:** [August 20, 2014, 7:29am UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304 "2014-08-20T07:29:26Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Daniel\_Moll](https://avatars.discourse-cdn.com/v4/letter/d/97f17d/32.png) [@Daniel\_Moll](https://community.gatling.io/u/Daniel_Moll)\
**Post date:** [August 20, 2014, 7:29am UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/1 "2014-08-20T07:29:26Z")

</div>

Hi Guys,

I am running into memory issues running my script with version 2.0.0-RC2

My simulation is running from a Windows 2008 server virtual machine.

During test execution heap usage keeps increasing up to a point CPU usage goes through the roof and my tests fails.

My scenario looks like this:

setUp(scn.inject(constantUsersPerSec(1) during (600) // warm up of the SUT  
,rampUsersPerSec(1) to(150) during(3600)  
,constantUsersPerSec(150) during (172800))).protocols(httpProtocol)

Some relevant script information:

The script uses ELFilebodys with are fed with values from a number of feeders.

val httpProtocol = http  
.baseURL(“[https://some-host.com](https://some-host.com)”)  
.acceptHeader(""“application/soap+xml, application/dime, multipart/related, text/\*”"")  
.acceptEncodingHeader(""“gzip, deflate”"")  
.userAgentHeader(""“Mozilla/4.0 (compatible; MSIE 6.0; Windows NT)”"")  
.check(regex(“faultstring”).notExists)  
.check(regex("[^1185]\<").notExists)  
.check(regex("[^1059]\<").notExists)  
.check(regex(“errorNumber”).notExists)  
.check(regex("`[^1140]<").notExists)
.check(regex(“HostErrorCode”).notExists)
.check(regex(“faultcode”).notExists)
.disableCaching`

`
.randomSwitch(
12.0 → exec(GetLocalDateTime_1A),
8.0 → randomSwitch(
90.0 → exec(GetSeatMap_1A),
8.0 → exec(GetSeatMap_COD),
2.0 → exec(GetSeatMap_GAT)
),
10.0 → randomSwitch(
90.0 → exec(ListDocument_1A),
8.0 → exec(ListDocument_COD),
2.0 → exec(ListDocument_GAT)
),
23.0 → exec(ListOperationalEligibility_1A),
15.0 → randomSwitch(
91.0 → exec(ListPassenger_1A),
9.0 → exec(ListPassenger_COD)
),
2.0 → exec(ProvidePassengerHandlingInformation_1A),
3.0 → exec(ProvidePassengerInformation_1A),
8.0 → randomSwitch(
90.0 → exec(UpdatePassenger_1A),
8.0 → exec(UpdatePassenger_COD),
2.0 → exec(UpdatePassenger_GAT)
),
1.0 → exec(ProvideTravelDocuments_1A),
3.0 → exec(GetEligibilityForMobileBP_AF),
14.0 → exec(publishConfirmation),
1.0 → exec(UpdateDocument_1A)
)
JVM arguments:
-Xms2g
-Xmx6g
-XX:NewSize=1g
-XX:+HeapDumpOnOutOfMemoryError
-XX:+AggressiveOpts
-XX:+OptimizeStringConcat
-XX:+UseFastAccessorMethods
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+CMSClassUnloadingEnabled
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1
I can share a heapdump and the full script with you via dropbox if you like.
Any help is appreciated since I’m trying to convince my management to use Gatling as a replacement for LoadRunner 🙂
Cheers
Daniel`

---

<div class="post-metadata">

**Author:** ![Alex\_Bagehot](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@Alex\_Bagehot](https://community.gatling.io/u/Alex_Bagehot)\
**Post date:** [August 20, 2014, 8:20am UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/2 "2014-08-20T08:20:20Z")

</div>

Hi Daniel,

Could you add -verbose:gc also to the gc args?  
ideally have (not all are always required but more information upfront provides a better chances of resolving in 1 pass:

heap dump  
any exceptions logged (OOM exception) in logs  
3 thread dumps as the problem kicks off  
verbose gc log  
any os stats (ie. unlikely but you have a 6G heap and 4G on the box…)

can you try to simplify the script to see if any one part of it is causing the problem? to isolate the problem a bit more if possible.

eg. maybe it’s all the nested switches, or the notExists, or the type of requests you have.

thanks,  
Alex

---

<div class="post-metadata">

**Author:** ![Daniel\_Moll](https://avatars.discourse-cdn.com/v4/letter/d/97f17d/32.png) [@Daniel\_Moll](https://community.gatling.io/u/Daniel_Moll)\
**Post date:** [August 20, 2014, 9:45am UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/3 "2014-08-20T09:45:20Z")

</div>

Hi Alex, thanks for your reply

I will try to deliver what you asked for as much as possible but I’m a bit in a time squeeze because I’m on the critical path for go-live of the SUT 🙂

For now I have modified the script by putting a .during() loop around the .randomSwitch and changing the scenario to

setUp(scn.inject(rampUsers(300) over (3600))).protocols(httpProtocol)

In other words I created a “closed system” simulation like LoadRunner would. This setup uses a lot less heap and CPU it seems. Would that fact point you in some direction?

I have uploaded a heapdump to Dropbox, which email address can I use to send a share invitation to?. I have also added a screendump of the VisualGC plugin of VisualVM (couldn’t find a way to export it to file), hope that helps.

Thanks for your help!

Cheers

Daniel

---

<div class="post-metadata">

**Author:** ![Alex\_Bagehot](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@Alex\_Bagehot](https://community.gatling.io/u/Alex_Bagehot)\
**Post date:** [August 20, 2014, 10:03am UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/4 "2014-08-20T10:03:45Z")

</div>

Hi Daniel,

… I suspect it may be

constantUsersPerSec(1) during (600) // warm up of the SUT  
,rampUsersPerSec(1) to(150) during(3600)  
,constantUsersPerSec(150) during (172800))

Gatling creates upfront all the users it needs for the whole test.  
here it’s dominated by the 150\* 172800 (2 days) = 25M in memory waiting to execute.

I saw this before a while back when I was looking at doing open model soak tests.  
The Iago load tool I believe feeds users into the system only when a buffer of to-be run users gets low to avoid the memory consumption.  
I’ll raise a ticket for this once confirmed in the heap dump.

Changing to closed for now is reasonable except my previous comment about the switches not providing the expected split in terms of throughput if the different cases have different durations, ymmv.

Else some other workarounds:  
run a shorter test first, get confidence with that.  
get a load injector with a larger amount of ram. 64bit jvm should scale up to 20-30GB. not sure if that would cover it though.  
a not good hack, but run multiple simulations back to back, thus spreading the heap usage over time. Not ideal though I know.

my email is [ceeaspb@gmail.com](mailto:ceeaspb@gmail.com) please include Stephane and Pierre though as well.

Thanks,  
Alex

---

<div class="post-metadata">

**Author:** ![Daniel\_Moll](https://avatars.discourse-cdn.com/v4/letter/d/97f17d/32.png) [@Daniel\_Moll](https://community.gatling.io/u/Daniel_Moll)\
**Post date:** [August 20, 2014, 10:32am UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/5 "2014-08-20T10:32:17Z")

</div>

Hi Alex,

I have sent you and Stephane Dropbox invites, couldn’t find Piere’s email address.

As for your suspicion: I have recently used a similar scenario (open model) for endurance testing another application with even a higher load (constantUsersPerSec(500)) without facing these issues. That script contained only plain (rest) urls without ELFilebody . Also that script did not contain any nested randomSwitches.

I just remembered some other possible relevant detail: during the failed tests the SUT has some hiccups resulting in significant numbers of request timeOuts. Perhaps these sessions are not cleaned up properly?

Cheers

Daniel

---

<div class="post-metadata">

**Author:** ![Alex\_Bagehot](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@Alex\_Bagehot](https://community.gatling.io/u/Alex_Bagehot)\
**Post date:** [August 22, 2014, 6:49am UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/6 "2014-08-22T06:49:09Z")

</div>

Hi Daniel,  
Thanks I loaded up the heap dump last night and raised a ticket for it. So it’s progressing and will be fixed.  
Thanks  
Alex

---

<div class="post-metadata">

**Author:** ![Pierre\_DAL-PRA](https://avatars.discourse-cdn.com/v4/letter/p/ecccb3/32.png) [@Pierre\_DAL-PRA](https://community.gatling.io/u/Pierre_DAL-PRA)\
**Post date:** [August 22, 2014, 1:41pm UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/7 "2014-08-22T13:41:22Z")

</div>

Hi Daniel,

My email is pdalpra at excilys dot com.  
Could you sent my invite so that I have a closer look at your heap dump ?

---

<div class="post-metadata">

**Author:** ![Excilys](https://avatars.discourse-cdn.com/v4/letter/e/8797f3/32.png) [@Excilys](https://community.gatling.io/u/Excilys)\
**Post date:** [August 22, 2014, 1:56pm UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/8 "2014-08-22T13:56:06Z")

</div>

Ok, on pre-schedule immédiatement 20M de sessions, mais quand même. Je soupçonne qu’il sauve trop de données en session.

---

<div class="post-metadata">

**Author:** ![Pierre\_DAL-PRA](https://avatars.discourse-cdn.com/v4/letter/p/ecccb3/32.png) [@Pierre\_DAL-PRA](https://community.gatling.io/u/Pierre_DAL-PRA)\
**Post date:** [August 22, 2014, 2:06pm UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/9 "2014-08-22T14:06:52Z")

</div>

Ouep, c’est ce que je me disais aussi, c’est pas 20 Mo de sessions qui crachent 6 Go de heap, c’est pour ça que je voudrais avoir plus de détails…

---

<div class="post-metadata">

**Author:** ![Alex\_Bagehot](https://avatars.discourse-cdn.com/v4/letter/a/d78d45/32.png) [@Alex\_Bagehot](https://community.gatling.io/u/Alex_Bagehot)\
**Post date:** [August 22, 2014, 2:09pm UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/10 "2014-08-22T14:09:34Z")

</div>

it look a while to load the dump as it;s large so to save you some time:

there’s 2 problems:

1. there’s a fair amount of memory being used by the upfront creation of the users. but clearly it starts and runs ok for some time. all is needed in this case is a larger heap size. having said that, if there is any quick low risk wins to reduce the memory here it would be of benefit to avoid carrying it for the duration of the test.for example there’s 22M Scenarios taking 1/2 GB alone.

Class Name | Objects | Shallow Heap | Retained Heap

---

<div class="post-metadata">

**Author:** ![Excilys](https://avatars.discourse-cdn.com/v4/letter/e/8797f3/32.png) [@Excilys](https://community.gatling.io/u/Excilys)\
**Post date:** [August 23, 2014, 8:44pm UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/11 "2014-08-23T20:44:10Z")

</div>

Hi Daniel,

Sorry, but could you compress your heap dump, please? My dropbox is only 2.5Go… ☹

Then, 64,319 sun.security.ssl.SSLEngineImpl instances seems crazy to me! Is there a chance that your server doesn’t have a keep alive timeout and never close the connections?

Cheers,

Stéphane

---

<div class="post-metadata">

**Author:** ![Daniel\_Moll](https://avatars.discourse-cdn.com/v4/letter/d/97f17d/32.png) [@Daniel\_Moll](https://community.gatling.io/u/Daniel_Moll)\
**Post date:** [August 24, 2014, 6:07pm UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/12 "2014-08-24T18:07:35Z")

</div>

Hi Stephane, did you have a good holiday?

The compressed heap dump was in the same dropbox folder already, but still \> 1Gb I’m afraid.

The server Gatling is talking to is an IBM DataPower. The hiccups I mentioned earlier seem to be caused by this component, our infrastructure team is currently investigating. The hiccups cause Gatling users to time out (response times increase up to \> 60s). Currently I am running my test in “closed system mode” and it looks like that Old Gen section of the heap increases after each hiccup as if those timed out sessions are not properly cleaned up by GC.

As soon as the DataPower issue has been fixed I will do a rerun of the test in “open system mode” to see if that fixes the leak in Gatling as well.

Cheers

Daniel

---

<div class="post-metadata">

**Author:** ![Excilys](https://avatars.discourse-cdn.com/v4/letter/e/8797f3/32.png) [@Excilys](https://community.gatling.io/u/Excilys)\
**Post date:** [August 24, 2014, 9:54pm UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/13 "2014-08-24T21:54:14Z")

</div>

> Hi Stephane, did you have a good holiday?

Yep, great, thanks!

> The compressed heap dump was in the same dropbox folder already, but still  
> \> 1Gb I'm afraid.

What happens is that you shared the whole folder, so Dropbox wants me to  
have enough space for the total size, not only a given file.  
I have 2.5Gb, so just sharing the compressed file would work.

> The server Gatling is talking to is an IBM DataPower. The hiccups I  
> mentioned earlier seem to be caused by this component, our infrastructure  
> team is currently investigating. The hiccups cause Gatling users to time  
> out (response times increase up to \> 60s). Currently I am running my test  
> in "closed system mode" and it looks like that Old Gen section of the heap  
> increases after each hiccup as if those timed out sessions are not properly  
> cleaned up by GC.
> 
> As soon as the DataPower issue has been fixed I will do a rerun of the  
> test in "open system mode" to see if that fixes the leak in Gatling as well.

Interesting. That would mean that timed out channels still linger in the  
pool?! I'll investigate tomorrow.  
How is your keep alive timeout set up?

Cheers,

Stéphane

---

<div class="post-metadata">

**Author:** ![Excilys](https://avatars.discourse-cdn.com/v4/letter/e/8797f3/32.png) [@Excilys](https://community.gatling.io/u/Excilys)\
**Post date:** [August 24, 2014, 9:59pm UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/14 "2014-08-24T21:59:02Z")

</div>

OK, just realized you’ve removed the other files.  
Downloading right now, thank!

---

<div class="post-metadata">

**Author:** ![Excilys](https://avatars.discourse-cdn.com/v4/letter/e/8797f3/32.png) [@Excilys](https://community.gatling.io/u/Excilys)\
**Post date:** [August 25, 2014, 7:34am UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/15 "2014-08-25T07:34:39Z")

</div>

So, there might be other problems that I will investigate later, but what’s for sure is that our upfront scheduled users eats between 2 and 5 Go!!!  
The good news is that it’s easy to fix.

Stay tuned.

---

<div class="post-metadata">

**Author:** ![Excilys](https://avatars.discourse-cdn.com/v4/letter/e/8797f3/32.png) [@Excilys](https://community.gatling.io/u/Excilys)\
**Post date:** [August 25, 2014, 10:13am UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/16 "2014-08-25T10:13:26Z")

</div>

Hey Daniel,

I’ve just pushed a first fix: [https://github.com/gatling/gatling/issues/2129](https://github.com/gatling/gatling/issues/2129)

Could you grab a snapshot from Sonatype in about 10min (once Travis has finished building) and give it a try, please?

---

<div class="post-metadata">

**Author:** ![Daniel\_Moll](https://avatars.discourse-cdn.com/v4/letter/d/97f17d/32.png) [@Daniel\_Moll](https://community.gatling.io/u/Daniel_Moll)\
**Post date:** [August 25, 2014, 6:02pm UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/17 "2014-08-25T18:02:50Z")

</div>

Hi Stephane,

Thanks for the swift response, as always 🙂

I’ve tried the Sonatype snapshot but it returns this error when I try to start the script:

java.lang.reflect.InvocationTargetException  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:601)  
at scala\_maven\_executions.MainHelper.runMain(MainHelper.java:164)  
at scala\_maven\_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26)  
Caused by: java.lang.NoSuchMethodError: io.gatling.core.util.StringHelper$.checkSupportedJavaVersion()V  
at io.gatling.app.Gatling.start(Gatling.scala:88)  
at io.gatling.app.Gatling$.fromMap(Gatling.scala:54)  
at io.gatling.app.Gatling$.runGatling(Gatling.scala:79)  
at io.gatling.app.Gatling$.runGatling(Gatling.scala:58)  
at io.gatling.app.Gatling$.main(Gatling.scala:50)  
at io.gatling.app.Gatling.main(Gatling.scala)  
… 6 more

Can you help me out please? 🙂

Cheers

Daniel

---

<div class="post-metadata">

**Author:** ![Excilys](https://avatars.discourse-cdn.com/v4/letter/e/8797f3/32.png) [@Excilys](https://community.gatling.io/u/Excilys)\
**Post date:** [August 25, 2014, 6:09pm UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/18 "2014-08-25T18:09:35Z")

</div>

Are you sure you didn’t merge with some old jars when you unpacked?  
This method is definitively here: [https://github.com/gatling/gatling/blob/master/gatling-core/src/main/scala/io/gatling/core/util/StringHelper.scala#L44](https://github.com/gatling/gatling/blob/master/gatling-core/src/main/scala/io/gatling/core/util/StringHelper.scala#L44)

---

<div class="post-metadata">

**Author:** ![Daniel\_Moll](https://avatars.discourse-cdn.com/v4/letter/d/97f17d/32.png) [@Daniel\_Moll](https://community.gatling.io/u/Daniel_Moll)\
**Post date:** [August 25, 2014, 6:48pm UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/19 "2014-08-25T18:48:52Z")

</div>

Hi Stephane,

I have removed all jars from my local maven repo and rebuild everything and now it works. I see Gatling is still allocating \>3Gb (of 6Gb max) at the start of the test, is this expected behaviour? I’ll let the test run through the night, let’s see if it’s still alive in the morning 🙂

Cheers

Daniel

---

<div class="post-metadata">

**Author:** ![Excilys](https://avatars.discourse-cdn.com/v4/letter/e/8797f3/32.png) [@Excilys](https://community.gatling.io/u/Excilys)\
**Post date:** [August 25, 2014, 6:54pm UTC](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304/20 "2014-08-25T18:54:32Z")

</div>

No, it’s not expected (your Xms is 2g, I wouldn’t expect more). Could you provide a heap dump on start up please?

[Next page](https://community.gatling.io/t/2-0-0-rc2-increasing-heap-size-issue/1304.md?page=2)