Heap size

My Gatling script runs out of memory after a while:

java.lang.OutOfMemoryError: Java heap space
at org.jboss.netty.buffer.HeapChannelBuffer.(HeapChannelBuffer.java:42) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.buffer.BigEndianHeapChannelBuffer.(BigEndianHeapChannelBuffer.java:34) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:68) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChannelBufferFactory.java:48) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.newCumulationBuffer(FrameDecoder.java:507) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.updateCumulation(FrameDecoder.java:345) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:487) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[netty-3.8.0.Final.jar:na]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [na:1.6.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [na:1.6.0_65]
at java.lang.Thread.run(Thread.java:695) [na:1.6.0_65]

I am starting Gatling with Maven:

$ mvn gatling:execute

Monitoring memory usage for the Gatling JVM, I notice that it starts small, about 100MiB, and grows slowly over the first couple of minutes to a little over twice that. Then the server is overloaded and requests start timing out, so there is a sudden jump in memory usage to about 650MiB. Soon after, the OutOfMemoryError occurs.
So I tried to increase heap space. I have tried various things, including MAVEN_OPTS, JAVA_OPTS and configuring the execute goal of the gatling plugin. Illustrating the latter, this is a fragment of my pom:

com.excilys.ebi.gatling gatling-maven-plugin test execute -Xmx2048m -Xms512m -XX:MaxPermSize=512m ...

I also tried with

-mx2048m -ms512m -XX:MaxPermSize=512m

Neither seemed to have any effect.
The machine that I am running this test on has plenty of spare RAM.
Also,

$ mvn --version
Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17 17:22:22+0200)
Maven home: /usr/local/Cellar/maven/3.1.1/libexec
Java version: 1.6.0_65, vendor: Apple Inc.
Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
Default locale: en_US, platform encoding: MacRoman
OS name: “mac os x”, version: “10.9.2”, arch: “x86_64”, family: “mac”

What am I missing?

thx for any suggestions,

Yo

My2cents: you’re saving too much data into your users’ sessions, such as full response bodies.

VisualVM/YourKit/JProfiler will tell you.

I don't think I am saving any response bodies. Here is my scenario:

scenario("Provisioning")
    .feed(sessionIDFeeder)
    .during(30 minutes) {
      exec(
        http("provisioning")
          .post(localURI)
          .headers(headers)
          .fileBody("SyncML-template",
            Map("vid" -> "${vid}",
              "target" -> (target + ":80" + localURI),
              "sessionid" -> "${sessionid}",
              "tp" -> "${tp}")).asXML
          .check(status.is(200))
          .check(xpath("""//pre:SyncBody/pre:Status/pre:Cmd[text()="SyncHdr"]/following-sibling::pre:Data""",
List("pre" -> "SYNCML:SYNCML1.1")).is("212"))
          )
        .pause(60)
    }

The Eclipse Memory Analyzer tells me that 75% of memory is consumed by
akka.jsr166y.ForkJoinWorkerThreads.
I am trying to run a lot of concurrent simulated users, so I am
expecting memory demand to be high. That's why I am trying to increase
available memory.

The Eclipse Memory Analyzer tells me that 75% of memory is consumed by
akka.jsr166y.ForkJoinWorkerThreads.

That doesn’t mean anything, of course threads are the root of most the state in this kind of application. :slight_smile:

How many concurrent users are we talking about?
Could you upload your memory dump somewhere please?

2250 users. I don't think they are all executing yet by the time the
script crashes.

You can download the memory dump here:
https://drive.google.com/file/d/0B1PDwg-loFn4QkprYzlrVVlCZnc/edit?usp=sharing

There’s a 11 org.apache.xerces.dom.DeferredDocumentImpl for a total retained size of 225Mb.
Any chance you’re doing XPath with sometimes huge documents?

The documents I am xpath'ing at the moment should be about 1500b. I
would not call that huge.

Would a regex require less memory?
Why can I not increase heap size?

The documents I am xpath'ing at the moment should be about 1500b. I
would not call that huge.

Well, the heap dump doesn't lie :slight_smile:

There's usually a 3 ratio between binary size and object size in Java XML,
so still, I don't think your documents are 1500b, most probably several Mb.

Would a regex require less memory?

Absolutely. And probably less CPU too.

FYI, we've switch XPath engine in upcoming Gatling 2M4 milestone.

Why can I not increase heap size?

There was a bug with the gatling-maven-plugin that caused the jvmArgs
option to be ignored.
This was fixed in 1.5.4, please upgrade.

Which version do you use?

The documents I am xpath'ing at the moment should be about 1500b. I
would not call that huge.

Well, the heap dump doesn't lie :slight_smile:

true

There's usually a 3 ratio between binary size and object size in Java XML,
so still, I don't think your documents are 1500b, most probably several Mb.

I will have to investigate where those are coming from.
Thx.

Would a regex require less memory?

Absolutely. And probably less CPU too.

FYI, we've switch XPath engine in upcoming Gatling 2M4 milestone.

Why can I not increase heap size?

There was a bug with the gatling-maven-plugin that caused the jvmArgs option
to be ignored.
This was fixed in 1.5.4, please upgrade.

Which version do you use?

I used Gatling 1.5.4 and high-charts1.5.3.
After upgrading high-charts to 1.5.4, I can now get through the test,
although the total JVM memory still hovers around 670 MB. The -ms512m
option seems to have been ignored too as the VM started up around 200
MB.
So, I am not convinced that the jvmArgs option is having effect.

But I probably have enough now to get my script to run. Thx,

Yo

So, I am not convinced that the jvmArgs option is having effect.

That's because you did it wrong :slight_smile:

bad:
<jvmArgs>-Xmx2048m -Xms512m -XX:MaxPermSize=512m</jvmArgs>

good:
<jvmArgs>
    <jvmArg>-Xmx2048m</jvmArg>
    <jvmArg>-Xms512m</jvmArg>
    <jvmArg>-XX:MaxPermSize=512m</jvmArg>
</jvmArgs>

just like in the documentation:
https://github.com/excilys/gatling/wiki/Maven-plugin#advanced

I also think the configuration block is misplaced.

In order to get the configured jvmArgs to be passed, I had to move the configuration block outside of the executions block.

Just took me something like a day (in between sorting through other upgrade issues) to figure that out :frowning:

any chance of digging out where the JVM args configuration needs to
go? I still do not seem to be getting the configured heap.

Something to do to quickly see if you’re getting the memory you want is to include this line before your setUp():

println("Heap: " + (Runtime.getRuntime().maxMemory() / 1024 / 1024))

This is my pom:

4.0.0

io.gatling
gatling-maven-plugin-demo
2.0.0-SNAPSHOT
jar

excilys Excilys Repository [http://oss.sonatype.org/content/repositories/snapshots/](http://oss.sonatype.org/content/repositories/snapshots/) excilys Excilys Repository [http://oss.sonatype.org/content/repositories/snapshots/](http://oss.sonatype.org/content/repositories/snapshots/) UTF-8 2.0.0-SNAPSHOT io.gatling.highcharts gatling-charts-highcharts ${gatling.version} test

org.scala-lang
scala-compiler
2.10.3


io.fastjson
boon
0.7

io.gatling gatling-maven-plugin 2.0.0-SNAPSHOT test execute -server -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms512M -Xmx6g -Xmn100M -Xss2M -XX:+HeapDumpOnOutOfMemoryError -XX:+AggressiveOpts

-XX:+OptimizeStringConcat
-XX:+UseFastAccessorMethods
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+CMSClassUnloadingEnabled
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1

advanced/*.scala

I have been busy with other things for a few days, so couldn't follow
up. Coming back to my simulation, I still do not see any evidence of
the JVM arguments being taken into account. I tried both Stéphane's
and Spencer's suggestions, but no joy.

I am running
+ Scala 2.9.3
+ Gatling 1.5.4
+ Gatling Highcharts 1.5.4
+ Scala Maven plugin 3.1.5
+ Gatling Maven plugin 1.5.4

I get it, I forgot to backport the fix on Gatling 1…
https://github.com/excilys/gatling/issues/1628

Sorry for this.

I will release 1.5.5 later today.