Heap size

Johan_Peeters · March 27, 2014, 10:12am

My Gatling script runs out of memory after a while:

java.lang.OutOfMemoryError: Java heap space
at org.jboss.netty.buffer.HeapChannelBuffer.(HeapChannelBuffer.java:42) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.buffer.BigEndianHeapChannelBuffer.(BigEndianHeapChannelBuffer.java:34) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:68) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.buffer.AbstractChannelBufferFactory.getBuffer(AbstractChannelBufferFactory.java:48) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.newCumulationBuffer(FrameDecoder.java:507) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.handler.codec.frame.FrameDecoder.updateCumulation(FrameDecoder.java:345) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:487) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:92) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) ~[netty-3.8.0.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) ~[netty-3.8.0.Final.jar:na]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [na:1.6.0_65]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [na:1.6.0_65]
at java.lang.Thread.run(Thread.java:695) [na:1.6.0_65]

I am starting Gatling with Maven:

$ mvn gatling:execute

Monitoring memory usage for the Gatling JVM, I notice that it starts small, about 100MiB, and grows slowly over the first couple of minutes to a little over twice that. Then the server is overloaded and requests start timing out, so there is a sudden jump in memory usage to about 650MiB. Soon after, the OutOfMemoryError occurs.
So I tried to increase heap space. I have tried various things, including MAVEN_OPTS, JAVA_OPTS and configuring the execute goal of the gatling plugin. Illustrating the latter, this is a fragment of my pom:

com.excilys.ebi.gatling gatling-maven-plugin test execute -Xmx2048m -Xms512m -XX:MaxPermSize=512m ...

I also tried with

-mx2048m -ms512m -XX:MaxPermSize=512m

Neither seemed to have any effect.
The machine that I am running this test on has plenty of spare RAM.
Also,

$ mvn --version
Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17 17:22:22+0200)
Maven home: /usr/local/Cellar/maven/3.1.1/libexec
Java version: 1.6.0_65, vendor: Apple Inc.
Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
Default locale: en_US, platform encoding: MacRoman
OS name: “mac os x”, version: “10.9.2”, arch: “x86_64”, family: “mac”

What am I missing?

thx for any suggestions,

Yo

Excilys · March 27, 2014, 10:18am

My2cents: you’re saving too much data into your users’ sessions, such as full response bodies.

VisualVM/YourKit/JProfiler will tell you.

Johan_Peeters · March 27, 2014, 10:51am

I don't think I am saving any response bodies. Here is my scenario:

scenario("Provisioning")
    .feed(sessionIDFeeder)
    .during(30 minutes) {
      exec(
        http("provisioning")
          .post(localURI)
          .headers(headers)
          .fileBody("SyncML-template",
            Map("vid" -> "${vid}",
              "target" -> (target + ":80" + localURI),
              "sessionid" -> "${sessionid}",
              "tp" -> "${tp}")).asXML
          .check(status.is(200))
          .check(xpath("""//pre:SyncBody/pre:Status/pre:Cmd[text()="SyncHdr"]/following-sibling::pre:Data""",
List("pre" -> "SYNCML:SYNCML1.1")).is("212"))
          )
        .pause(60)
    }

The Eclipse Memory Analyzer tells me that 75% of memory is consumed by
akka.jsr166y.ForkJoinWorkerThreads.
I am trying to run a lot of concurrent simulated users, so I am
expecting memory demand to be high. That's why I am trying to increase
available memory.

Excilys · March 27, 2014, 10:57am

The Eclipse Memory Analyzer tells me that 75% of memory is consumed by
akka.jsr166y.ForkJoinWorkerThreads.

That doesn’t mean anything, of course threads are the root of most the state in this kind of application.

How many concurrent users are we talking about?
Could you upload your memory dump somewhere please?

Johan_Peeters · March 27, 2014, 11:10am

2250 users. I don't think they are all executing yet by the time the
script crashes.

You can download the memory dump here:
https://drive.google.com/file/d/0B1PDwg-loFn4QkprYzlrVVlCZnc/edit?usp=sharing

Excilys · March 27, 2014, 11:24am

There’s a 11 org.apache.xerces.dom.DeferredDocumentImpl for a total retained size of 225Mb.
Any chance you’re doing XPath with sometimes huge documents?

Johan_Peeters · March 27, 2014, 12:26pm

The documents I am xpath'ing at the moment should be about 1500b. I
would not call that huge.

Would a regex require less memory?
Why can I not increase heap size?

Excilys · March 27, 2014, 12:36pm

The documents I am xpath'ing at the moment should be about 1500b. I
would not call that huge.

Well, the heap dump doesn't lie

There's usually a 3 ratio between binary size and object size in Java XML,
so still, I don't think your documents are 1500b, most probably several Mb.

Would a regex require less memory?

Absolutely. And probably less CPU too.

FYI, we've switch XPath engine in upcoming Gatling 2M4 milestone.

Why can I not increase heap size?

There was a bug with the gatling-maven-plugin that caused the jvmArgs
option to be ignored.
This was fixed in 1.5.4, please upgrade.

Which version do you use?

Johan_Peeters · March 27, 2014, 1:26pm

The documents I am xpath'ing at the moment should be about 1500b. I
would not call that huge.

Well, the heap dump doesn't lie

true

There's usually a 3 ratio between binary size and object size in Java XML,
so still, I don't think your documents are 1500b, most probably several Mb.

I will have to investigate where those are coming from.
Thx.

Would a regex require less memory?

Absolutely. And probably less CPU too.

FYI, we've switch XPath engine in upcoming Gatling 2M4 milestone.

Why can I not increase heap size?

There was a bug with the gatling-maven-plugin that caused the jvmArgs option
to be ignored.
This was fixed in 1.5.4, please upgrade.

Which version do you use?

I used Gatling 1.5.4 and high-charts1.5.3.
After upgrading high-charts to 1.5.4, I can now get through the test,
although the total JVM memory still hovers around 670 MB. The -ms512m
option seems to have been ignored too as the VM started up around 200
MB.
So, I am not convinced that the jvmArgs option is having effect.

But I probably have enough now to get my script to run. Thx,

Yo

Excilys · March 27, 2014, 1:30pm

So, I am not convinced that the jvmArgs option is having effect.

That's because you did it wrong

bad:
<jvmArgs>-Xmx2048m -Xms512m -XX:MaxPermSize=512m</jvmArgs>

good:
<jvmArgs>
    <jvmArg>-Xmx2048m</jvmArg>
    <jvmArg>-Xms512m</jvmArg>
    <jvmArg>-XX:MaxPermSize=512m</jvmArg>
</jvmArgs>

just like in the documentation:

Spencer_Chastain · March 27, 2014, 3:31pm

I also think the configuration block is misplaced.

In order to get the configured jvmArgs to be passed, I had to move the configuration block outside of the executions block.

Just took me something like a day (in between sorting through other upgrade issues) to figure that out

Johan_Peeters · March 27, 2014, 3:44pm

any chance of digging out where the JVM args configuration needs to
go? I still do not seem to be getting the configured heap.

Spencer_Chastain · March 27, 2014, 4:34pm

Something to do to quickly see if you’re getting the memory you want is to include this line before your setUp():

println("Heap: " + (Runtime.getRuntime().maxMemory() / 1024 / 1024))

This is my pom:

4.0.0

io.gatling
gatling-maven-plugin-demo
2.0.0-SNAPSHOT
jar

excilys Excilys Repository [http://oss.sonatype.org/content/repositories/snapshots/](http://oss.sonatype.org/content/repositories/snapshots/) excilys Excilys Repository [http://oss.sonatype.org/content/repositories/snapshots/](http://oss.sonatype.org/content/repositories/snapshots/) UTF-8 2.0.0-SNAPSHOT io.gatling.highcharts gatling-charts-highcharts ${gatling.version} test

org.scala-lang
scala-compiler
2.10.3

io.fastjson
boon
0.7

io.gatling gatling-maven-plugin 2.0.0-SNAPSHOT test execute -server -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms512M -Xmx6g -Xmn100M -Xss2M -XX:+HeapDumpOnOutOfMemoryError -XX:+AggressiveOpts

-XX:+OptimizeStringConcat
-XX:+UseFastAccessorMethods
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:+CMSClassUnloadingEnabled
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1

advanced/*.scala

Johan_Peeters · April 1, 2014, 10:22am

I have been busy with other things for a few days, so couldn't follow
up. Coming back to my simulation, I still do not see any evidence of
the JVM arguments being taken into account. I tried both Stéphane's
and Spencer's suggestions, but no joy.

I am running
+ Scala 2.9.3
+ Gatling 1.5.4
+ Gatling Highcharts 1.5.4
+ Scala Maven plugin 3.1.5
+ Gatling Maven plugin 1.5.4

Excilys · April 1, 2014, 11:08am

I get it, I forgot to backport the fix on Gatling 1…
https://github.com/excilys/gatling/issues/1628

Sorry for this.

I will release 1.5.5 later today.

Topic		Replies	Views
OOM on gatling Gatling (Open-Source)	8	111	June 28, 2013
Java heap space exception Gatling (Open-Source)	8	106	February 5, 2015
java.lang.OutOfMemoryError: Java heap space Gatling (Open-Source)	5	165	September 11, 2014
Memory Issue due to 'java.lang.OutOfMemoryError: Java Heap Space' in Scenario with Extensive Code Gatling (Open-Source)	4	353	September 16, 2023
java.lang.OutOfMemoryError: Java heap space after upgrading to 3.2.0 Gatling (Open-Source)	1	117	August 8, 2019

Heap size

Related Topics