OutOfDirectMemoryError when testing large file downloads

Gatling (Open-Source)
1

Hi there,

I am testing an app for its ability to provide large downloads to concurrent users. However, I am unable to complete my execution due to what seems to be a memory leak in my test scripts.

Here is my code:

def pollExportCompletionAndDownload()={

exec(session=> {
val newSession = session.set(“statusCode”,"")
newSession
})
.asLongAs(condition=session => session(“statusCode”).as[String] != “EXPORT_COMPLETE”, exitASAP=true) {
exec(_.remove(“statusCode”))
exec(flushHttpCache)
exec(http(requestName = “PollExportStatus”)
.get(baseurl + “/export/${exportID}/status?cb=” + System.currentTimeMillis.toString)
.header(“content-type”,“application/json”)
.header(“Accept”,“application/json, text/plain, /”)
.header(“accept-encoding”, “gzip, deflate, br”)
.basicAuth("", “”)
.check(jsonPath("$.statusCode").saveAs(“statusCode”)))
.exec(session => {
println(“Status:” + session(“statusCode”).as[String])
val newSession = session.set(“statusCode”, session(“statusCode”))
newSession
})
.pause(10)}
.exec(http(requestName = “DownloadExport”)
.get(baseurl+"/export/${exportID}")
.header(“Accept”,“text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8”)
.header(“accept-encoding”, “gzip, deflate, br”)
.basicAuth("","")
.check(status is 200))

}

The error I get is:

i.n.u.i.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 1056964615, max: 1073741824)

I have done this in my gradle.properties:

-XX:MaxDirectMemorySize=4g

Here is what my logback file looks like:

<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
        <pattern>%d{HH:mm:ss.SSS} [%-5level] %logger{15} - %msg%n%rEx</pattern>
        <immediateFlush>false</immediateFlush>
    </encoder>
</appender>
<timestamp key="bySecond" datePattern="yyyyMMdd'T'HHmmss" />
<appender name="FILE" class="ch.qos.logback.core.FileAppender">
    <file>Perftest-${bySecond}.txt</file>
    <append>false</append>
    <encoder>
        <pattern>%d{HH:mm:ss.SSS} [%-5level] %logger{15} - %msg%n%rEx</pattern>
        <immediateFlush>false</immediateFlush>
    </encoder>
</appender>
2

Anyone?

3
  1. How much native memory do you have available on this machine?
  2. What’s the size of the file you’re trying to download?
  3. How many concurrent downloads do you have when the crash happens?
4

Hi Stephen,

Answers:

  1. 16GB
  2. Size of each of the larger files is around 200MB or above
  3. 9 concurrent downloads of larger files

I am using gradle wrapper https://github.com/lkishalmi/gradle-gatling-plugin version 3.0.0

I do not know how to configure settings so that files are not stored in memory during or at completion of download.

Kindly advise.

Thanks,
Hasan

5

So you need a lot of direct memory, at least 3GB. You can set -XX:MaxDirectMemorySize.
How to configure this on this third party gradle plugin is not a question for this mailing list. Try reaching out the authors on their bug tracker.

6

Ok Stephen. So I assume there is not a way to make sure that the files are written to the disk through a stream as it arrives, instead of holding everything in memory?

Thanks,
Hasan

7

No.
Gatling only piles up the chunks in memory and assembles the full body when needed, typically when using a body check or enabling debug logging like you’re doing here.
You’re definitely killing Gatling’s performance.

8

Hi Stephen,

So I disabled debug logging, and removed the check on the header status code, but it did not make a difference. I did however manage to get gradle to assign more memory by putting this in build.gradle:

tasks.withType(JavaExec) {
    jvmArgs = ['-Xms512m', '-Xmx8g', '-XX:MaxDirectMemorySize=2g']
}

I kept monitoring the memory usage with the status check disabled and it still went up all the way to the limit.

I am being encouraged to look at your source code and figure out a way to stream directly to disc rather than hold the download in memory. Any pointers you can provide to that end would be most appreciated.

Thanks,
Hasan

9

The best to investigate this is to provide a sample application and a simulation against it that reproduce your problem.
Your wild guess that you can solve it by streaming onto the filesystem is wrong. This direct memory is used by Netty.