java.lang.OutOfMemoryError: Direct buffer memory issue

Gatling (Open-Source)
1 
Hi, 



I keep on getting Direct buffer OOM and am not sure what can cause it.

I've already tried to increase Direct buffer memory but no success. 

My VM settings are:



-Xms10g
-Xmx16g
-XX:MaxDirectMemorySize=5g


Did anyone encounter this or can guide me at what could be the cause?



Thanks!





  [gatling-http-1-17][ERROR][HttpAppHandler.java:230] i.g.h.c.i.HttpAppHandler - Fatal error
  java.lang.OutOfMemoryError: Direct buffer memory
  	at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
  	at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
  	at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
  	at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:758)
 	at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:734)
  	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245)
  	at io.netty.buffer.PoolArena.allocate(PoolArena.java:227)
  	at io.netty.buffer.PoolArena.allocate(PoolArena.java:147)
  	at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:342)
  	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
  	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
  	at io.netty.handler.ssl.SslHandler.allocate(SslHandler.java:2141)
  	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1324)
  	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224)
  	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)
  	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
  	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
  	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
  	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
  	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
  	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
  	at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  	at java.base/java.lang.Thread.run(Thread.java:834)
2

How much total memory do you have on this instance?
You’re eagerly allocating 10G for heap, so it leaves (total memory - 10G) for OS, other processes and IO direct buffers.

3

Hi Stephane and than you for your answer.
I am using a c5.4xlarge AWS instance with 32GB ram.

By the way, I came across the following thread which seems to point on the same issue.
Seems to be resolved by not using tcnative, is it relevant to us?

4

I am using a c5.4xlarge AWS instance with 32GB ram.

So you have plenty of RAM and you hit your “-XX:MaxDirectMemorySize=5g” limit.

Does your test involve large uploads? If so, I suspect you’re uploading too fast and hit a limit: bandwidth, NIC, network, system under load that can’t keep up with the pace and read from the socket.

By the way, I came across the following thread which seems to point on the same issue.

Seems to be resolved by not using tcnative, is it relevant to us?

No. This ticket was nonsensical and the user didn’t fully understand what he was doing. The ticket was closed because of that.

5

I am not uploading any files but post JSONS which are relatively small (Largest one is 515 Byes).
However, due to the amount of users I am running (50k), I end up uploading 291 MB per second.

c5.4xlarge NIC supports “Up to 10” Gbps so I dont see any issue here.

CPU on the machine reaches around 90% which could be an issue.

6

CPU on the machine reaches around 90% which could be an issue.

You probably have your answer. Moreover, you want to check how fast the server side can ACK the packets.

7

Thank you very much.