java.net.BindException: Address not available

Gatling (Open-Source)
1

Hi All,

I am using Gatling to load test a spring application build using Spring webflux and mongo databse. My API just takes a string parameter and does a get call on the mongo reactive repository. My application is deployed in Google cloud GKE and GCE (mongo DB). Gatling test access the API end point over internet using public IP.

Test works fine till the load 28000 concurrent users, when I inject 30K users at once , I get ‘java.net.BindException: Address not available’. Can someone please give some idea on the issue.

Request:
getEligibilityMVCSimulation: KO i.n.c.AbstractChannel$AnnotatedSocketException: Address not available: /some_ip:80

Session:
Session(getEligibilityMVCSimulation,30975,1556538559546,Map(gatling.http.ssl.sslContexts → SslContexts(io.netty.handler.ssl.OpenSslClientContext@3ff8fa07,None), gatling.http.cache.dns → io.gatling.http.cache.DnsCacheSupport$$anon$1@5273a181, gatling.http.cache.baseUrl → http://localhost:8182/eligibility-service/v1/),0,KO,List(),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$435/194107588@4249260d)

HTTP request:
GET http://some_ip/eligibility-service/v1/8672
headers=
Accept: application/json, text/javascript, /; q=0.01
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
origin: http://some_ip
host: some_ip

HTTP response:

11:49:30.614 [WARN ] i.g.h.e.GatlingHttpListener - Request ‘getEligibilityMVCSimulation’ failed for user 30983
java.net.BindException: Address not available
… 34 common frames omitted
Wrapped by: io.netty.channel.AbstractChannel$AnnotatedSocketException: Address not available: /some_ip
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:454)
at sun.nio.ch.Net.connect(Net.java:446)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83)
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80)
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:312)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:254)
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1366)
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545)
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:497)
at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:298)
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545)
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530)
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545)
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530)
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50)
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:545)
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:530)
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:512)
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:1024)
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:259)
at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:252)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
11:49:30.614 [WARN ] i.g.h.e.r.DefaultStatsProcessor - Request ‘getEligibilityMVCSimulation’ failed for user 30983: i.n.c.AbstractChannel$AnnotatedSocketException: Address not available: /some_ip:80

2

You ran out of local port that Gatling can use to open new connections, did you try the optimizations showed here : https://gatling.io/docs/current/general/operations/ ?

3

Hi , Thanks for reply. I tried almost all the steps given in link but no luck with the exception.
I am working on GCE VM with Debian 9 stretch. Initially ulimit -n was returning 1024 and after I followed all the steps in https://gatling.io/docs/current/general/operations/ ulimit -a is returning as 65535. But I am getting same exception.
ulimit -a gives below result,

joy123456@gatling-tests-vm-2:~$ ulimit -a

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 60087
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 60087
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

I am running gatling as a docker container using image https://github.com/denvazh/gatling . I checked ulimit on docker container, and it gives below result,

joy123456@gatling-tests-vm-2:~$
sudo docker run -it --entrypoint sh denvazh/gatling

/opt/gatling # ulimit -a

-f: file size (blocks) unlimited
-t: cpu time (seconds) unlimited
-d: data seg size (kb) unlimited
-s: stack size (kb) 8192
-c: core file size (blocks) unlimited
-m: resident set size (kb) unlimited
-l: locked memory (kb) 64
-p: processes unlimited
-n: file descriptors 1048576
-v: address space (kb) unlimited
-w: locks unlimited
-e: scheduling priority 0
-r: real-time priority 0

Apparently limit is set properly. am I missing anything ?

Thanks,
Vinay

4

Hi,

I got doubt if I pasted correct output.

On host machine (GCE VM Debian 9 stretch) I get below ulimit

`
joy123456@gatling-tests-vm:~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 29848
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 29848
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

`

and on Container ,

joy123456@gatling-tests-vm:~$ sudo docker run -it --entrypoint sh denvazh/gatling /opt/gatling # ulimit -a -f: file size (blocks) unlimited -t: cpu time (seconds) unlimited -d: data seg size (kb) unlimited -s: stack size (kb) 8192 -c: core file size (blocks) unlimited -m: resident set size (kb) unlimited -l: locked memory (kb) 64 -p: processes unlimited -n: file descriptors 1048576 -v: address space (kb) unlimited -w: locks unlimited -e: scheduling priority 0 -r: real-time priority 0

I followed all the steps mentioned in optimization reference doc but do not have any luck on the exception.

Thanks,
Vinay

5

IHMO this issue is not related to local port starvation as that’s the REMOTE address that can’t be reached.
Googling this error message a bit seems to point to a Docker issue.
You might want to ask on a Docker mailing list instead.

6

Thanks… I will try. It will be helpful if you can point me to any useful link on the issue.