"java.net.ConnectException: Cannot assign requested address"

Gatling (Open-Source)
1

Hi,

I am running tests on a m3.xlarge (4 CPU, 15GB RAM).

I am ramping up to 500rps for a period of 10 minutes.

rampUsersPerSec(1) to(500) during(10.minutes)

I have set my open-file limit to 65535

$ ulimit -n
=> 65535

But, I am still receiving “java.net.ConnectException: Cannot assign requested address”.

Do I now need to distribute my testing?

Aidy

2

How long do your virtual users live?
How many open connections do you have?
How are your time wait and ephemeral port range values?

3

Hi Stephane,

I followed all the Linux tuning suggestions here: http://gatling.io/docs/2.0.0-RC3/general/operations.html - which got rid of the problem!

Many Thanks

Aidy

4

Great.
I thought you had already followed them, and was a bit puzzled :slight_smile:

5

Hi

I got the same error.

Also I followed the instructions from http://gatling.io/docs/2.0.0-RC2/general/operations.html#os-tuning

These are my configuration:
ulimit -n
65535
nano /proc/sys/net/ipv4/tcp_fin_timeout
30
nano /proc/sys/net/ipv4/ip_local_port_range
32768 65535
Also I changed

/etc/pam.d/sshd , I added session required pam_limits.so
/etc/ssh/sshd_config, I changed UseLogin yes

I am running this test on 10 machines . This should throw: 500k RPM during 5 min → 2500000 request.

setUp(

test1.inject(constantUsersPerSec(833) during (500 seconds))

).protocols(httpConf)

What am I doing wrong? When I tried to reach 1 million is ok, but with more than 2 million gatling or the machines go down.

Can someone help me?

Thanks in advance.

6

I’m having the same issue on ubuntu 14.04.

Just checked the OS tuning section:

$ ulimit -a
…
open files (-n) 65536

$ cat /proc/sys/fs/nr_open
1048576

$ cat /proc/sys/fs/file-max
1621050

session required pam_limits.so is set in all 3 files

$ cat /proc/sys/net/ipv4/tcp_fin_timeout
60

Can you please advise?

Thank you,
Emanuel

7

On Linux run

ss -s

and look at your TCP count to see if it is around your file descriptor limit.

Otherwise, the message may be from your web server (I think)!

8

Thank you for the quick reply.

$ ss -s
Total: 1087 (kernel 0)
TCP: 64775 (estab 233, closed 64510, orphaned 0, synrecv 0, timewait 64509/0), ports 0

It seems to break when it gets around 65535, however, I’ve now set all limits 100 times that

$ ulimit -n
6553600

$ cat /proc/sys/fs/file-max
6553600

$ cat /proc/sys/fs/nr_open
6553600

Since available ports for testing are between 1025 and 65535, it’s probably because of that.

I’m currently having constantUsersPerSec(1000) that just make one request.

I’m trying to make gatling recycle connections with below settings but no success so far.

allowPoolingConnections = true             # Allow pooling HTTP connections (keep-alive header automatically added)
allowPoolingSslConnections = true          # Allow pooling HTTPS connections (keep-alive header automatically added)
maxConnectionsPerHost = 1000                 # Max number of connections per host (-1 means no limit)
maxConnections = 1000

Can you spot anything wrong?

Thanks

9

Are you the one asking on SOF too? http://stackoverflow.com/questions/34355828/why-is-gatling-pausing-to-send-requests/34356777

ephemeral ports starvation

10

No, I’m not…

11

Try .shareConnections at the protocol level.

12

But then your connection profile will be different. Chose the one that matches your needs/live system behavior.

You can also increase your ephemeral port range and reduce your tcp time_wait.

If you have control over your network, you can also create aliases for your NI and bind from different local addresses.

13

Yes, you must ask yourself whether a thousand users per second will hit one URL and close the connection. Other load test tools share connections which is unrealistic for browser based tests.

I found this site useful for Linux tuning: http://www.nateware.com/linux-network-tuning-for-2013.html#.VnQ7hXOnzbt

And you can also run off multiple machines and cat the logs.

14

Thanks, that helped. Although, in the end I’ve ended up refactoring my tests so that a user would make multiple requests.

15

Sharing connections did the trick for me!

16

shareConnections = simulating server to server traffic

If that’s your use case, fine.

But if your use case is simulating web traffic, no, it didn’t “make the trick”.
You actually broke your test and made it meaningless.
You have to properly lift your OS limits as described in this thread and in our documentation, and if it’s not enough, resort to distributed tests eg with Gatling Enterprise.