j.n.ConnectException: Received fatal alert: handshake_failure with Gatling 2.2.0 but not with 2.1.7

Nguyen · May 6, 2016, 8:29pm

Hi,

We have simulation that consistently fails on some AWS EC2 instances with Gatling 2.2.0 with j.n.ConnectException: Received fatal alert: handshake_failure. This happens on multiple client machines. However, once we switched back to 2.17, the issue doesn’t occur with the same simulation. We made sure the gattling.conf are the same (default).

Has anyone else seen this?

Nguyen

2016-05-06 10:21:07,002; HttpTx$; DEBUG- Sending request=01. QuoteRequest uri=https://test-north.rc.ripple.com/v2/payments/quote?sender_uri=acct%3AMoses.Lloyd%40test-north.rc.ripple.com&receiver_uri=acct%3AKaylee.Kido%40test-south.rc.ripple.com&receiver_amount=10.29%2BEUR: scenario=RC Payment, userId=10
2016-05-06 10:21:07,004; LoggingHandler; DEBUG- [id: 0xb8be179f] REGISTERED
2016-05-06 10:21:07,004; LoggingHandler; DEBUG- [id: 0xb8be179f] CONNECT(test-north.rc.ripple.com/54.165.49.107:443, null)
2016-05-06 10:21:07,083; LoggingHandler; DEBUG- [id: 0xb8be179f, L:/10.15.21.151:56906 - R:test-north.rc.ripple.com/54.165.49.107:443] ACTIVE
2016-05-06 10:21:07,157; NettyConnectListener; DEBUG- Trying to recover from failing to connect channel [id: 0xb8be179f, L:/10.15.21.151:56906 - R:test-north.rc.ripple.com/54.165.49.107:443] with a retry value of true 
2016-05-06 10:21:07,157; NettyConnectListener; DEBUG- Failed to recover from connect exception: javax.net.ssl.SSLException: Received fatal alert: handshake_failure with channel [id: 0xb8be179f, L:/10.15.21.151:56906 - R:test-north.rc.ripple.com/54.165.49.107:443]
2016-05-06 10:21:07,158; AsyncHandler; DEBUG- Request '01. QuoteRequest' failed for user 10
java.net.ConnectException: Received fatal alert: handshake_failure
	at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:138)
	at org.asynchttpclient.netty.channel.NettyConnectListener$1.onFailure(NettyConnectListener.java:109)
	at org.asynchttpclient.netty.SimpleFutureListener.operationComplete(SimpleFutureListener.java:26)
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:683)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:604)
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:564)
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:425)
	at io.netty.handler.ssl.SslHandler.notifyHandshakeFailure(SslHandler.java:1239)
	at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1234)
	at io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1209)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1064)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:904)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:387)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:245)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:278)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:962)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:485)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:399)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:371)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
	at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLException: Received fatal alert: handshake_failure
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
	at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
	at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1098)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:970)
	... 14 common frames omitted
2016-05-06 10:21:07,160; ResponseProcessor; WARN - Request '01. QuoteRequest' failed: j.n.ConnectException: Received fatal alert: handshake_failure

Jose_Manuel_Vega_Mon · May 13, 2016, 12:05pm

It looks like a SSL issue.

Certificate is right?

Cheers,

slandelle · May 13, 2016, 8:15pm

For the record: server was using a 256 bit RSA key, but user’s JDK didn’t had the extended policies.

yuliya.brynzak · September 28, 2016, 11:52am

Hi,

I have a similar problem. For some of our web application the recorder works fine (I can add certificate exception), but for one web application, Gatling throws the exception(as provided in the topic).

The web app has a valid certificate. I use Gatling 2.2.0.

Does someone know what it could be?

Yuliya

Rich_Aurora · October 6, 2016, 1:43pm

Hello,

would you be able to provide a little more detail or direction on the fix? I’m running into this error using 2.2.2, on a fairly recent 1.8 jdk. thank you.

Henry_Rodrick · October 26, 2016, 2:47pm

I’ve run into the problem as well on various versions of the 1.8 JDK (tried various JDKs such as OpenJDK, Oracle JDK, and IBM JDK, on various systems running Debian, RHEL 6 and Mac OS X). Saw the issue with both 2.2.0 and a 2.2.3-SNAPSHOT version.

The workaround of downgrading to 2.1.7 worked, as mentioned in the title.

However, I had no luck with extended policies (I’m assuming extended policies means “Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 7”).

In any case, the problem seems to be in the AsyncHttpClient library which is used by Gatling. I filed a report with my investigation over here: https://github.com/AsyncHttpClient/async-http-client/issues/1290

Cheers
Henry

Kavya_reddy · October 26, 2016, 8:02pm

I have seen this issue too from AWS machines. Did the workaround(downgrading) work for you?

-Kavya

Henry_Rodrick · October 26, 2016, 10:02pm

Yes, downgrading worked. My specific issue was related to ECDSA cipher suite support in netty, which has been fixed recently. Another workaround that was successful for me was to explicitly set a list of allowed ECDHE+ECDSA cipher suites in gatling.conf (please see the discussion here: https://github.com/AsyncHttpClient/async-http-client/issues/1290)

According to a Gatling developer, using the lastest 2.2.3-SNAPSHOT build of Gatling should in theory solve the problem, but I haven’t tested it yet.

Your issue might be a slightly different one, but I would definitely try downgrading in any case as a starting point.
Cheers
Henry

Nikolay_Degkin · November 8, 2016, 11:53am

Could you guys please provide more details to resolve this issue?
I’m facing it on 2.1.7

`
javax.net.ssl.SSLException: Received fatal alert: handshake_failure
at sun.security.ssl.Alerts.getSSLException(Unknown Source) ~[na:1.8.0_111]
at sun.security.ssl.SSLEngineImpl.fatal(Unknown Source) ~[na:1.8.0_111]
at sun.security.ssl.SSLEngineImpl.fatal(Unknown Source) ~[na:1.8.0_111]
at sun.security.ssl.SSLEngineImpl.recvAlert(Unknown Source) ~[na:1.8.0_111]
at sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source) ~[na:1.8.0_111]
at sun.security.ssl.SSLEngineImpl.readNetRecord(Unknown Source) ~[na:1.8.0_111]
at sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source) ~[na:1.8.0_111]

at javax.net.ssl.SSLEngine.unwrap(Unknown Source) ~[na:1.8.0_111]
at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218) [netty-3.10.4.Final.jar:na]
… 18 common frames omitted
Wrapped by: java.net.ConnectException: Received fatal alert: handshake_failure
at com.ning.http.client.providers.netty.request.NettyConnectListener.onF
utureFailure(NettyConnectListener.java:133) [async-http-client-1.9.30.jar:na]
at com.ning.http.client.providers.netty.request.NettyConnectListener.acc
ess$200(NettyConnectListener.java:37) [async-http-client-1.9.30.jar:na]
at com.ning.http.client.providers.netty.request.NettyConnectListener$1.o
perationComplete(NettyConnectListener.java:104) [async-http-client-1.9.30.jar:na
]
`

Henry_Rodrick · November 8, 2016, 12:04pm

Well, handshake_failure is quite generic so it might just be a different SSL related issue.

Running with -Djavax.net.debug=ssl (or -Djavax.net.debug=all) should give you more details (make sure that you look at the output from when the test is running and not when Gatling itself is starting up).

Cheers
Henry

Nikolay_Degkin · November 8, 2016, 12:53pm

By the way, everything is fine on my 2.1.7 with AES_128_GCM but having problems with AES_256_GCM

Henry_Rodrick · November 8, 2016, 1:18pm

There are known performance issues with GCM (http://stackoverflow.com/questions/25992131/slow-aes-gcm-encryption-and-decryption-with-java-8u20) so perhaps longer keys are disabled for that reasons in your jvm… Does AES_256_CBC work? (Also, would be interesting to see what cipher suites you get from your debug output. It should have a full listing.)