Server certificate upgraded, now everything is broken

jarrowwx · May 15, 2015, 4:49pm

Yesterday, our OPS person updated the HTTPS certificate in our performance environment. By hand, everything seems to work. But now all of our tests are failing with the error below. Is there a cache somewhere I need to clear in order to make things work again, or does this suggest that the upgrade process was not done correctly?

12:43:53.410 [DEBUG] c.n.h.c.p.n.r.NettyRequestSender - server certificate change is restrictedduring renegotiation javax.net.ssl.SSLHandshakeException: server certificate change is restrictedduring renegotiation at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.7.0_71] at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1619) ~[na:1.7.0_71] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:278) ~[na:1.7.0_71] at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:266) ~[na:1.7.0_71] at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1402) ~[na:1.7.0_71] at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:209) ~[na:1.7.0_71] at sun.security.ssl.Handshaker.processLoop(Handshaker.java:878) ~[na:1.7.0_71] at sun.security.ssl.Handshaker$1.run(Handshaker.java:818) ~[na:1.7.0_71] at sun.security.ssl.Handshaker$1.run(Handshaker.java:816) ~[na:1.7.0_71] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_71] at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1237) ~[na:1.7.0_71] at org.jboss.netty.handler.ssl.ImmediateExecutor.execute(ImmediateExecutor.java:31) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1453) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1326) ~[netty-3.9.4.Final.jar:na] ... 18 common frames omitted Wrapped by: javax.net.ssl.SSLHandshakeException: server certificate change is restrictedduring renegotiation at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1300) ~[na:1.7.0_71] at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:513) ~[na:1.7.0_71] at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:790) ~[na:1.7.0_71] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:758) ~[na:1.7.0_71] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[na:1.7.0_71] at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1285) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:917) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.9.4.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.9.4.Final.jar:na] at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.9.4.Final.jar:na] at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.9.4.Final.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_71] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_71] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]

slandelle · May 15, 2015, 5:01pm

This post suggests that your OPS person has just hardened security and finally got rid of SSLv3.
Modern versions of Gatling disable SSL (so you use TLS) and let you configure the cypher suite.

You should try a simple test with latest Gatling.
I guess it’s time to upgrade

jarrowwx · May 15, 2015, 5:31pm

It probably is time to upgrade. Sadly, I don’t have time to do it today.

However, we have code that DOESN’T exhibit this behavior, even on the same version of Gatling. Any thoughts how that could happen?

slandelle · May 15, 2015, 5:45pm

Do they all use same JDK and Gatling version?

jarrowwx · May 15, 2015, 5:49pm

Theoretically. But when we turned up the volume on the old, it started exhibiting it, too. Very odd.

We found a temporary workaround: http://stackoverflow.com/questions/27105004/what-means-javax-net-ssl-sslhandshakeexception-server-certificate-change-is-re

But yes, it’s time to upgrade.

slandelle · May 15, 2015, 5:56pm

Yep, that’s the same thread I previously sent
POODLE

jarrowwx · June 10, 2015, 2:29pm

Okay, I upgraded. I’m still seeing this behavior. I was able to work around it, the same way I did before. But I’d like to handle it the “right” way if I can. What should I do?

slandelle · June 10, 2015, 3:18pm

Disable SSLv3 and force something like TSLv1: https://github.com/gatling/gatling/blob/v2.1.6/gatling-core/src/main/resources/gatling-defaults.conf#L109

Possibly upgrade JDK: https://bugs.openjdk.java.net/browse/JDK-8072385

slandelle · June 18, 2015, 12:03pm

Any update on this?

Thanks,

jarrowwx · June 18, 2015, 6:27pm

Thanks for asking. I tried tweaking the config, and it didn’t work correctly. I didn’t want to bug you, so I just put the old workaround back in place. But if you’re in the mood, you could try hitting https://p-api.cigna.com/services/ and tell me what the right configuration parameters would be.

slandelle · June 18, 2015, 8:21pm

I have no issue hitting this url with Gatling, neither with JDK 1.7.0_79 nor 1.8.0_45.

Have you tried upgrading your JDK? From the stacktrace you provided, you were running 1.7.0_71 and you’re probably trying to use SSLv3 but your ops people disabled it (POODLE). SSLv3 is disabled by default since 1.7.0_75.

Cheers

jarrowwx · June 18, 2015, 9:54pm

Try this instead:

https://pvs-api.cigna.com/services/

It should work fine with p-api, and be broken with pvs-api. See if you see the same thing.

slandelle · June 18, 2015, 9:59pm

Same thing: all good.

23:57:28.833 [DEBUG] o.a.n.c.NettyConnectListener - Using non-cached Channel [id: 0xcefab0b7, /192.168.0.12:51450 => pvs-api.cigna.com/170.48.27.70:443] for GET ‘/services/’
23:57:29.085 [DEBUG] o.a.n.h.HttpProtocol -

Request DefaultHttpRequest(chunked: false)
GET /services/ HTTP/1.1
Accept-Charset: ISO-8859-1,utf-8;q=0.7,;q=0.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
Connection: keep-alive
Host: pvs-api.cigna.com

Response DefaultHttpResponse(chunked: false)
HTTP/1.1 200 OK
Server: Cigna
X-Frame-Options: DENY
Strict-Transport-Security: max-age=31536000
Content-Type: application/json
Content-Encoding: gzip
Content-Length: 341

jarrowwx · June 18, 2015, 10:05pm

Well, naturally, I wasn’t hitting that particular URL, but I got:

17:51:31.605 [DEBUG] o.j.n.h.s.SslHandler - Swallowing an exception raised while writing non-app data

java.nio.channels.ClosedChannelException: null

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:128) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:84) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.Channels.write(Channels.java:725) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.Channels.write(Channels.java:686) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:1110) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1252) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.handler.ssl.SslHandler.unwrapNonAppData(SslHandler.java:1166) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.handler.ssl.SslHandler.channelDisconnected(SslHandler.java:582) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:360) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:58) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.Channels.close(Channels.java:828) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1462) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1314) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty-3.10.3.Final.jar:na]

at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty-3.10.3.Final.jar:na]

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_45]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_45]

at java.lang.Thread.run(Unknown Source) [na:1.8.0_45]

17:51:31.605 [DEBUG] o.a.n.h.Processor - Channel Closed: [id: 0x0bc63897, /10.25.244.107:58226 :> pvs-api.cigna.com/170.48.27.70:443] with attribute null

17:51:31.611 [DEBUG] o.a.n.h.Processor - Unexpected I/O exception on channel [id: 0x0bc63897, /10.25.244.107:58226 :> pvs-api.cigna.com/170.48.27.70:443]

javax.net.ssl.SSLHandshakeException: server certificate change is restricted during renegotiation

at sun.security.ssl.Alerts.getSSLException(Unknown Source) ~[na:1.8.0_45]

at sun.security.ssl.SSLEngineImpl.fatal(Unknown Source) ~[na:1.8.0_45]

at sun.security.ssl.Handshaker.fatalSE(Unknown Source) ~[na:1.8.0_45]

at sun.security.ssl.ClientHandshaker.serverCertificate(Unknown Source) ~[na:1.8.0_45]

at sun.security.ssl.ClientHandshaker.processMessage(Unknown Source) ~[na:1.8.0_45]

at sun.security.ssl.Handshaker.processLoop(Unknown Source) ~[na:1.8.0_45]

at sun.security.ssl.Handshaker$1.run(Unknown Source) ~[na:1.8.0_45]

at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_45]

at sun.security.ssl.Handshaker$DelegatedTask.run(Unknown Source) ~[na:1.8.0_45]

at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1392) ~[netty-3.10.3.Final.jar:na]

at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1255) ~[netty-3.10.3.Final.jar:na]

… 18 common frames omitted

Wrapped by: javax.net.ssl.SSLHandshakeException: server certificate change is restricted during renegotiation

at sun.security.ssl.Handshaker.checkThrown(Unknown Source) ~[na:1.8.0_45]

at sun.security.ssl.SSLEngineImpl.checkTaskThrown(Unknown Source) ~[na:1.8.0_45]

at sun.security.ssl.SSLEngineImpl.readNetRecord(Unknown Source) ~[na:1.8.0_45]

at sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source) ~[na:1.8.0_45]

at javax.net.ssl.SSLEngine.unwrap(Unknown Source) ~[na:1.8.0_45]

at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218) ~[netty-3.10.3.Final.jar:na]

at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852) ~[netty-3.10.3.Final.jar:na]

at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425) ~[netty-3.10.3.Final.jar:na]

at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310) ~[netty-3.10.3.Final.jar:na]

at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) ~[netty-3.10.3.Final.jar:na]