SSLEngine closed already

Hi,

When running my tests (a bunch of users hammering a server without pause), after a while (usually about 150 seconds), I get this error “SSLEngine closed already”. So

  • None before a precise moment
  • A bunch at a given moment
  • Then some every 5 seconds
    So far, I thought it was my server that was having a problem but I’m not so sure anymore.

If I look at the Netty code (https://searchcode.com/codesearch/view/10525295/), it looks more like the server that decides to close the connexion and then the client that should just reopen one. I’m guessing a bit here.

So it shouldn’t be a real KO. It’s some kind of normal behavior.

Strangely, when using Apache Benchmark, I also have some similar errors on stdout but then Apache Benchmark doesn’t consider it to be a failure in the final report.

Am I making any sense? Can someone explain the error?

Thanks,
Henri

Salut Henri,

Do you some kind of reproducer, please?

Thanks,

The script is quite basic

class MySimulation extends Simulation {

  val users = Integer.getInteger("users",  1).toInt
  val ramp = Integer.getInteger("ramp", 1).toInt
  val duration = Integer.getInteger("duration", 10).toLong
  val baseUrl = System.getProperty("baseUrl")

  val httpProtocol = http
    .baseURL(baseUrl)
    .doNotTrackHeader("1")
    .disableCaching
    .userAgentHeader("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.49 Safari/537.36")
    .acceptHeader("text/html,application/xhtml+xml,application/xml;")
    .acceptEncodingHeader("gzip, deflate, sdch, br")

  val scn = scenario("test")
    .during(duration seconds) {
      exec(http("fake")
        .get("/img.png"))
    }

  setUp(scn.inject(heavisideUsers(users) over(ramp seconds))).protocols(httpProtocol)
}

It hits an https url (load balancer in front handling the SSL termination and then balancing on two servers, with a new https connection).

I will try to reproduce locally.

But in fact, I’m pretty sure I won’t be able to reproduce out of my environment. If it can be of any use, the LB is managing the keep-alive correctly, does SSL session caching but do not support SSL tickets.

So question is: “How is the Netty piece of code below handled by Gatling?”

Right now, I fell that if the connection is closed, the client should recreate it without complaining. Because it’s normal to get a closed connection after a while (I kinda think/guess). WDYT?

                SSLEngineResult result = wrap(engine, buf, out);

                if (!buf.isReadable()) {
                    buf.release();
                    promise = (ChannelPromise) pending.recycleAndGet();
                    pendingUnencryptedWrites.remove();
                } else {
                    promise = null;
                }

                if (result.getStatus() == Status.CLOSED) {
                    // SSLEngine has been closed already.
                    // Any further write attempts should be denied.
                    for (;;) {
                        PendingWrite w = pendingUnencryptedWrites.poll();
                        if (w == null) {
                            break;
                        }
                        w.failAndRecycle(SSLENGINE_CLOSED);
                    }
                    return;
                }

There’s such a mechanism implemented for retrying such failure, but it might not work in this case.
I’d need to be able to reproduce, and will only be able to investigate when I’m back to the office in 2 weeks.

Also, if you have some exceptions in the logs, it would help.

Sure. I will send you everything I have

Please do :slight_smile:

So the error is: j.n.s.SSLException: SSLEngine closed already

It is logged as a warning like this:

11:39:10.971 [DEBUG] i.g.h.a.AsyncHandler - Request ‘my-webfont.ttf’ failed for user 86
javax.net.ssl.SSLException: SSLEngine closed already
11:39:10.973 [WARN ] i.g.h.a.ResponseProcessor - Request ‘my-webfont.ttf’ failed: j.n.s.SSLException: SSLEngine closed already
11:39:10.978 [DEBUG] i.g.h.a.ResponseProcessor -

Request:
my-webfont.ttf: KO j.n.s.SSLException: SSLEngine closed already

Session:
… (the session)

HTTP request:
GET https://site.com/my-webfont.ttf
headers=
Accept: /
DNT: 1
Accept-Language: en-CA,en;q=0.8,fr;q=0.6
Accept-Encoding: gzip, deflate, sdch, br
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.49 Safari/537.36
Referer: https://site.com/thing
Cookie: …
Host: site.com

HTTP response:

No stack trace when the log is enabled for errors

Henri, I’m seeing the same issue. Did you figure this out?

Sadly no. I never managed to get a fully reproductible use-case after the fact (not my fault, I wasn’t there anymore). The root cause seemed to be the IDS that was closing connection but they should then have been recreated by gatling.

Hi
I have the same problem, have you found a
olution
Thanks

No. I haven’t. I tune the proxy that was causing issues and it disappeared. But that obviously isn’t the solution from gatling side.

In my case, a load balancer was misbehaving and closing my ssl connections

Thanks

Hi Henri

I am also getting the same error and raised it in gatling forum but no reply so thought to check with you as well if you managed to resolve that or now. if yes then how? it looks like issue from Gatling side?

No. As I said, something was closing the connection. But it was allowed to. We reconfigured the appliance. But Gatling wasn’t recreating the connection for some reason.

Again, here isn’t much we can do without being able to reproduce.
The only time we saw this error, the frequency was 1/1,000,000 in customer environment so not something we could debug.

Yes Stephane, Exactly same thing in happening in my environment. out of 34000 transaction we got 1 SSLException error but because script is running continuous mode so other transactions are failing.

is there anyway in gatling to stop the iteration and start new iteration if any failure occurs ?

If I recall correctly a way to reproduce would be to establish an SSL connection. Then to have the server cutting the connection because of an SSL connection timeout. Check if the client reconnects every time.

Hi All, I am also facing the same problem . Anyone have solution for this issue.