Gatling Websocket Checks Slowing Down over Time

Gatling (Open-Source)
1

Hi,

I’ve encountered an issue when using gatling websocket (gatling version 3.5.1).
When there is a delay between the sent message and a correct response (several lengthy messages which are to be ignored), for some reason the checks on messages are slowing down over time. The messages are not being logged as received while the PC is kind of idling around.
When I’ve implemented the same interaction with jetty websocket client, the speed is as expected.

I am thinking that the gatling websocket implementation is storing a fair amount of information about the received messages, failed checks, etc. If it is so, is there an easy way to change this behaviour?

I am close to rewriting websocket protocol…

Any help is much appreciated.
Thank you.

Kind regards

2

Please provide a way to reproduce your issue.

3

sorry Stephane, I don’t have time to provide a reproducible example.

but I’ve managed to implement websocket “protocol” with Jetty and it solved my problem, the messages are being processed fluently.

4

Hi Gatling team,

I am facing a similar problem, where I see the memory usage linearly increasing due to the messages received on the web socket.
Is there any way we can control / clean the data stored to reduce the memory footprint during the simulation?
This issue was noticed only with Gatling 3.x, I did not face this problem in Gatling 2.x

Thank you.

5

Have you checked the content of the memory with a heap dump?
Have you tried building a SSCCE so we can investigate, as required in this group’s terms?

6

Thank you for your response.
I am using - Gatling 3.0.1, I had three questions here:

  1. Can you confirm if Gatling websocket stores some information (including the message payload) regarding messages received on the websocket?
  2. Is there a way to disable / reduce / clean up this information being stored for received messages?
  3. When gatling report shows that websockets crashed due to failure to allocate “direct memory” - which memory does this refer to and try to allocate?

Your response to these will be of great help, Thank you.

This is a simple example of what my script is doing:

  • Gatling (client) sets up websockets and keeps them open for the entire simulation
  • Gatling pushes messages into Kafka
  • Server reads messages from Kafka and pushes each message on one of the websockets it is meant for
  • We see that with Gatling 2.x this worked fine (constant memory usage on client), but with Gatling 3.x memory seems to be linearly rising based on the messages received on the websocket
  • We see that the linear increase also depends on the size of the payload used (My payload is about 1.5 kB)

______________________________________________ [ Code Sample ] ______________________________________________

private val httpProtocol: HttpProtocolBuilder = http
.baseUrl(baseURL)
.acceptHeader(“application/json;q=0.9,/;q=0.8”)
.doNotTrackHeader(“1”)
.acceptLanguageHeader(“en-US,en;q=0.5”)
.acceptEncodingHeader(“gzip, deflate”)
.userAgentHeader(“Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0”)

private val kafkaMessageScn = scenario(“publishKafkaMessage”)
.exec(feed(kafkaFeeder))
.forever(
exec { session =>
val trackingId = Random.alphanumeric.take(32).mkString
info(s"setting trackingID $trackingId for user : ${session.userId} ")
session.set(“trackingId”, trackingId)
}.exec(publishNotification(“kafka-publish-payload”, payload))
)

private val registerWSScn = scenario(“Notifications service load test”)
.exec(feed(userDatafeeder))
.exec(registerClientWithOptionalHeaders)
.exitHereIfFailed
.pause(1)
.exec(openWebSocket)
.pause(loadTotalTime)
.exec(closeWebSocket)

setUp(
registerWSScn
.inject(rampUsers(totalUsers) during usersRampUpTime.seconds)
.protocols(httpProtocol),
kafkaMessageScn
.inject(nothingFor((usersRampUpTime + 60).seconds), rampUsers(kafkaUsers) during usersRampUpTime.seconds)
.throttle(ThrottleSteps: _*)
.protocols(kafkaProtocol)
)

7

Gatling 3.0.1 that was released 3 years ago!
There’s been countless bug fixes since then (we’re about to release 3.7.0).
Please comply with this group’s terms and first upgrade to the latest version when you face an issue.

8

Okay sure, I will give this a shot with the new Gatling versions as well.
Meanwhile, would it still be possible to answer the 3 questions I asked - since abilyak has faced the issue even with 3.5.1 and I just want at least a general clarity on those 3 questions.
I will post my findings with the newer versions as well shortly.

Thank you.

9

No, Gatling WebSocket support doesn’t buffer any payload.
Direct memory allocation failure means you’re having an IO issue:

  • either you’re writing too fast (for your network or your counterpart)
  • or you’re reading too slowly (not enough CPU to process all the inbound payloads)
  • or you’re trying to send humongous payloads and can’t find a large enough contiguous memory space

Again, there’s no way to help if you don’t provide a step by step procedure (including a sample server) to reproduce your problem.

10

Thank you Stephane.
Just to confirm, when we make statements like:

  • either you’re writing too fast (for your network or your counterpart)
  • or you’re reading too slowly (not enough CPU to process all the inbound payloads)
    Doesn’t this mean that there is a buffer for received messages which is getting filled up if not read fast enough?
    Thank you for your time and clarifications.
11

The Linux IO stack.

12

Ok I Will do it