TPS drop after heap dump on gatling target

Gatling (Open-Source)
1

Hi,

I have simulation with several scenarios. Each scenario has its own users and throttling.

After 2 hours of test heap was dumped on server (gatling target/base url). In that moment (several seconds) TPS decreased considerably and never got back to previous, correct level. This TPS drop is clearly shown in HTML report. Report also shows expected (set in scenario/simulation) number of active users.

Does gatling use any kind of TPS adaptation when server slows down? If so, TPS level should go back to initial level after server recovers, right?
Can it be a bug?

I’m using gatling 2.1.7 with maven plugin.

Thanks in advance for any help
Michal

2

Hi There,

HeapDumpOnOutOfMemory is enabled by default and I believe that’s how you are seeing heap being dumped in the file. Please user Eclipse MAT to parse the heap dump and share the top objects. Maybe Stephane or senior folks will be able to help.

As a workaround for now, try doubling your Xmx in gatling.sh, if your server has enough memory available and see if it helps.

-Mahesh

3

I see I wasn’t clear enough :slight_smile:

Heap dump is made on server process - target application of gatling simulation. Gatling isn’t interrupted or changed in any way during the test.

Heap dump takes 17 seconds. In that time server is paused. But there are no errors on gatling side, probably because of longer timeouts set in gatling confing.

The test looks like this:

  1. Server and gatling are running normally, there is constant TPS set to let say 200.
  2. Heap dump is taken on server (not gatling).
  3. I can see that server pauses for 17 seconds.
  4. Server resumes its work, but:
  • TPS drops to 120 TPS
  • Number of opened sockets on server machine drops
  • It seems that gatling sends less requests - it’s also visible on the HTML report generated by gatling
    There are no errors in gatling log, response times of transactions don’t change (except heap dump period).

What is more, simulation contains a few scenarios - each has its own number of users and throttling rules.
After that heap dump TPS decreases in general, but:

  • For some scenarios it decreases (there is 100 TPS set in throttling and after the dump I can see 60 in the report and log)
  • For some scenarios it increases (there is 30 TPS set in throttling and after the dump I can see 40 in the report and log)
  • For some scenarios it doesn’t change

Maybe gatling gets confused because of this server delay and changes TPS in some way? Is there anything like this implemented?
Maybe there’s a bug in gatling and this delay messes the throttling (or gatling users) up?
Maybe some threads/users stop working?

I know it’s pretty weird situation and it’s hard to say anything for sure but I would be thankful for any clue.

Best regards,
Michal