Sometimes scenario not terminating

I am doing bulk data loading through Gatling. I need to step through 5.5 million records in a .csv file. WAY too many to load in at once, so I broke the data into 1000 line .csv files, and use a symlink to point to the “current” chunk of the data. I have a scenario (see below) that plows through the list and makes the necessary requests. When the feeder runs out, everything shuts down, and it’s all good. USUALLY.

In a wrapper script, I consult a configuration file (so I can turn on and off the data load based on time of day), and if the time of day says no-data-load, then it just sleeps. Otherwise, it points the symlink to the next file, and restarts the scenario.

After a couple of weeks of working just fine, something weird happened. Over the weekend, Gatling sat there doing nothing for over 200,000 seconds. So Monday I added a maxDuration of 2,000 seconds. Last night, around midnight, it locked up again and in spite of the maxDuration, it sat there and spun until around 8 am. So far this morning, it has done it at least 3 more times.

NORMALLY, when it exits “cleanly” there is a stack trace right after the line that says “13:12:59.163 [ERROR] i.g.c.a.SingletonFeed - Feeder is now empty, stopping engine”

When it doesn’t exit cleanly, there are log entries about dead letters, and then it sits there indefinitely.

Any ideas about what could be causing this? Or how to fix it? (I am on 2.0.1 at the moment, I will update to 2.2 when it goes gold)

Here’s what the simulation looks like:

`
val findUser =
scenario(“Find User”)
.exec( SessionVariables.initialize )
.exec( RTDE.Login.useExistingTrustedClientToken )
.exitHereIfFailed
.forever {
feed( csv(“add-to-pvs.csv”) )
.exec( _.set( INDIVIDUAL_ID, “NOT-FOUND” ) )
.exec( search( “CED” ) )
.exec( search( “HCMS” ) )
.exec( search( “PYSL” ) )
.exec( search( “FML” ) )
.exec( search( “GWFC” ) )
.exec( search( “CBHR” ) )
.exec( search( “GADB” ) )
.exec( search( “CBH” ) )
.exec( search( “DMND” ) )
.exec( search( “BEN” ) )
.exec( search( “ACLM” ) )
.exec( search( “IFP” ) )
.exec( search( “CPTK” ) )
.exec( search( “QCRE” ) )
.exec( search( “WDEN” ) )
.exec( search( “CWEB” ) )
.exec( search( “CIEB” ) )
.exec( search( “RA” ) )
.exec( search( “PCLG” ) )
.exec( search( “MRR” ) )
.exec( search( “CPI” ) )
}

setUp(
TrustedLogin.flow.inject( atOnceUsers(1) ),
findUser.inject(
nothingFor( 30 seconds ),
rampUsers( Test.numUsers ) over ( Test.rampTime )
)
)
.maxDuration( Test.duration )
`

Still suffering with this problem. I’m leaning towards Gatling doing the “wrong thing” when the feeder is empty. To see if that could be the source, I have a question:

What EXACTLY happens when virtual user 7 of 9 reaches a .feed() node, and the feeder says “I got nothin’”? Does it terminate THAT virtual user, or does it terminate the whole scenario?

I assert that it SHOULD only terminate that virtual user. Let virtual user 7 terminate now, let the rest of them terminate when they need data and can’t get it. But it does not look like that is what is happening, at least not in 2.0.1.

Thoughts?

No, the opposite happens: simulation is terminated. Not being able to feed is an unexpected behavior, and most likely a bug. Would you really want to have a 48hr simulation keep on running while it’s not able to feed logins and passwords?

There are different behaviors that make sense in different circumstances.

USE CASE: Single flow scenario - The user logs in, flows through the application, and then terminates. The simulation is driven by constantUsersPerSec()

In this case, it makes sense to allow the user to finish its flow and let it terminate itself.

USE CASE: Looping flow - In a loop, the user gets the data, and uses it to log in, do its flow, and then loop back. The simulation has a certain number of users.

In this case, it makes sense to terminate when the virtual user gets back to the .feed()

USE CASE: Do something forever - read from the feeder once, then in a loop do some actions using that user.

In this case, what if the tester tries to inject 100 users, but there are only 98 records in the feeder? Then it makes sense to terminate the simulation, in a way. At the same time, using .exitHereIfFailed would be a perfectly viable alternative. The number of active users would only be 98, but at least you would get a test run with 98 users, instead of a test run that dies because you don’t have enough data.

IN MY OPINION - and it is only my opinion, of course - I should have the OPTION of continuing the simulation until it naturally terminates.

One way to get what I’m thinking would be to have another feeder behavior. In addition to .random and .circular, there could be a .exitWhenEmpty. So, for example,

.feed( csv( some-file ).exitWhenEmpty )

Then when .feed() tries to read from the feeder, and the feeder is empty, then the virtual user exits rather than terminating the whole simulation.

Thoughts?

Addendum - see below …

There are different behaviors that make sense in different circumstances.

USE CASE: Single flow scenario - The user logs in, flows through the application, and then terminates. The simulation is driven by constantUsersPerSec()

In this case, it makes sense to allow the user to finish its flow and let it terminate itself

BUT THE user injection should stop, obviously…

Doable, but requires quite some work, and I don’t feel like making it a priority without more community demand.

Cheers,

I can work around it. But if you ever find yourself in that code… :slight_smile:

Hi John,

what is your workaround please?

I would like to do the same configuration in gatling.

Run tests until there is data in the feeder. If feeder is empty stop the tests, but do not fail them.

Regards,
Andras

+1 for the feature request of gracefully ending the test when the feeder doesn’t have more items.

+1 to making the behaviour configurable.

Taking the example of a big load test with a large number of scenarios - if a single scenario runs out of data then I would generally prefer to be able to have that throw errors/warnings and allow the wider simulation to continue. Any results are better than no results!

Barry

Why not use the circular method?

csv("urls.txt").circular

Aidy

Hi Aidy,

That would work for some people, but bearing in mind that some data can only be used once, configurable behaviour when all data has been used would make Gatling a lot more flexible (doesn’t impact me too much as I tend to use Redis for any burnable data).

Barry

Hey,

First, I think we can all agree that hitting a feeder starvation is a test design flaw. Even if we were to keep on running and make users who hit a starved feeder exit, you’d get less load than we you expected.

Then, as a user, how would you realize that you hit a feeder starvation? Check the logs? Most users would fail to do that and realize it, and we would end up with tons of invalid issues.

Yes, if we are using Gatling exclusively as a load testing tool, then it might be said to be a test design flaw.

But you made such an awesome tool that not all of us use it exclusively for generating load. It is perfectly suited to parallel process automation, too, e.g. data loading. Which is why this behavior is annoying: it makes it harder to use Gatling for something it wasn’t designed for. :slight_smile: