I’ve set up gatling with sbt and am now trying to run two simulations sequentially. However, no matter what I pass to ‘testOnly’, the scenarios are run in the wrong order:
testOnly *sim1 *sim2
runs sim2 before sim1. And the same goes for
testOnly *sim2 *sim1
Any ideas?
I’m afraid that’s how sbt works and testOnly only applies a filter on elected classes and can’t do anything about the order (and other test plugins can run in parallel, so order doesn’t make sense).
Try playing with the fully qualified class names (package + class).
That’s a good point. It is not a good practice to have separate tests dependant on each other. They should run as independent units and the order should not matter.
Simply run those simulations as separated commands:
;testOnly sim1;testOnly sim2
What you’re doing simply applies a filter to run only specific simulations, and no specific order is guaranteed.
Cheers,
Pierre
The current dependency is because I’m also using gatling to set up initial data (users, permissions, etc) before running the stress tests, and to clean up data after they are run. Why do you think this is not good practice?
If you create and then delete test data on every test run through the system under test, you have to be really sure about what you’re doing.
You might be setting your system in a specific state that might be very different from production, like all your data being in caches because you’ve just injected it, database statistics being completely off because their where not rebuilt, etc.
I personally prefer dedicated restore scripts (that could be exposed through a web API), launch them in before and after blocks in the Simulation (a single HTTP call is very easy to do, either with AsyncHttpClient or UrlConnection) and keep the test standalone.
Thanks Pierre, that works, but only from within sbt. I don’t see any way to replicate this behavior from outside of sbt (like from a windows batch file). Fore example, calling “sbt testOnly simulation1” behaves the same as “sbt test”, i.e., it runs all tests without any filter. In the end I’m looking for a way to automate my tests or at least allow them to be run with a click.
I’m not sure I see how that approach is different than having it written in gatling, and since gatling makes it easy to run lots of concurrent request, it seems like a more time-efficient approach to me. Also, since the data is being set up specifically for stress tests in my case, it seems that it should be created in the same environment and flow (i.e., within gatling as part of the flow of scenarios) .
Either way, I agree you need to know what you’re doing when you set up data. But it’s an important part of attempting to simulate a production situation. Not setting up initial data could skew results even more.
This can of course also works from outside SBT :
sbt “testOnly mySimulation1” “testOnly mySimulation2”
Cheers,
Pierre
The current dependency is because I’m also using gatling to set up initial data (users, permissions, etc) before running the stress tests, and to clean up data after they are run. Why do you think this is not good practice?
The setup and tear-down should be in the before and after hooks (as you would do with XUnit or a “BDD tool”.
In the end I’m looking for a way to automate my tests or at least allow them to be run with a click
Use the Gatling Jenkins plugin:
docker run -d -p 8080:8080 --name gatling-jenkins aidylewis/gatling-jenkins
Thanks, that works!
But then they would run before and after every simulation, which is not what I’m looking for. I’m trying to separate every scenario that has its own SLA into a separate simulation and run all the simulations sequentially. There is some configuration that needs to be done once before all of these tests are run, but only once. And when all the simulations are complete I want to clean up.
We probably will not be using Jenkins in this case. It’s intended for running manually against a performance environment, not as part of the CI.