My tests have been working fine so far, but lately I have been getting errors when running 10 constantsUserPerSec and more and I have 10 testdata users.
When running 9 and less I get no errors.
This “smells like” the web services I am testing dos not allow requesting with the same username in paralell.
Request:
getProfile: KO jsonPath($.resultCode).find(0).is(SUCCESS), but actually found PERMANENT_ERROR
This is a server issue. You have to talk to your developer to understand how many concurrent connections web service can handle of same user.
Ok, thanks!
I guess from a functional perspective it is not something we want (that one user can log on and have several connections to the server at the same time in production).
But for performance testing this should be ‘allowed’.
Do you happen to know what this ‘setting’ is called on a web server? As in pointing the developers in the right direction.
That’s not the right way to fix it. You invalidate your testing when you tweak the application to support the test.
The proper thing to do is ensure that you have more users than you can possibly use at a time. In fact, I would go a step beyond that. If you might have 30k users in an hour, make sure you are testing with 30k users! Because using the same users over and over again does not put the same burden on the database that would be incurred by randomly picking from a large pool of users. Make sense?
It’s extra work, but it makes for a more realistic test.
John, this absolutely makes sense!
Actually I have ‘ordered’ 2 million unique users in terms of testdata from the DBA to assure ‘real-world test’.
The scenario I described earlier (in this post) is not realistic.
In additon to that, I have a challenge in that for every user in my scenarios the testdata is getting used-up/eaten-up (the user activates all available coupons he gets), so if I run the scenario again with the same user he would just hit a cache and I won’t get the call to the backend system as is the case for the real-life scenario/usage.
The scope of the perf test is to measure how long it takes to ‘activate a coupon’ / going back to the legacy sys.
Thanks for the clarification on what a test really should do (test the actual scenario).
M