The Simulation Setup page in the Gatling Documentation contains a brief
explanation of closed and open models. That is a subtle but important point,
and my conversations with other Gatling users point out that many do not
understand the difference, and are unaware that the closed models' request
queuing/sequencing differs significantly from the typical access patterns
for web application/service. Tip off the hat to the Gatling team for
contrasting the two!
I'm reaching out to see whether anybody could provide pointers to a more
detailed treatment of these models and their differences.
Here's some book references:
The Art of computer systems performance analysis, Jain, 1992 - chapter
32: Queuing Networks. "An open queuing network has external arrivals
and departures .... "
Analysing Computer system performance with perl::pdq, Gunther, 2005 -
2.6 Unlimited Request (Open) Queues. "Fig 2.2 assumes that the number
of customers arriving from outside the checkout queue is not limited.
In other words there is an infinite pool of customers outside the
grocery store. Some fraction of this pool is arriving into the
checkout queue at rate lambda..."
https://github.com/gatling/gatling/issues/2077 (originally called
"gatling fails CO test compared with wrk ")
How do you assess
the fit of one model over the other based on access patterns? Queuing theory
comes to mind... Additional references appreciated.
possibly see "7 Choosing a system model"
Is your system under test more like a grocery store or a call centre?
Ie. is the arrival of one request or user dependent on some other
request/user completing work in progress?
If there is a dependency there, then (unless the requests fan out to
slaves) there is likely some kind of closed feedback loop going on.
It may be that some of the requests are in a feedback loop as a user
proceeds down some ordered navigation flow but the users themselves
arrive independently, in which case the model is partly-open. Which
end of the open vs. closed spectrum is it? ie. is the session length
short or long in terms of number of requests.
Even if it's like a grocery store, are you testing the whole system or
just an internal part of it?
If just part of it then likely it is behind some kind of system
component that has a concurrency limit (ie. max connections on a web
server or load balancer)...
Let's assume for sake of argument that you chose the right workload
model, implement the test script perfectly, run the test flawlessly.
The remaining challenge is that the results you then obtain may still
suffer from co-ordinated omission (as this may be found in any system
including production). I don't know if Gatling or any other tool has
(the option of) correction built in to the results as they are
Hopefully this gives you some pointers.