Closed and open models

Hello –

The Simulation Setup page in the Gatling Documentation contains a brief explanation of closed and open models. That is a subtle but important point, and my conversations with other Gatling users point out that many do not understand the difference, and are unaware that the closed models’ request queuing/sequencing differs significantly from the typical access patterns for web application/service. Tip off the hat to the Gatling team for contrasting the two!

I’m reaching out to see whether anybody could provide pointers to a more detailed treatment of these models and their differences. How do you assess the fit of one model over the other based on access patterns? Queuing theory comes to mind… Additional references appreciated.

Thanks,

-Dragos

Hi,

Hello --

The Simulation Setup page in the Gatling Documentation contains a brief
explanation of closed and open models. That is a subtle but important point,
and my conversations with other Gatling users point out that many do not
understand the difference, and are unaware that the closed models' request
queuing/sequencing differs significantly from the typical access patterns
for web application/service. Tip off the hat to the Gatling team for
contrasting the two!

I'm reaching out to see whether anybody could provide pointers to a more
detailed treatment of these models and their differences.

Here's some book references:

The Art of computer systems performance analysis, Jain, 1992 - chapter
32: Queuing Networks. "An open queuing network has external arrivals
and departures .... "

Analysing Computer system performance with perl::pdq, Gunther, 2005 -
2.6 Unlimited Request (Open) Queues. "Fig 2.2 assumes that the number
of customers arriving from outside the checkout queue is not limited.
In other words there is an infinite pool of customers outside the
grocery store. Some fraction of this pool is arriving into the
checkout queue at rate lambda..."

Other refs:
https://www.youtube.com/watch?v=99RABfKNfcY /
http://twitter.github.io/iago/philosophy.html
http://users.cms.caltech.edu/~adamw/papers/openvsclosed.pdf
10. Frequently Asked Questions — Tsung 1.7.0 documentation
Redirecting to Google Groups
Redirecting to Google Groups
57258 – address coordinated omission latency reporting risks
blog - devmio - Software Know-How
HTTP: response time measurement omits time to connect and send request · Issue #2077 · gatling/gatling · GitHub (originally called
"gatling fails CO test compared with wrk ")
The Pith of Performance: A Little Triplet
The Pith of Performance: Using Think Times to Determine Arrival Rates

How do you assess
the fit of one model over the other based on access patterns? Queuing theory
comes to mind... Additional references appreciated.

possibly see "7 Choosing a system model"

Is your system under test more like a grocery store or a call centre?
Ie. is the arrival of one request or user dependent on some other
request/user completing work in progress?
If there is a dependency there, then (unless the requests fan out to
slaves) there is likely some kind of closed feedback loop going on.
It may be that some of the requests are in a feedback loop as a user
proceeds down some ordered navigation flow but the users themselves
arrive independently, in which case the model is partly-open. Which
end of the open vs. closed spectrum is it? ie. is the session length
short or long in terms of number of requests.

Even if it's like a grocery store, are you testing the whole system or
just an internal part of it?
If just part of it then likely it is behind some kind of system
component that has a concurrency limit (ie. max connections on a web
server or load balancer)...

Let's assume for sake of argument that you chose the right workload
model, implement the test script perfectly, run the test flawlessly.
The remaining challenge is that the results you then obtain may still
suffer from co-ordinated omission (as this may be found in any system
including production). I don't know if Gatling or any other tool has
(the option of) correction built in to the results as they are
presented.

Hopefully this gives you some pointers.
Thanks,
Alex

Hi Alex,

Many thanks for your valuable share.

As long as I give you credit, would you mind if I take some of this information and include in an open-source book I am writing on performance testing?

https://github.com/bbc/notes-on-perf-testing

Aidy

Thank you Alex, that’s the level of detail I was looking for.

-Dragos

Hi Adrian,
Sure, hopefully it's been of use.
Thanks,
Alex

Hi Dragos,
Thanks, I have been out of the office for a few days where I had a
couple more books on the desk. So last one then:

Foundations of software and systems performance Engineering, Bondi, 2015 -
Ch.3 open/closed queues
Ch.4 workloads
Sections 9.3, 9.6.4 talk about notional load drivers with reference to
the queue network type and the workload type.
It describes open/closed models in good detail. Although it references
the open vs. closed paper it doesn't mention partly open models. It
doesn't mention asynchrony at the user vs. request level.

Will be interested to hear your thoughts if you get a chance.
Thanks
Alex