No one is ever going to argue that Mercury don’t know how to throw a party. Mercury World Australia was a great opportunity to hear experts talk about the direction our industry is headed and ways that business can increase the value they get from technology but, more importantly, it was also an opportunity to network (or schmooze if you prefer) and enjoy their excellent catering.
While I definitely wasn’t going to miss the Gala dinner on Thursday, due to client commitments I only ended up seeing one presenter – Paul McLean from RPM Solutions and his talk on “Analysing System Behaviour During a Load Test”.
His main point was that we, as Performance Testers, should use the scientific method as a framework to help diagnose the root cause of a performance problem.
The scientific method is designed so that the results of a test are not biased by the results you are expecting, or the results that you want to see. Physicist and academic Richard Feynman summarised the scientific method as 1. Make a guess; and 2. See if you’re wrong, but it is more traditionally defined as an iterative process comprised of the following four steps…
- Observation – accurately characterising the behaviours that were witnessed, making sure to differentiate carefully between cause and effect (correlation does not equal causation).
- Hypothesis – when you formulate a hypothesis, you determine a possible reason for the behaviour your are witnessing.
- Predict Behaviour – use the hypothesis you formulated in the previous step to predict the behaviour of the system. Paul used the example of a system problem that he believed was related to user concurrency. To determine if this was likely, he repeated the test with the same number of users, but half the transaction rate. If the number of concurrent users was the key factor, he would expect the problem to still manifest.
- Test Prediction – devising an experiment to determine if your predictions are correct, and therefore if your underlying hypothesis is valid is demonstrated in the previous example. The results of Paul’s test would tell him whether his predictions were correct or not, and whether he should formulate an alternative hypothesis.
It’s always nice to hear a presentation where, even if you do not necessarily learn anything new, you agree with everything the presenter is saying. Paul even used a project we were both involved in as an illustrative example of applying the scientific method to isolate the root cause of a real-world problem.
Even if all the talks aren’t this good, I will be making every effort to attend the next Mercury World in Australia.
Related Posts
3 Comments
Comments are closed.
The MSDN page on Testing for Performance has the following to say about using the scientific method when performance testing…
You can often attribute performance problems to more than one factor. So, finding a solution for poor performance is quite similar to conducting a scientific experiment. Scientific experimentation traditionally follows a six-step process that involves observation, preliminary hypothesis, prediction, tests, controls, and a theory. The theory consists of a hypothesis supported by the best collection of evidence accumulated by the process. You can solve performance problems by following the same process.
I think this is what good testers do also (ie. apply scientific method), though it is perhaps most applicable to exploratory testing. James Bach, Cem Kaner and the other context-driven-testing guys have much to say on this. Every test is a little experiment…
Even if all the talks aren’t this good, I will be making every effort to attend the next Mercury World in Australia.
It looks like I will be presenting at (rather than just attending) Mercury World Australia 2006.