load testing

First off, the good news: if you are load testing based on a defined Workload Model, you are already waaaay ahead of a lot of performance testers. I am frequently horrified by testers whose approach is…well, just not very scientific. When their test case is “we’ll just throw some load at it and see what happens” or their load test has been defined in a way that is vague and imprecise, like “our test is to run load with x concurrent users”, then the probability of them not knowing what they are doing approaches 100%, and it is clear that [...]

Image macros – those pictures superimposed with vaguely humorus text – are everywhere on the Internet. I decided that it was time for someone to create some that performance testers could relate to; something we could all print out and pin to our cubicle walls. To get everything started, I have created 8 images. Maybe they will become successful memes and help spread performance testing culture across the world…or maybe they will just generate a chuckle or two. If you create your own performance testing-related image macro, please leave a comment (with a link), or send me an email. I [...]

On August 23rd, 21 JDS performance testers got together for their quarterly breakout session to share tips and tricks with each other. One of the breakout topics was Useful Tools for Performance Testers. This was about useful tools that were not load generation tools (or everyone would have just said “LoadRunner” and it would have been a very short session). I have included everyone’s contributions in this post… If you are a performance tester and have a different tool that you find useful in your job, please add it in the comments below. […]

As a performance testing consultant, I get to see a lot of the work that other performance testers do…and a lot of the time it horrifies me. If performance testing was a licensed profession, like law or medicine, then it would be necessary to revoke the licenses of 90% of testers. Their work is not just “low quality”, it is actually wrong or misleading, or based on such a shaky foundation that any predictions made from the test results are tenuous at best. Bad performance testers frequently treat their testing as a ritual, rather than as a science and they [...]

It seems unnecessarily obvious to even bother making the statement that “errors are bad”; surely this is an idea that everyone agrees with, like “crime is bad” or “you shouldn’t put your underwear on backwards”. But a lot of performance testers that I have worked with don’t seem to care too much about windows errors. A senior performance tester with more than 15 years experience at some of the largest companies in Australia recently said to me:   “Yeah, as long as the error rate is below 5%, I don’t bother putting it in my report.” Are. You. Kidding? How [...]

I've spent the last year or so writing notes and fleshing out chapters for a book called Performance Testing Web Applications, so imagine my very slight feeling of annoyance when I did a google search for my book title and found that someone had released a very similar book 3 months earlier... J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, and Dennis Rea have collaborated on a book called Performance Testing Guidance for Web Applications, which is available either as a free download, or in dead-tree format through Amazon. The book is really good, so I highly recommend that you [...]

I have made a list of the top ten things load testers frequently fail to do that make me feel like smiting them. Thou shalt know how thy test tool works. The worst performance testers I have met were always more concerned about whether they could get their scripts to run, rather than whether the tests they were running were realistic. Read the documentation, practice, spend some time figuring out what all the settings do, then relate how your scripts are running back to how real users exercise your application. Thou shalt gather realistic usage data. Garbage in, garbage out. [...]

I just found something interesting on Google Video; it is a 1 hour Google TechTalk presentation by Goranka Bjedov about Using Open Source Tools for Performance Testing. Goranka has some interesting things to say. She makes the point that there is really no standard terminology in performance testing circles, and goes on to prove this by giving her own definitions of performance, stress, load, scalability, and reliability testing. As an example of reliability testing she notes that "typically, when I was at AT&T, we would run for about a month at a time after everything was done just to find [...]

If the application that you are performance testing will be operating over a WAN, it is really important to test whether the reduced bandwidth and higher latency of the network link will increase transaction response times to an unacceptable level. If you are being thorough, you should also run some tests to determine whether your response times will be impacted by other network traffic on the WAN (and vice versa). Let the following anecdotes about testing two different applications serve as examples: The first application was to be used by staff who would be interacting directly with customers. There would [...]

It's always nice to base your load test on real-world usage patterns rather than someone's "best guess". If the system is used internally, Business Analysts are usually quite good at providing this information, but if the system is used by external users (like most high-traffic web applications), then they can typically only tell you "big picture" information; e.g. they know how many orders are created, but not how many searches are made per hour. Basic web log analysis tools provide low-level technical detail that isn't very useful - hits per hour. To be useful for load testing, we need to [...]

I just received an evaluation copy of the new version of Rational Performance Tester. IBM Rational have re-written the tool from scratch and based it on the open-source Eclipse framework. The tool has only just come out of beta testing and currently supports web protocols only. Everyone in the industry seems to be keen to know how it stacks up against the competition. First off, my impressions of the old version... Coming from a LoadRunner background, my opinions on the tool were a little mixed. Performance Tester was easy to learn and easy to use but it is definitely missing [...]

I am yet to work on a project where environmental issues haven't impacted my schedule to some degree. But it seems that I only start to document the outages once they start to seriously impact my schedule. This stops now. From now on I will document every outage (no matter how trivial) before they start to have any impact. Once the cumulative downtime reaches more than a few hours, I will include a link to the outage document as part of my weekly (or daily) status report. Here is a template for my Environment Availability Spreadsheet.

I have decided to release an early draft of this document so that others may provide feedback. Please let me know what you think. Writing test cases for performance testing requires a different mindset to writing functional test cases. Fortunately it is not a difficult mental leap. This article should give you enough information to get you up and running. First, lets set out some background and define some terms that are used in performance testing. Test case - a test case is the same as a use case or business process. Just as with a functional test case, it [...]

I haven't written a huge number of Detailed Test Plans for performance testing, so I am still interested in reading other people's (non-confidential) documents or templates. If anyone out there has a template, I would love to have a look.