How to write a performance test case

I have decided to release an early draft of this document so that others may provide feedback. Please let me know what you think.

Writing test cases for performance testing requires a different mindset to writing functional test cases. Fortunately it is not a difficult mental leap. This article should give you enough information to get you up and running.

First, lets set out some background and define some terms that are used in performance testing.

• Test case – a test case is the same as a use case or business process. Just as with a functional test case, it outlines test steps that should be performed, and the expected result for each step.
• Test script – a test script is a program created by a Performance tester that will perform all the steps in the test case.
• Virtual user – a virtual user generally runs a single test script. Virtual users do not run test scripts using the Graphical User Interface (like a functional test case that has been automated with tools like WinRunner, QuickTest, QARun or Rational Robot); they simulate a real user by sending the same network traffic as a real user would. A single workstation can run multiple virtual users.
• Scenario – a performance test scenario is a description of how a set of test scripts will be run. It outlines how many times an hour they will be run, how many users will run each test script, and when each test script will be run. The aim of a scenario is to simulate real world usage of a system.

Writing a test case for performance testing is basically writing a simple Requirements Specification for a piece of software (the test script). Just as with any specification, it should be unambiguous and as complete as possible.

Every test case will contain the steps to be performed with the application and the expected result for each step. As a performance tester will generally not know the business processes that they will be automating, a test case should provide more detail than may be included in a functional test case intended for a tester familiar with the application.

It is important that the test case describes a single path through the application. Adding conditional branches to handle varying application responses, such as error messages, will greatly increase script development time and the time taken to verify that the test script functions as expected. If a test script encounters an error that it does not expect, it will usually just stop. If the Project Manager decides that test scripts should handle errors the same way a real user would, then information should be included on how to reproduce each error condition, and additional scripting time should be included in the project plan.

The main reason a user may be presented with a different flow through the application is the input data that is used. Each test case will be executed with a large amount of input data. Defining data requirements is a critical part of planning for a performance test, and is the most common area to get wrong on a first attempt. It is very easy to forget that certain inputs will present the user with different options.

The other important data issues to identify are any data dependencies and any potential problems with concurrency. Is it important that data is used in some business functions before they are used in others? And, will data modified by virtual users cause other virtual users to fail when they try to use the same data? The test tool can partition the data used by each virtual user if these requirements can be identified. It can be difficult for a performance tester to debug test script failures with little knowledge of the application, especially if the failures only occur when multiple virtual users are running at once.

One of the most important pieces of information a performance test is designed to discover is the response time of the system under test – both at the overall business function level and at the low level of individual steps in the test case, such as the time it takes for a search to return a result set. Any test cases provided to a performance tester should clearly define the start and end points for any transaction timings that should be included in the test results.

It is important to remember that the test script is only creating the network traffic that would normally be generated by the application under test. This means that any operations that happen only on the client do not get simulated and therefore do not get included in any transaction timing points. A good example would be a client application that runs on a user’s PC, and communicates with a server. Starting the client application takes 10 seconds and logging in takes 5 seconds but, since only the login is sending network traffic to the server, the transaction timing point will only measure 5 seconds.

Operations that only happen on the client, including the time users take to enter data or spend looking at the screen is simulated with user think time – an intentional delay that is inserted into the test script. If no think time is included, virtual users will execute the steps of the test case as fast as they can, resulting in greater (and unrealistic) load on the system under test. Depending on the sophistication of the performance test tool, the user think time may be automatically excluded from the transaction timing points. Think times are generally inserted outside of any transaction timing points anyhow.

While a functional test case will be run once from start to finish, a performance test case will be run many times (iterated) by the same virtual user in a single scenario. Information on how the test steps will be iterated should be included in the test case. For example, if a test case involves a user logging in and performing a search, and the entire test case is iterated by the virtual user; then a test scenario may be generating too many logins if the real users generally stay logged into the application. A more realistic test case may have the virtual user log in once and then keep doing the same action for as long as the virtual user is run.

When a script is iterated, consideration should be given to the non-obvious details of how it is iterated. A good example would be a test script simulating users using an Internet search engine. When the test script is iterated, simulating a new search operation, should the virtual user establish a new network connection and empty their cache or should every iteration simulate the same user conducting another search?

As all performance test tools have different default behaviours, a good performance tester should clarify this type of detail with business and technical experts. Some performance test tools make these details easier to change than others. If it is not practical to emulate all the attributes of the expected system traffic with a particular tool, then it should be noted as a limitation of the performance test effort, and a technical expert should assess the likely impact.

Hopefully this article has provided some insight into the extra considerations that must be given when writing a performance test case, rather than a functional test case. As with any software specification, a performance test case may need to be refined as questions are raised by the performance tester.

An example test case: