Creating a plan for load and performance testing SAP involves a lot of decisions. While most projects follow a similar process, every project has unique needs, and it can be difficult to balance the pros and cons of each choice. This post outlines some of the key questions to be answered when load testing SAP.
Timeline
There are a lot of activities to coordinate on an SAP project and schedule slips get expensive very quickly. It is important to keep load and performance testing off the critical path, as it can quickly blow out timelines when problems are found. It is impossible to estimate the time required to fix performance defects when the root cause is unclear, or when it requires SAP OSS tickets to be responded to. Even when a load-related defect is resolved, it is unknown whether there is going to be another bottleneck at a slightly higher level of load.
The best way to keep load testing off the critical path is to test early. Single-user response times can be measured as functionality is delivered in a smaller QA environment. Performance under load must still be measured in a production-sized environment, but it is not necessary to wait for 100% of functionality to be complete before testing the system under load for the first time.
The load testing timeline takes its cues from the timeline for functional testing which, in-turn is based on code drop dates from the dev schedule. Early load tests are great, but the final validation of the system under peak load cannot happen until functional testing has finished. If previous load tests have been run successfully, this final validation should be a rubber-stamp exercise.
If the project has paid for the SAP Volume Test Optimization (VTO) service, then this must be in the schedule too. The VTO team will visit the project for a few days as the deployment date gets close, double-check the performance-related configuration settings and observe system metrics while a peak load test is run (they do not run any load tests themselves). It is a waste of time (and money) if there are still lots of load and performance defects. The peak load test should be running cleanly before they arrive on-site.
Resourcing
Who is going to support load testing? Business Analysts? Functional Consultants? Basis team members? Who will fix defects? What if it is a network problem, or related to an external system that the Basis team does not control? Who will create accounts and other data needed for load testing?
Who is actually going to do the load testing? Is it going to be someone supplied by the vendor? Does that risk a conflict of interest situation? Should you hire a contractor, or a consultancy firm? Will they provide an SAP performance testing specialist, or just a generalist performance tester? Will they accomodate a flexible roll-on/roll-off model so that they can test early, then take a break while more functionality is developed? Does the vendor provide a warranty period after go live? Will the performance tester be available to compare real-world usage metrics against those that were used for testing, or to update the tests and re-run to reproduce a problem in production?
Environment
Which environment will load tests be run in? Does it match the size of the production environment? Is your project greenfield (new) or brownfield (adding to an existing system)? Greenfield projects can use the future production environments, while brownfield projects may have to use the DR environment (and double its disk capacity to fit a production and a test database. Cloud deployments have made it much easier to get a temporary production-like environment in a budget-friendly way.
Where will the data for the environment come from? Does the project include data conversion/data migration? Having a realistically sized database is important because complex SQL queries are always fast when the database is nearly empty.
How will changes be managed? Will the client be open (allowing untracked changes), or will changes require a transport? Functional changes might mean that load testing scripts have to be updated or rewritten, but you can’t freeze changes forever, and you want to balance early performance testing with the fact that you need to load test the final version. Should you take functional changes in the middle of an iterative performance tuning exercise? Should you sync changes with the QA environment?
What load generators do you need to organize? Will the be on-premises or in the cloud? Where will your user be located? How many servers (or CPU cores) will your load testing tool need to generate the required load?
What will happen with system interfaces? Will they be stubbed? What stubbing tool will be used? Will they be connected to real, non-production instances of the interfacing system? Who is responsible for the interfacing systems? Are they production-sized? If the interface is in-scope for load testing, can you run load against them, or will load have to be injected at the interface boundary?
Will there be an environment available after go live? If a load-related problem happens in production, where can the problem be reproduced (so it can be investigated, and so that a fix can be validated instead of blindly deploying it into production)? Where will future updates be load tested before they are deployed?
Scope
What will be the scope of load and performance testing? Which components in the SAP landscape? Are there interfacing systems? Which interfaces drive sufficient volume to be included in load testing? Will there be remote users? Mobile users? How will high latency/low bandwidth mobile data connections impact response times?
What does the workload model look like? You can’t load test 100% of functionality. Which business processes are high-volume? Which transactions/applications have been customized, or are high risk for another reason? Which batch jobs should be tested with bulk data? How many users will be on the system?
Which performance test cases should be in scope? Will you run a soak test? Should you test failover under load? Should you include a load test as part of Dress Rehearsal? Will there be batch jobs that run during the day? Will they run at the same time as your daytime peak without slowing down the system or failing due to locked records from online users?
Tools and Monitoring
What load testing tool will you use? Does it support all the interfaces that are in-scope? If you are testing Fiori, you will need a load testing tool that is based on a headless browser, because you will be scripting forever if you use an HTTP-level tool. To you need to buy licenses for your testing tool? Which license is best for the project? Will you be using LoadRunner?
How will the infrastructure components be monitored? How will monitoring data be integrated into your load testing tool? Will the monitoring tool used during load testing be different to the one used for production monitoring? What tools will be used to investigate performance defects? Does the project have the ability to see where time is being spent at a code-level? Does the project have someone who has the skillset required for this kind of problem investigation? Can you use the Real User Monitoring capabilities of SAP to measure response times from functional testing in the QA environment?
Victor Kouznetsov
After working on a lot of SAP projects, it has become clear that an on-demand model makes the most sense for SAP performance testing. All projects experience delays – whether it is as simple as waiting an extra week for a test environment or waiting six months for critical functionality to finally be delivered. It makes no sense to have a full-time performance tester sitting around waiting for something to test.
The MyLoadTest SAP performance testing service is remotely delivered, and designed to flex up and down depending on the needs of the project. Sometimes this requires full-time focus, and sometimes this requires answering an email or two just to keep things moving.
