Why HP is wrong about the Web Security Lifecycle

In my first post I gave a very brief overview of the web security testing products offered by HP. Unfortunately people’s understanding of where the products should fit into the software development lifecycle is still weak. This is even the case inside HP.

Here is a current slide from HP Software…

The obvious, glaring problem with this diagram is that WebInspect is being promoted as a tool to be used in Production. As someone who has spent a long time working in highly technical areas of testing, I have some huge problems with this, and most of my clients will not like this either.

The bottom line is…if you use WebInspect in Production, you are a cowboy and have no place in enterprise-class IT.

1. If you are waiting until after Go-Live to test something, then YOU ARE DOING IT WRONG! Hopefully you don’t wait to test for functional or performance problems until after you have rolled out your application to your customers. End users will probably not notice if your website has security problems, but if customer data is stolen, then there is no way to get it back. Unlike other defects, you can’t fix theft of customer data by releasing a patch (you can only prevent it happening again).
2. Defects found in Production are much more expensive to fix than defects found earlier in the software lifecycle. The earlier you find (or prevent) the defect, the less it costs to fix. DevInspect helps prevent security problems during the Build phase through code analysis and by providing secure components, but it does not allow you to conduct a thorough audit of your web application. WebInspect should be firmly targeted at the Test phase (I will discuss QAInspect shortly). Also, if you find problems in Production and the development team fixes them, then it will be necessary to run another round of functional and performance testing to verify that nothing has been broken by the security fixes.
3. When WebInspect does a crawl and audit of a website, it tests input points by submitting data. Larry Smith likes to tell a story about a company that used WebInspect in Production, and ended up generating 60,000 emails, and previously I mentioned that a Google search for one of the values that WebInspect uses for form submissions returns ~87,500 results.  You do not want to create garbage data in your Production database. This might not be a problem if you have a basic brochure-style website with no points for user input, but if you run an important website with online ordering (the sort of website you really must security test), then garbage data will cause lots of headaches.

The obvious question is “Isn’t QAInspect already targeted at the Test phase? Why does WebInspect need to be used during this phase too?”. Well, I’m glad you asked…

First, let’s have a look at comparative pricing of these products for a single seat perpetual license. QAInspect is around 25% of the price of WebInspect, so obviously WebInspect is offering something that QAInspect is not.

QAInspect is geared towards users who are not expected to have any understanding of web security testing. It is a fully automated push-button testing tool that is launched from Quality Center. It will scan an application for known vulnerabilities and execute heuristic attacks for some classes of problem (like SQL injection). There are many vulnerability types that are impossible to create automated tests for, and that is where a skilled security tester armed with a copy of WebInspect really shines.

This leads to some conclusions…

1. DevInspect is good for preventing security problems during the Build phase (code analysis, secure components). Like QAInspect, it can also perform a crawl and audit of the web application for known vulnerabilities and some classes of problem (using heuristics). It is a low-cost product, so each web developer could be given a copy. It would be useful for preventing/detecting web security problems early, but it has no chance of finding some classes of vulnerability.
2. WebInspect allows a skilled user to perform a thorough security audit of a web application through automated scanning and tool-assisted testing. This ideally happens in a test/pre-Production environment before a new version of the software is released.
3. QAInspect is simple enough for anyone to use but, as it is a fully automated solution, will not detect every class of security defect. It is cheap enough that it makes sense for a project (that uses Quality Center) to buy a copy so that they can run a regular regression test for security problems.
4. As the crawl and audit part of DevInspect and QAInspect will miss certain classes of defect, it makes sense to do a pass or two with WebInspect with a skilled security tester driving it. This fact also makes it highly irresponsible to sell either of these products to a company that does not already have WebInspect.
5. So…use DevInspect to find problems early in the lifecycle. Once the software is close to feature-complete and stable enough for functional testing, then it would be great to do bring in your expensive security testing guru and do a pass with WebInspect. As you don’t want to keep an expensive security tester hanging for too long, you can get your regular testers to run regular overnight regression tests on each build. The security tester will need to be available to investigate any defects found, and eliminate any false positives from the report. Before deploying the application, a final pass should be executed with WebInspect. Any future releases will be regression tested using QAInspect and, depending on what has changed, can be fully regression tested using WebInspect (or WebInspect could be used to do focussed tests on new functionality).