Static code analysis for LoadRunner scripts

The idea of static code analysis has been around since at least the 1970s. These days, some kind of static analysis is usually built into most good quality IDEs (or it is available as a plug-in).

Static analysis tools find bugs in software by looking at the program’s source code rather than executing the program. While the compiler will pick up syntax errors, there are thousands of other types of errors that a developer can accidentally create while programming – and, unlike compiler errors, there is nothing to tell you that you have made a mistake unless you are using a static analysis tool.

Even though LoadRunner scripts are relatively short and are generally conceptually simple, there are still a large number of ways to make (non-obvious) mistakes. LoadRunner users are in desperate need of their own static analysis tool (even if they probably don’t know it yet).

John Carmack (creator of the computer game Doom) wrote a great piece at the end of 2011 called In-Depth: Static Code Analysis, where he concluded that “it is irresponsible to not use it.” Here are a couple of paragraphs from his piece:

The first step is fully admitting that the code you write is riddled with errors.
…
We had a period where one of the projects accidentally got the static analysis option turned off for a few months, and when I noticed and re-enabled it, there were piles of new errors that had been introduced in the interim.
…
Automation is necessary…Exhortations to “write better code”, plans for more code reviews, pair programming, and so on just don’t cut it, especially in an environment with dozens of programmers under a lot of time pressure. The value in catching even the small subset of errors that are tractable to static analysis every single time is huge.

What mistakes do LoadRunner users make?

A majority of performance testers approach the development of LoadRunner scripts with the wrong mindset. They think that their goal when writing their LoadRunner script is to get the script to run without error (“It finally runs all the way to the end! I’ve finished!”). But a LoadRunner script is “test code”. Its purpose is to expose errors in the system that is being tested.

Testers who are focussed on getting their scripts running rather than focussing on making their scripts run in a way that is likely to find real errors are doing it wrong. They are like developers who make their unit tests pass by commenting out the assert() statements. The worst example of this I have seen was a tester who was scripting a Create Order business process. He thought his script was “working” because it ran all the way to the end without throwing any errors, but no orders were being created in the system when the script ran.

There are three main areas where performance testers make errors when writing VuGen scripts:

1. Script runtime settings
   A single runtime setting can cause your entire load test results to be inaccurate. My favourite example is the misleadingly named “non-critical resource errors as warnings” setting, which causes transactions to pass even if so-called “non-critical” resources like JavaScript files, CSS files, images, JSON objects, or PDFs are causing HTTP 500 errors. This dangerous setting is enabled by default.
2. Source code: LoadRunner-specific code
   VuGen already has one check built into the product. You will receive a warning if you include web functions in the vuser_init action when “new user on each iteration is selected in the runtime settings. Of course there are many other things people can do wrong with their LoadRunner code.
3. Source code: Badly written C code
   Doing things with memory (even simple things like manipulating strings) always seems to trip people up. Data types are another good source of errors.

This last point is solidly in the domain of classic C-based static analysis tools. There are lot ways to mess up C code, especially if you are already familiar with languages like Java that look similar, but handle memory in a totally different way.

If the script you have written is throwing intermittent memory access violation errors, then it is going to be a problem with the code you have written, rather than a bug in LoadRunner. Here is an example:

// An example of a function that seems to work, but contains a memory error.
char* reverse_string(char* string_to_reverse) {
  int i; // position in output string
  int j; // position in input string
  int length; // length of input string
  char buf[100]; // allocate a buffer to hold the reversed string (99 chars + NULL terminator)
  
  // Check that the input string is not too long (buffer overflow)
  if (strlen(string_to_reverse) > 99) {
    lr_error_message("Input string must be less than 100 characters");
    lr_abort();
  }
  
  // Reverse the string
  for (i=0, j=length; i
When I was working as a Performance Testing Practice Lead I would train new performance testers and, for their first few projects, I would generally sit with them and do a code review. Over the years I found myself saying the same things again and again. Like any good technically-inclined person, I started thinking about whether I could automate this part of my job.
A web-based VuGen script validator
When I was building websites, I would always run my HTML through the W3C Validator. It would show me a list of all the errors I had made, and tell me why each one was wrong. This "validator" concept was my original mental model for how I could get lots of people to use a static analysis tool for LoadRunner (low barrier to entry, nothing to install).

But it quickly became obvious that the model that worked great for checking websites was ill-suited to checking large LoadRunner scripts. The workflow was horribly clunky (zip script, upload 50MB zip file to website, read report, update script, repeat multiple times until all errors are fixed), and the web stats showed that hardly anyone was using it.
Turning it into an Add-in
I was part of the LoadRunner 11.5x beta program in late 2011, so I knew that HP would be changing VuGen from the being based on the SlickEdit IDE to being build on top of the SharpDevelop IDE.
One of the SharpDevelop features I got excited about was the ability to extend the IDE with add-ins. Writing an add-in would be the perfect way to integrate static code analysis seamlessly into a performance tester's script development workflow.
In June 2012, I gave a presentation at HP Discover in Las Vegas called "Extending VuGen 11.5 with custom add-ins where I spoke enthusiastically about how add-ins would kick-start a new era for LoadRunner users and showed off the simple user interface of my own add-in.

My pitch for the add-in was:

Find common errors in VuGen scripts and runtime settings
Consistently apply best practices during script development
Apply a quality check to scripts developed by external teams
Automatically fix some errors
Develop reliable scripts faster

Turning it into a product
It wasn't long before the company I worked for started thinking about how they could make money from this neat idea. This was hugely exciting for me, but I was soon to learn some painful lessons about building and trying to sell new software products...
Firstly, the market for a LoadRunner add-in is going to be limited by the number of people who actually use LoadRunner. While LoadRunner is a popular tool, the number of users is quite small when compared with (for example) the number of people who use C#. This effectively prevents you from following a "sell cheaply to many" business model.
Secondly, all of your customers are the kind of companies who buy LoadRunner: large corporates - utilities, banks, telcos, etc. The kind of companies that are horrifically expensive to sell to. The sales cycle will probably stretch over 6 months and will require endless face-to-face meetings, PowerPoint presentations, proof-of-concept exercises, and tricky navigation of their internal procurement processes. Cost-of-sale tends to dominate development cost over even a small number of units. This is the very opposite of a low-touch sales model where someone sees a Google ad, then clicks through to a web form to make a purchase without talking to anyone.
Finally, there is the difficulty of who to pitch the product to. If you pitch it to a technical audience and say "this product is not free", they generally have one of two reactions:

They will think of a vaguely similar product that has a much larger market and a low-touch sales model (such as ReSharper from JetBrains, which costs $249), and insist that they wouldn't ever pay any more than that.
They say "I could build that" because, as far as they care, the cost of their time is $0/hour. At the risk of offending people, engineers seem to have three main weaknesses: 1. Massively under-estimating the development effort that went into the products they are evaluating. 2. Being incapable of performing the basic math involved in a build vs buy decision. 3. Preferring to spend months building tools or "frameworks" rather than actually doing things that create business value for their company ("it will totally save time in the long run...and it's way more fun than the boring stuff my boss wants me to do.")

Selling to higher-level people who are in charge of testing is not much better. Their basic unfamiliarity with standard software development tooling means that static analysis tools are an alien concept. I found that a large number of them couldn't seem to grasp why you would want to automatically find coding errors at the time they were being made, rather than occasionally applying an unreliable manual check at some distant point in the future. 
Some things I heard while talking to customers:

"We already have a Word document containing our scripting standards. Our testers know that they will get in trouble if they don't follow them."
"For the price of your tool for my entire team, I could hire someone for a few hours a week to check everyone's scripts manually."
"My team already does internal peer-review of each others scripts."
"Can't you just give me a list of the things your tool checks for? I will get someone in the team to manually check all the scripts for your list of problems."
"Rather than buying a copy of your tool for every member of my team, I will just buy one seat license, and make a team member responsible for checking everyone's scripts."
"If I buy your tool and improve the quality of my performance testing team's scripts, I am worried that the new LoadRunner scripts will find a much larger number of defects. This might make our test cycles take longer."


I once did a proof-of-concept for a large bank. I ran the tool over the latest version of all the scripts they were currently using for testing. The report showed that 0% of scripts followed their internal scripting standards, and over 60% of scripts "lacked even basic checks".
At the time, they were paying their outsourced performance testing partner (an offshore consulting services company) millions of dollars per year to write LoadRunner scripts and run tests. You would think that the news that they were wasting most of their budget would light a fire under them, but their response was "in six months we will be allocating budget for IT efficiency initiatives; so let's talk about it then."
Conclusion
I would love to see static code analysis used widely by performance testers while they are developing their LoadRunner scripts. Unfortunately there is little demand from performance testers or their managers, and the kind of companies that use LoadRunner are very expensive to sell to.