Testing doesn't increase quality

At a first glance this is a rather surprising thought. But the explanation McConnel gives in "Code Complete" is obvious: Testing doesn't include debugging or restructuring the code. So testing technically doesn't change the code at all. Finding and fixing the defects hopefully does increase code quality, though.

Speaking about tests, there's another interesting observation in McConnel's writing: You find only less than 50% of all errors by testing. So regardless how much you test, you're still doomed.

Since one of my mental quirks is to state the extremes and think about logic afterwards, what if we don't test code, but just write it - the worst thing we could get is about 1.8 times the defects. The code is buggy anyways, so who cares? Go banana software! We just save a lot of development time (for managers: read "money") and our time to market rocks (did I hear anybody say the name of some company in Redmond?). We just don't test. Remember: The process is as easy as 1: write software, 2: release it to the customer without testing, 3: yea, um, right.

The point is that 3 is the magic number: Software is not finished once it's released to the customer. In fact, many books state that most of the software development cost is spent after the program is in outer space. This is called "Software Maintenance", which is just a nice term for "fixing defects that shouldn't have been there in the beginning". Now what does this mean for our banana software?

Since 40 years software development textbooks feverishly try to get the message out that fixing defects is more expensive the later the defect is found in the software life cycle. Why's that? Simple. Did you ever get an error report from the customer where you needed 4 phone calls and half an hour of talking to different people before figuring out that "The Big Red Button isn't working" means that the program just segfaults every time the cancel button is pressed by the customer and it's simply working at your own computer. Now you spend a day or two, trying to get an exact copy of the customer's configuration, even installing Windows 95 (just to be sure), before finding out the reason of the problem. And than you know what the problem is, but you'll still have to fix it. See the big glowing blue productivity cloud going <poof> over your head.

Now if we had found this error early in a testing stage, it would have been a lot cheaper to fix. So taking into account what we learned earlier, namely that testing doesn't increase quality, what does testing yield? Here's the answer: Testing gives me the opportunity to gain productivity by fixing problems found now [1]. Later the problem will stay clear of my neck and I'm free to do other things, like not staying at work late (again).

The other extreme to completely bananaing your software is found in Kent Beck's "Extreme Programming" practices: Test Driven Development. Although test driven development is more than just "testing" [2], it addresses the old law of software engineering: In order to boost (mean) productivity you have to boost quality. And improving quality by fixing defects at the earliest possible time is the cheapest you'll get.

In my opinion there's still a big BUT. Test driven development increases productivity, BUT only on the long run. You have to pay up front. This means that more capital is bound at an early stage in development. Kent's solution is to include the customer in the development cycle, release as early as possible and as often as possible to get feedback and minimize risk. With my limited experience in software products, I really can't decide if this approach is feasible, or if time to market constraints sometimes get a project enough trajectory to compensate for the increased develpment cost. If early time to market limits competition, history tells us that reality is a lot more complicated than textbook-optimized methotology.


[1]: Erdogmus, H.; Morisio, M.; Torchiano, M.; Software Engineering, IEEE Transactions on Volume 31, Issue 3, March 2005 Page(s):226 - 237 Digital Object Identifier 10.1109/TSE.2005.37
[2]: Janzen, D.; Saiedian, H.; Computer Volume 38, Issue 9, Sept. 2005 Page(s):43 - 50 Digital Object Identifier 10.1109/MC.2005.314