Integrating Unit Tests In Ruby Scripts

When I write ruby scripts I like to use a single file, containing the program and all unit tests. It took me some time to find out how to add a command line switch to my ruby scripts that makes them run in script mode with full access to the Test::Unit command line arguments, while being able to run the script without the test framework interfering in the execution:

#!/usr/bin/ruby -w
require 'test/unit'

if ARGV.include?("--test")
  ARGV.delete_at(ARGV.index("--test"))
else
  Test::Unit::run = true
  puts "Running..."
end

Now you can simply run the program by typing

$ ./testme.rb
Running...

or run the tests with

$ ./testme.rb --test
Loaded suite ./testme
Started

Finished in 0.0 seconds.

0 tests, 0 assertions, 0 failures, 0 errors

The nice thing is that only the first "--test" will be removed, so you can still leverage the Test::Unit command line argument interface.

My First OCaml Tests - So Close To Heaven!

Some time ago a a discussion in the testdrivendevelopment Yahoo-Group evolved around the concept of "testable languages". I thought about this for a while and came up with the idea that I want to be able to have expressions as first class citizens:

assertThat { assertThat(false) } abortsWith TestFailedException

Today I played a little with OCaml, sorted my functional programming skillz out, and finally arrived at my first test driven unit test environment for OCaml! Note that it's nearly what I wanted to be able to write, but unfortunately there's not enough syntactic sugar for lambda expressions (or I didn't find out, yet), so I'm stuck with using the quite ugly ( function () -> expression ) syntax. But hey, it's really close to heaven.

exception TestFailed
exception TestError

let failsWith expectedError expression =
	try
		expression ();
		false
	with error ->
		expectedError = error

let isTrue expression: bool =
	expression

let isFalse expression: bool =
	not (expression)

let assertThat expression conditionMatchesOn =
	if not (conditionMatchesOn (expression)) then 
		raise TestFailed 
	else 
		()

let _ = (
	assertThat true isTrue;
	assertThat (isTrue true) isTrue;
	assertThat (not (isTrue false)) isTrue;
	
	assertThat false isFalse;
	assertThat (isFalse false) isTrue;
	
	assertThat (TestFailed = TestFailed) isTrue;
	assertThat (failsWith TestError (function () -> raise TestError)) isTrue;
	assertThat (failsWith TestFailed (function () -> raise TestError)) isFalse;
	assertThat
		( function () -> assertThat false isTrue )
		( failsWith TestFailed );

	Printf.printf "OK\n";
);;

XPDays Germany 2007 - Ideas Going Mainstream

I had the opportunity to take part in the XPDays Germany last week. The company I work for enabled Uwe, our project lead, Holger and me to participate. It all started with a three and a half hour ride from Munich to Karlsruhe where we heroically overcame a nearly empty tank, a shaking car that felt like it just drank the wrong kind of gas and my own card reading skills - or lack thereof.

Day 1

In the end we arrived on time for a Randori session by Dave Nicolette and Rod Coffin. From the moment I learned of this experimental learning session where two people sit in front of a computer and test drive a piece of code while the whole audience is throwing in questions, I was kind of scared of the prospect of being watched while writing code by a hundred people - which is probably kind of normal, given that some people even dislike being watched while pair programming.

The session turned out to be very interesting. One of the key elements of this style of learning is that the audience's energy level stays high for a long time - you have to pay close attention, since due to the random selection of the next person to come to the front you could always be this person. And since you don't want to look like a fool when doing stuff in front of a hundred people (um, what was the problem, again?) my adrenaline level alone was enough to keep me awake.

The other thing I learned from this experience besides a new way to coach technical stuff is that there is a very good reason we do "pair programming" and not "group programming". Throwing two brains at a problem can be a very mighty tool to solve programming tasks, but in some situations throwing a hundred brains onto a very simple programming example felt like sitting in one of Dilbert's most unproductive meetings:

"So we add the game to the character: character.addGame(game)"
"But why don't you add the character to the game?"
"I see duplication, I see duplication!"
"But isn't this, um, less expressive?"
"Duplication is bad!"
"Why do you assign null to a variable, this is done automatically"

After some time of silent observation I realized that sometimes I am exactly like this! So to all of you who have to cope with me on a daily basis: just hit me on the head with a big club from time to time.

The next session was a series of "lightning talks" about all kind of Agile topics where I learned about Alistair Cockburn's Crystal, which is a set of methodologies built upon the Agile principles and an extreme tailoring approach.

Day 2

Despite the enormous amount of a whole liter of "badisches Helles" I had in the evening I was wide awake and ready to suck in new ideas during the conference's main part. The day started with an interesting presentation by Dave Nicolette about how to communicate TDD and design debt to your management. I was particularly stunned by the fact that he did talk about the cost of design debt and the refactoring part in the TDD cycle for a very long time without mentioning the cost of fixing an error in terms of when it is found and the shortened feedback cycles that TDD provides.

The next presentation was titled "why Agile projects fail", but turned out to be about why projects fail in general and provided some insight into the ideas of root cause analysis (5 Why), the dimensions in which failure can occur (The Broken Triangle), and the psychological factors that are the real cause of ineffective development practices.

After lunch the keynote by "Dark Side" Rod Austin from HBS showed how Agile development fits the icy wind of change that swirls today's leading companies in the world from a cost competitive to an innovative business model. After all, who doesn't want a designer trash bin?

Stfan Roock's talk on "Simplicity in Software Projects" was a very entertaining lecture on how easy it is to get so accustomed to complexity that you don't even realize how simple things could be. Well, that and that the Borg are the only entities in the universe who understand that when you travel through space aerodynamics is pointless.

In the end it was very interesting to see big German companies like SAP, EADS and Siemens to take interest in extreme programming. Looks like those ideas are finally going mainstream.

Does "Test After" Work For You?

"Why not write a test for this?"
"Why should I, it works..."

The idea of Test-After Development is to write a set of automated white-box tests after writing your production code. Since probably every CS student in the world has learned that unit tests are a good idea, you'd expect unit testing to be an industry state standard for quite a while now. Interestingly the idea of automated unit and integration test is lately becoming more popular due to the widespread use of Test-Driven Development.

So why do we need Test-Driven Development to be able to efficiently write automated unit tests?

• If you write your code first and don't think about how to test the code, the code will not be testable. Thus testing becomes expensive and frustrating. Test-Driven Development will guide your software design by the old mantra of "how-do-I-want-to-use-this-class", leading to a highly decoupled design.


• When you write your tests, you'll discover a lot of errors. But instead of the red bar in Test-Driven Development, which you expect, the red bar in Test-After Development is the demotivating sword of reality.


• The most important reason why I have never seen Test-After Development work, is that developers just don't believe in errors once they wrote the code. This seems to be an eternal wisdom of software development psychology: once the code works, why bother testing it? Let's just implement the next feature.

A Program Is Born: QCMake's First Functional Test

When you write a GUI that is just a thin layer for an existing business layer and you don't see how to integrate test fixtures into this business layer, you'll be down in the dirty functional testing work very quickly. This happened to me today when I tried to write my first test for a small Qt facade object for cmake.

I started the test very enthusiastically: To test cmake I create a directory, cd into that directory, create a CMakeLists.txt and let cmake create a CMakeCache.txt. In the end I know that cmake ran when CMakeCache.txt exists.

void QCMakeControlTest::shouldExecuteCMakeInTheCurrentDirectory()
{
  QDir currentDirectory;
  QDir testDirectory(currentDirectory.path() + "/ExecuteInCurrentDirectory");
  QVERIFY(currentDirectory.mkdir(testDirectory.dirName()));
  QVERIFY(QDir::setCurrent(testDirectory.path()));
}

I hit F5 and everything runs just fine. Once. The second time the directory ExecuteInCurrentDirectory already exists. Of course to have a nice and clean starting point the test must remove the test directory if it already exists. So I added:

void QCMakeControlTest::shouldExecuteCMakeInTheCurrentDirectory()
{
  QDir currentDirectory;
  QDir testDirectory(currentDirectory.path() + "/ExecuteInCurrentDirectory");
  if(currentDirectory.exists(testDirectory.dirName())) 
  {
    QVERIFY(currentDirectory.rmdir(testDirectory.dirName()));
  }
  QVERIFY(currentDirectory.mkdir(testDirectory.dirName()));
  QVERIFY(QDir::setCurrent(testDirectory.path()));
}

Green. Perfect. Now let's create a CMakeLists.txt.

void QCMakeControlTest::shouldExecuteCMakeInTheCurrentDirectory()
{
  QDir currentDirectory;
  QDir testDirectory(currentDirectory.path() + "/ExecuteInCurrentDirectory");
  if(currentDirectory.exists(testDirectory.dirName())) 
  {
    QVERIFY(currentDirectory.rmdir(testDirectory.dirName()));
  }
  QVERIFY(QDir::setCurrent(testDirectory.path()));

  QFile cmakeLists("CMakeLists.txt");
  QVERIFY(cmakeLists.open(QIODevice::ReadWrite));
}

Green again. Once. The test fails the second time it's executed:

********* Start testing of QCMakeControlTest *********
Config: Using QTest library 4.3.2, Qt 4.3.2
PASS   : QCMakeControlTest::initTestCase()
FAIL!  : QCMakeControlTest::shouldExecuteCMakeInTheCurrentDirectory()
  'currentDirectory.rmdir(testDirectory.dirName())' returned FALSE. ()
..\..\..\..\..\Source\CMake\Source\QTDialog\qcmaketest\QCMakeControlTest.cpp(30) :
  failure location
PASS   : QCMakeControlTest::cleanupTestCase()
Totals: 2 passed, 1 failed, 0 skipped

Yep, no problem, all I need to do is to rmdir recursively. Just a quick glance into the Qt docs. But I found nothing. Well, it's not too hard to implement a recursive rm -rf, but still... I was so sure that this function would be hidden somewhere that I spent more time googling and doc-reading than implementing it when I finally realized that I was on my own. So in the end the test looked a little bloated:

#include "qcmaketest/QCMakeControlTest.h"

#include "qcmakeui/QCMakeControl.h"

bool removeRecursiveForced(QDir& directory, const QFileInfo& entry)
{
  if(!entry.isDir())
  {
    return directory.remove(entry.fileName());
  }
  QDir directoryEntry(entry.filePath());
  QList entries(directoryEntry.entryInfoList
    (QDir::Dirs | QDir::Files | QDir::NoDotAndDotDot));
  for(int entryIndex = 0; entryIndex < entries.count(); ++entryIndex)
  {
    if(!removeRecursiveForced(directoryEntry, entries.at(entryIndex)))
    {
      return false;
    }
  }
  return directory.rmdir(entry.fileName());
}

void QCMakeControlTest::shouldExecuteCMakeInTheCurrentDirectory()
{
  QDir currentDirectory;
  QDir testDirectory(currentDirectory.path() + "/ExecuteInCurrentDirectory");
  if(currentDirectory.exists(testDirectory.dirName())) 
  {
    QVERIFY(removeRecursiveForced(currentDirectory, 
      QFileInfo(currentDirectory, testDirectory.dirName())));
  }
  QVERIFY(currentDirectory.mkdir(testDirectory.dirName()));
  QVERIFY(QDir::setCurrent(testDirectory.path()));

  QFile cmakeLists("CMakeLists.txt");
  QVERIFY(cmakeLists.open(QIODevice::ReadWrite));

  QCMakeControl qCMakeControl;
  qCMakeControl.configure();

  QFile cmakeCache("CMakeCache.txt");
  QVERIFY(cmakeCache.exists());
}

#include "QCMakeControlTest.moc"

At least I have an idea where this could lead me - a nice class to generate a clean cmake directory. But let's see whether I'll be right, perhaps YAGNI will finally get back at me. And if you know an easier way to delete a directory recursively with Qt, please leave a comment.

Fasten Your Seat Belts! - Chances And Pitfalls Of Test Driving Your Development

Ten years ago I had a rendezvous with a beautiful girl, and at the end of the evening I gave her a ride home. Back than I thought playing the gentleman to be posh, so I opened the door for her. She slid with one elegant movement into her seat and I paced around the car and folded my wiry frame behind the steering wheel. I looked at her.

"Don't you want to start?", she asked and watched me curiously. "Um", I said, obviously always finding the right words at the right moment. "Um. - Not before you buckle up...". She frowned at me: "But I never buckle up". I replied "Well, if you don't buckle up, we're not gonna go anywhere tonight". "Oh come on!", she now somewhat furiously stated. "Nope!" I insisted eloquently, finally feeling her shield of stubborn resistance falter. It took a few seconds before she realized that I really wouldn't drive her home without her being properly protected from falling through the windshield and decomposing her pretty head by hitting the next best fireplug. So she buckled up.

Even back than I was so used to the secure feeling of the protective belt that just thinking of driving unbelted drove an uneasy quiver through my guts. This feeling is so strong that if I don't drive strapped when moving the car just a few inches there's always a sense of awareness that makes me want to fasten the seat belt immediately.

Today I felt exactly the same way while writing code.

The path of the test

At the beginning of the last iteration we identified a story that affected some legacy modules in our code base. When we recognized that the changes we needed to make would touch more code than we had thought, so we decided to try to test drive a part of the system from scratch to replace the tangled old code. So Richard, Reinhard and myself started to pair on the story alternately. Besides some private experiments with a Sudoku solver in Java this was the first time I was doing real full time TDD pair programming for a couple of days. Aside from some initial irritation and the constant realization that pair programming is hard to learn I was quickly pulled into the red-green-refactor cycle, as usual. But this time I held the pace for longer than ever. And was pulled into the cycle deeper and deeper. Write a test, make it work, look for redundancy. Write a test, make it work, look for redundancy. Write a test...

Today I wanted to quickly integrate the changed interface into an existing module. I didn't have a test yet. The cpp file was readily opened in my editor. Just a quick edit, nothing more than integrating this interface. A few simple edits. Only three lines or something. And suddenly a nagging question materializing in my head:

How can I make sure this works?

At this moment I felt like driving unbuckled. I felt unsafe. I wanted my cozy safety net back. Like an addict I went for the next test.

What use is a seat belt when you hit a tree at 200 MPH?

Since I'm the one driving the adoption of XP in our company, I wanted to try TDD for myself on a save playground to learn more about the ins and outs before applying it at work. Since Java has really nice tools for TDD, I started test driving a small Sudoku solver in Java. This was my first real test driven code and I often wondered about how nicely the test suite covered my errors. Spirited in the Agile fashion, I began with a really straight forward brute force implementation. Everything went a lot more smoothly than I had expected and after some coding I had a simple solution that needed over 90 seconds for one simple Sudoku.

After a while I wanted to optimize the runtime. So I introduced some caching variables. I struggled with the failing tests as my solution grew more and more sophisticated, but the tests helped me to get to a deeper understanding of the real problem. Finally I arrived at a point where the algorithm managed to work through 1400 Sudokus in less than a second. I was thrilled. And I wanted more. So I installed a profiling framework to find out where the next optimization sweet spot would be hidden. When I browsed the profiling data I realized that the real solver didn't even call the algorithm. So I had benchmarked a program that didn't solve any Sudoku at all.

At this moment I felt like hitting a tree with 200 MPH, suddenly realizing that it is not a good idea to drive that fast into a 90-degree turn on a wet street, even if you have a seat belt.

After the blood had returned to my head on it's way to my brain I implemented a test into the main program to check every solution with a simple algorithm before claiming to have solved anything. In the meantime I have a solution that runs 1400 Sudokus in 6 seconds on my core 2 notebook. I'm even quite convinced that I got the solution part correct...

Do I get my driver's license?

So, here's the lesson I learned on this journey on my path to the test:

• Don't rely on your tests too quickly.
  If you want to heed the XP advice to "test everything that can possibly break", be aware that it's often the things of which you think that they can't break that finally break.
  


• Use a healthy mixture of tests on all abstraction levels.
  Unit and functional tests are orthogonal - they cover different aspects of the code. But of course you'll already have a lot of unit and functional tests if you don't rely on your tests too quickly.
  


• Buckle up!
  The unsafe feeling while trying to modify code without having a test was a very impressive experience for me. I know that from now on I'll fasten my code's seat belt.
  

Can You Remember Hungarian TLAs?

Scene 1. Karl and J.B. are pairing on a small web service. Karl is just returning to the workplace with a steaming, hot cup of coffee in his hand.

Karl: 'Let me see what you wrote just now...

usUserName = request.getParameter("UserName");

Um... This variable, usUserName, what does the us-prefix stand for?'
J.B.: 'Well, that is the unescaped user name the way we get it from the user. I wanted to make sure that we don't accidentally write it into a database or send it back in it's evil, unescaped form to the webbrowser. If we use the us-prefix every time we have an unsafe string, we'll immediately recognize any error that could otherwise escape us because we will learn to look for such errors. This is the application Hungarian notation I read about over at Joel's site, where you actually use a prefix that has a meaning instead of just a shorthand for the type.'
Karl: 'So... why not just call it unescapedUserName?'

Confusion

In a time where you enter veryLongVariableNames by typing 'v', 'e', 'r', Crtl-Space, I don't see why we can't finally get rid of TLAs. You know that the Hungarian notation got out of hand when your colleagues check in code that changes ucpBuffer to pucBuffer ("fixed a segfault"). Why not just name a variable for what it contains, in plain old English? I definitely know that I should think about my method name if my partner asks during a pairing session: "And what exactly do you intend to do in this method?".

In which example is the error easier to spot? Does the second example really take longer to write? To read? To understand?

for(unsigned int i = 0; i < iLineCount(); ++i)
{
  for(unsigned int j = 0; j < iNodeCount(i); ++i)
  {
    pGetNode(i, j)->layout();
  }
}

for(unsigned int lineIndex = 0;lineIndex < getLineCount(); ++lineIndex)
{
  for(unsigned int nodeIndex = 0; nodeIndex < getNodeCount(lineIndex); ++lineIndex)
  {
    getNode(lineIndex, nodeIndex)->layout();
  }
}

If you can really remember mnemonic prefix TLAs (or any TLAs for that matter) and think Hungarian notation or abbrVarNames are a great way to safe yourself some typing, please let me know.