Today The Test Suite Broke

When I arrived at work today I fired up Outlook and checked my mail. I found five mails from our auto-build server telling me that the build broke. Since we introduced test driven development and continuous integration only a short time ago this was not out of nowhere - the build usually breaks at least once a day.

But today was the first day a unit test broke since we introduced TDD and CI.


At first I thought that perhaps somebody checked in a broken test yesterday evening. But then I dug into the whole thing and found out that yesterday the build was perfect. Somewhat bewildered I got myself a nice cup of hot, steaming coffee, started Visual Studio and tried to find the reason for the failed test. Since there was some cryptography involved and I didn't know the ins and outs of this particular part of the system I asked the pair that wrote the code for help and we began a thorough debugging session.

As you can probably already imagine it was a date problem. A part of the systems cryptography was dependent on the system time and one of the algorithms broke today. We extracted a small mock-up that enabled us to simulate an arbitrary date as system date for the algorithm and found that this particular error would show up on two of 500 days since January the 1st.

So today was a 1:250 chance day.

What would have happened if this part of the code had no unit test. Well, most likely at some day in the future there would have been a bug report stating that the key that was entered by the service technician wasn't accepted. Since at that point the same procedure would have worked a thousand times we'd probably blamed the poor technician and told ourselves that this "can't possibly happen".