Unit Testing Anti-Pattern #2: Not Covering Bugs In Your Tests With Tests
Roy just posted a unit testing anti-pattern based on his personal experience. I'd like to follow up with one from my own personal experience:
A few days ago, while working with one of our unit test suites, I discovered that one of my test utility classes had a bug. The bug didn't cause any tests to fail, but it caused a few tests to become false negatives - they didn't fail, but they actually didn't test what they were supposed to test either. Embarrassed, I corrected the test utility class (it was a very simple fix), all unit tests still succeeded, and now the tests that previously didn't test the correct behavior actually do that.
At that point, I happily checked in my changes and went along with my life.
Later, it occurred to me that I couldn't be certain that the fix I had implemented would stay that way. What if someone else comes along and changes the test utility class back to the way it was before? I know from experience that it isn't going to break any tests, although some tests do rely on specific behavior of the test utility class.
When I first wrote the test utility class, I thought it was so simple that it didn't need testing in itself. When I fixed it, it was still simple, but if I could make the mistake once, I could conceivably make it again, so I ought to protect myself against this sort of regression. As Gerard Meszaros writes in xUnit Test Patterns, you may definitely have a need for writing Test Utility Tests, and that was obviously the correct thing to do here, so I wrote a test that verifies that the test utility class behaves as expected, and I can now feel much more confident that my 'real' tests will not produce false negatives.
So, following Roy's pattern language, anti-pattern #2:
Forces: The bug is in your test code, so you just want to fix the test and get on with the 'real' feature you are working on.
Behavior:
You don't think test code needs testing, even when you just discovered a bug there. Instead of writing a test that reproduces (or exposes) the test bug, you just fix the test code and hope it's going to stay fixed forever.
How to avoid: Apply the same mindset to your test code as you do to your production code: If you discover a bug in the test code, write one or more unit tests that reproduces the bug, and then you can fix it. The ensures that no test regressions will occur duing a later test refactoring.
It would be interesting to read about other unit testing anti-patterns, so I'd hereby like to start a little relay race and pass the baton to Ayende, whom I much admire, in the vain hope that he will divert us with unit-testing anti-pattern #3, and subsequently pass on the baton to someone else...
