Friday, 22 April 2011

A Case For Test Driven Development

I've had to sell TDD into my last two consultancy roles. Here's the case I presented. Thankfully I was successful.

TDD is not Unit Testing
The purpose of TDD is not to reduce defects, but to encourage good low level design and provide fast, targeted feedback when things do not work as expected. This is not to say that you cannot write well designed or clean code without TDD, or that using TDD will guarantee that you do, just that since poorly designed code is difficult to unit test, if you start by writing the test you are goaded into keeping methods small, responsibilities separate, classes loosely coupled etc. This is just one way in which TDD differs from unit testing (which may be done after the ‘real’ code is written). Another is that if you adhere to TDD practice you are required to examine the code after writing a passing test, in order to make small improvements to it, such as factoring out duplication. Once again there is nothing to prevent you from doing this without TDD, however if you follow TDD this behaviour is expected and you are protected when doing so by a regression test suite.

Functional Testing is Insufficient
Where TDD and unit testing do overlap is in providing the benefit of fast, targeted feedback. If in modifying the code base you introduce a bug, it is more cost effective to find it immediately, before the context of the change has been forgotten or complicated by further modification. If you rely solely on functional tests then you are limited in three regards. Firstly, a functional test is less likely to pinpoint the origin of the defect in the same way a unit test can, typically making its diagnoses more time consuming. Secondly, a functional test takes longer to execute than a unit test. A defect can break multiple tests and may not be fixed on the first attempt. As a result the broken tests are re-run repeatedly so the faster they run the better. Finally it is dangerous to rely purely on functional tests, because it is difficult to fully exercise the internal workings of a system when you only have control over its extremities. A unit test by definition provides direct access to the components within a system, and is therefore better able to fully exercise the behaviour of those individual components. As a result unit tests can detect defects that may otherwise be missed by functional tests. The cost associated with missing a defect early is typically high, because once out of development the defect needs to be recorded, managed and fixed. Often the whole release is invalidated and has to be rebuilt, reinstalled and retested. Clearly the best time to find a defect is just after it has been introduced.

Regression Tests Enable Sustainable Pace
The next benefit of TDD is that by placing an emphasis on refactoring, and by providing the safety net of a fine-grained automated regression test suite you are encouraged to keep producing well written software. This is even more important in an environment with frequently changing requirements, since it is the safety net that allows you to make wide reaching changes to your code base while minimising the risk of introducing defects. Were it not for this safety net, developers may be tempted to reduce risk by minimising the changes required, even if this means compromising the design, i.e. hacking. Maintaining poorly designed code makes further changes more risky still, and begets yet more hacks. Over time the quality of the code base can degrade to a point where delivery is unacceptably slow and release candidates repeated fail formal testing. In contrast, TDD helps achieve a sustainable pace for the lifetime of the application.

TDD Encourages Clear Requirements
TDD does more than encourage good design and providing fast feedback however. With TDD you start by writing an assertion. This provides focus. If you cannot assert what you want to test, you cannot continue and have to seek clarification.

TDD Motivates Developers
Another benefit is that TDD is in harmony with the typical developer mindset. People who enjoy writing software are problem solvers. Good developers enjoy writing software, but don’t necessarily enjoy writing tests. By writing the assertion first and seeing it fail you turn a chore into a challenge, meaning motivation is intrinsic as opposed to extrinsic, which studies have shown yields better results for cognitive tasks.

Tests Document Actual Behaviour Rather than Expected Behaviour
Lastly TDD provides “living” documentation of what the system actually does. Requirements, design documents and even comments / javadoc can get out of date, but the tests communicate how the system really behaves. This can make it easier for a developer to maintain code they have little or no experience of. In recognition of this testing libraries are evolving to the point where it is now possible to write tests automated as English specifications (see spock, JBehave, concordion).

The Cost Of TDD
I have been using TDD in Agile teams for about six years. My first observation is that providing you write the test first, it does not take very much longer to write the test plus the real code, than to just write the real code. The reason for this is that modern IDEs offer automatic code completion – they will create the classes and methods for you from your test. My second observation is that test driven code involves less time debugging, so the overall time spent in development for a release is actually less than if the code had been written without the tests. More defects also tend to be found in development, rather than during formal testing, reducing the time for each release further. My third observation is that since TDD demands refactoring, and provides a mechanism for doing this safely, changes do not become exponentially more difficult (and therefore time consuming) over the lifetime of the application. Unfortunately there is little empirical evidence to support or refute my observations. The best I could find is this report, which compared the findings from 13 other studies. It concluded that:

“TDD seems to improve software quality, especially when employed in an industrial context. The findings were not so obvious in the semi-industrial or academic context, but none of those studies reported on decreased quality either. The productivity effects of TDD were not very obvious, and the results vary regardless of the context of the study. However, there were indications that TDD does not necessarily decrease the developer productivity or extend the project lead-times: In some cases, significant productivity improvements were achieved with TDD while only two out of thirteen studies reported on decreased productivity. However, in both of those studies the quality was improved.”

Critisms of TDD
The most common critism of TDD is that it takes a big investment. I disagree with this since I believe the cost of not doing TDD is significantly greater than the cost of doing it. Another is that TDD will involve a learning curve. I stuggle to comprehend this argument since using the same logic one would conclude that we shouldn't write simple code since this requires a learning curve too.

There are some other critisims that I do agree with however. TDD may be not be appropriate when prototyping or writing exploratory code, when what you are really trying to do is understand the problem space. After you do understand the problem space you either need to throw away the code and test drive from the start, or retro fit the tests. The former feels wasteful, the latter feels like a chore. I have also found TDD to be burdensome when fine tuning heuristic algorithms, e.g. the weightings used by a search engine to determine relevance. Test driving the code to call the search engine is easy, but creating a suitable set of test data and tweaking the weightings until the resutls are desirable is time consuming and usually takes several iterations.

My final critism of TDD is that it can require you to expose class variables that would otherwise remain private in order to assert state. Since the tests usually live in the same package as the classes under test this is often done by creating a package scoped accessor, and while undesirable is only a minor infringement.

So there you have it. My case for TDD. I'm sure there are benefits that I've missed. When the boat stops rocking I'm going to push for pair programming too.

3 comments:

Marcin said...

Stephen, thanks for this nice summary. I will certainly bookmark it for reference.
There is only one point I don't necessarily follow and would appreciate your further thoughts.
One of the rules I follow when writing TDD code is to never pollute any production code with test-related code so your last criticism of TDD doesn't relate for me. Contrary, when it feels like changing prod code for a test than it's a good indication design needs a change for me.

Stephen Cresswell said...
This comment has been removed by the author.
Stephen Cresswell said...


Thanks for the feedback. I agree that in principle it is undesirable to write production code solely to facilitate testing, however I think there are situations when it can be the lesser of all evils. e.g. in order to facilitate TDD when working with poorly designed third party APIs.

So I agree that it's a good indication design needs a change, its just that the design may not be fully under your control.

Post a Comment