Recently I held a presentation at the Budapest Agile Meetup event with the title “Agile and Lean projects at very large scale at SAP Labs Hungary”, and we’ve had very lively discussions both during the meetup and later online.
The biggest reason for having such heated discussions was that I quoted the recent conference talk and blog post of David Heinemeier Hansson (or DHH, inventor and developer of Ruby on Rails web app development framework) titled “TDD is Dead. Long Live Testing.” Of course, proponents of Test-Driven Development didn’t feel good about this and brought up arguments.
The most notable one was a reference to Rod Hilton’s MSc in Software Engineering thesis from 2009 titled “Quantitatively Evaluating Test-Driven Development by Applying Object-Oriented Quality Metrics to Open Source Projects“. This is a really great scientific work, summarizing the topic very well.
During my meetup talk I happened to mention that I was not aware of any precisely calculated scientific result of anyone that would undoubtedly prove that TDD (or more broadly speaking, high unit test coverage, or even more broadly speaking, high test automation coverage) makes economic sense, meaning that it would have higher business benefits than costs in the long term (and in long term I mean max. 3 years, we live in a fast-paced world, don’t we). Of course I believe this long term benefit exists, the broader we talk about automated testing, the clearer this is (especially in the context of cloud apps, Continuous Delivery and DevOps, having self-testing code is a must).
However, this statement about benefits vs ROI was somewhat misunderstood. The referred thesis concludes that TDD produces 21% “better” code overall. I have every reason to accept that this is true. However, there was absolutely no measurement done on how much bigger the costs of achieving these better results were. Even more, the costs of more unit testing are immediate, while the benefits appear only longer term, and it’s probably needless to say that business management people tend to have a rather short-term focused thinking.
Probably not much of a coincidence is that DHH’s claims generated high debate in agile circles globally. He has had a series of live Google Hangouts on this topic with Kent Beck (creator of eXtreme Programming and TDD, and developer of the jUnit framework) and Martin Fowler (another XP celebrity and the biggest advocate of refactoring). I recommend everyone watching it. Here you can find the link of the 3 episodes so far, each one lasting only about 30 minutes:
- https://plus.google.com/events/ci2g23mk0lh9too9bgbp3rbut0k
- https://plus.google.com/events/couv627cogaue9dahj08feoa6b8
- https://www.youtube.com/watch?v=YNw4baDz6WA
Martin Fowler has also created a nice summary page on the whole topic: http://martinfowler.com/articles/is-tdd-dead/.
Some random excerpts from these discussions and my thoughts:
- Too much mocking is expensive and inflexible (isolated vs not isolated unit testing)
- Various personal styles: upfront planning of output (unit test expected result) or just coding through the problem
- Self testing code in general is much more important than TDD in particular
- More code == bad code – true for too long code due to too many levels of indirection and isolation of layers forced by unit testing and mocking. I have a personal opinion on this: it may be true in the TDD context, but there is also an opposite extreme: nowadays functional programming and other fancy new language constructs often lead to such forms of compact code that is extremely hard to maintain. Sometimes even the author doesn’t understand his ultra-sophisticated lambda-heavy code just 2 weeks after writing it. Not even speaking of junior developers or if you happen to be asked to shift code ownership to an offshore team. Maintainability is very critical, this is what I call software crisis 2.0, it is exponentially harder and harder to find and hire enough superman-programmers. So both too verbose and too compact coding is dangerous.
- Impact of unit testing on design – good or bad?
- Role of QA: Does having self-testing code mean we don’t need testers anymore? Probably not.
Summary: self-testing code is increasingly a must (fact), having lots of unit tests is great (belief) and using TDD is good in some cases (an option).
Follow me on twitter for future discussions on this topic.