TestDrivenDevelopment

Programming - Methodology

Coplien: Prophet In The Wilderness

Coplien has written a few essays warning about why TDD is not the panacea some think:

• Why Most Unit Testing is Waste
• Segue
• Is TDD Dead
• Interview: Jim Coplien and "Uncle" Bob Martin

Bob Martin: "nowadays it is irresponsible for a developer to ship a line of code he has not executed in a unit test." (2008)

Coplien does NOT say people should never use TDD. But he fears that over-emphasis on TDD will crowd out other potentially useful practices, such as QA, system testing, up-front architecture thinking. This is not to say that he thinks all TDD is antithetical to such practices.

One particularly relevant point Coplien makes is that the potential number of states of software is very large. Just because you are line-covered with your TDD tests is no guarantee of correct functioning.

TDD in Context

TDD is perhaps best put in context by considering it alongside other testing practices. Sometimes people talk about Four Schools Of Testing:

• Analytic: "objective", proof-theoretical, telecoms/safety-critical, academia
• Factory / Standard / Routine: managed process, enterprise IT/govt, IEEE standards/certifications
• Quality: QA and process improvement, large bureaucracies, CMM/ISO
• Context-Driven: exploratory/adaptive/psychological, commercial/market driven software, LAWST workshops

Where TDD then provides a fifth school:

• Agile / TDD: developer-supplied, consulting, pattern workshops

DHH and Recent Flaps

DHH of Ruby fame kicked the ant's nest more recently in April 2014: TDD is dead. Long live testing

Kent Beck, DHH, Martin Fowler had a follow-on discussion in May 2014: Is TDD Dead

Some interesting points:

• Kent Beck doesn't use TDD all the time
• If you don't understand the potential drawbacks of TDD you will not be able to make intelligent judgement calls on whether to use it
• TDD does not have to replace QA, but QA can't replace TDD: also old QA relationships were unhealthy, devs taking some testing responsibility is better than that
• TDD is not enough by itself to give you confidence everything works. Exploratory / system testing is also needed.
• People should be willing to throw away tests that are no longer needed
• Conversely, sometimes you can throw away code and re-implement from the tests: being able to do this can be of great value
• It's good to learn the discipline of test-first, it's like a 4WD-low gear for tricky parts of development.

1: TDD and Confidence:

• Kent doesn't say use TDD everywhere. In a recent hackathon about half of what he was doing wasn't TDD friendly.
• DHH has been in situations where TDD works - but the question is what's being traded off when you use TDD? Heavy mocking etc can be a bad trade-off.
• Kent agrees it's about tradeoffs. Sometimes (e.g. compiler) an intermediate parse-tree is a good test point and good design. But Kent only uses mocks rarely.

2: Test-induced design damage:

• Kent: something hard to test is often an indication of a needed design insight - a good trigger to get up, take a walk, think
• DHH: yes this can happen but too often his experience was the opposite, that there wasn't a good testable design
• Kent: DHH should have more self-confidence, yes proceed for now but eventually you'll get the better idea
• DHH: no that's "faith-based TDD" - he used to feel this but got stuck in a depressing loop when he wasn't finding an ideal solution that wasn't there.
• Kent: clarified he wasn't talking about TDD, but about software design in general, it's not about TDD it's about how to get feedback.

3: Feedback and QA:

• Kent: old relationship with QA was dysfunctional... Facebook didn't have QA until recently and programmers live up to that responsibility.
• Kent: "It's a question of 'compared to what?'" Compared to having an effective QA then no-QA is worse, but no-QA is better than the old dysfunctional relationship.
• MF: at ThoughtWorks we almost always have QA on our projects. I also feel that the big shift from the 90's is not just getting rid of the dysfunctional adversarial relationship, but also getting rid of manual scripted tests.
• MF: And it's liberating that startups can operate without QA.
• DHH: agreed that it was good to mindfully trade-off QA for initial speed, but some have taken programmer testing too far and don't see the value of exploratory testing.
• DHH: If developers think they can create high-enough quality software without QA they are wrong, your tests may be green but when it's in production users do things you don't expect.

4: Costs of Testing:

• DHH: to talk about trade-offs, you really have to understand the drawbacks, because if there are no drawbacks there are no trade-offs. TDD doesn't force you to do things, but it does nudge you in certain directions.
• DHH: Over-testing can be bad: when you need to change behavior, you have more code to change
• DHH: Kent has said 'you aren't paid to write tests, you just write enough to be confident' - so he asked if Kent and I wrote tests before every line of production code?
• Kent: "it depends, and that's going to be the beginning to all of my answers to any question that's interesting".
• Kent: With JUnit they were very strict about test-first and were very happy with how it turned out
• Kent: Herb Derby came up with the notion of delta coverage - what coverage does this test provide that's unique? Tests with zero delta coverage should be deleted unless they provide some kind of communication purpose.
• Kent: He said he'd often write a system-y test, write some code to implement it, refactor a bit, and end up throwing away the initial test.
• Kent: Many people freak out at throwing away tests, but you should if they don't buy you anything. If the same thing is tested multiple ways, that's coupling, and coupling costs.
• MF: I'm sure there is over-tested code, indeed if anyone does it would be ThoughtWorks since we have a strong testing culture.
• MF: It's hard to get the amount just right, sometimes you'll overshoot and sometimes undershoot. I would expect to overshoot from time to time and it's not something to worry about unless it's too large.
• MF: On the test-every-line-of-code point I ask the question: "if I screw up this line of code is a test going to fail?" I sometimes deliberately comment a line out or reverse a conditional and run the tests to ensure one fails.
• MF: My other mental test (from Kent) is only test things that can possibly break. I assume libraries work (unless they are really wonky). I ask if I can mess up my use of the library and how critical are the consequences of the mistake.
• Kent: declared that the ratio of lines of test code to lines of production code was a bogus metric. A formative experience for him was watching Christopher Glaeser write a compiler, he had 4 lines of test code for every line of compiler code - but this is because compilers have lots of coupling. A simpler system would have a much smaller ratio. David said that that to detect commenting out a line of code implies 100% test coverage. Thinking about what can break is worth exploring, Rails's declarative statements don't lead to enough breakage to be worth testing, so he's comfortable with significantly less than 100% coverage.
• MF: "you don't have enough tests (or good enough tests) if you can't confidently change the code," and "the sign of too much is whenever you change the code you think you expend more effort changing the tests than changing the code." You want to be in the Goldilocks zone, but that comes with experience of knowing what mistakes you and your team tend to make and which ones don't cause a problem. I said I like the "can I comment out a line of code" approach when I'm unsure of my ground, it's a starting place but as I work more in an environment I can come up with better heuristics.
• DHH: felt that this tuning is different between product teams that are stable rather than consulting teams that are handing the code over to an unknown team and thus need more tests.
• Kent: said that it's good to learn the discipline of test-first, it's like a 4WD-low gear for tricky parts of development.
• DHH: introduced the next issue: many people used to think that documentation was more important than code. Now he's concerned that people think tests are more important than functional code. Connected with this is an under-emphasis on the refactor part of the TDD cycle. All this leads to insufficient energy to refactoring and keeping the code clear.
• Kent: described that he just went through an episode where he threw away some production code, but keeping the tests and reimplementing it. He really likes that approach as the tests tell him if the new code is working. This leads to an interesting question: would you rather throw away the code and keep the tests or vice-versa? In different situations you'd answer that question differently.
• MF: I found situations where reading the tests helped me understand what the code was doing. I didn't think one was more important than the other - the whole point is the double check where there is an error if they get a mismatch. I agreed with David that I'd sometimes sensed teams making the bad move of putting more energy into the testing environment than in supporting the user, tests should be means to the end. I find I get a dopamine shot when I clarify code, but my biggest thrill is when I have to add a feature, think it will be tricky, but it turns out easy. That happens due to clean code, but there is a distance between cleaning the code and getting the dopamine shot.
• Kent: showed a metaphor for this from Jeff Eastman, that is too tricky to describe in text. He got his rush from big design simplifications. He feels that it's easy to explain the value of a new test working, but hard to state the value of cleaning the design.
• DHH: we often focus on things we can quantify, but you can't reduce design quality to a number - so people prioritize things that are low on the list like test speed, coverage, and ratios. These things are honey traps, and we need to be aware of their siren calls. Cucumber really gets his goat - glorification of a testing environment rather than production code. Only useful in the largely imaginary sweetspot of writing tests with non-technical stakeholders. It used to be important to sell TDD, but now it's conquered all, we need to explore its drawbacks.
• MF: disagreed that TDD was dominant, hearing many places where it's yet to gain traction.