Extreme Programming Perspectives

Test-First and Test Automation in Extreme Programming

The testing practice is a key component of XP. Not only is it one of the 12 core practices, but it also plays a vital role in at least five others. Continuous integration, refactoring, and collective code ownership would all be downright foolhardy in the absence of automated unit tests to validate code changes. And without those supporting practices, simple design and small releases would be impractical as well.

Two of the main ingredients of the XP testing practice are test automation and test-first programming, in which programmer pairs write unit tests first before they write the code. Not only does this ensure that unit tests exist to detect any defects in the code, it also focuses the programmers on potential defects immediately before coding, presumably eliminating many from coming into being at all.

This combination of test-first programming and unit test automation, coupled with the requirement that 100% of the unit tests pass before code goes into the repository, can practically eradicate unit- and integration-level defects from XP-developed software.

User-Apparent Problems and Unit- and Integration-Level Defects

What may be surprising is that even when the unit tests are comprehensive, automated, and pass 100%, the system may still be incorrect in the eyes of the customer. This is hard to appreciate in traditional software development, because the unit testing is so haphazard that the systems are literally teeming with low-level bugs. Examination of user-apparent problems in this environment almost always implicates unit- and integration-type bugs. But even when these low-level defects are eradicated, as is possible with XP, the user-apparent problems still remain. There may even appear to be more, because the real gaps between what the customer wants and what the system does are no longer masked by nonfunctional areas of the system.

With the goal of banishing the user-apparent defects in the same manner as unit and integration bugs, we asked the question, "Why not adapt the test-first pattern for acceptance testing, and write functional tests at the very beginning, even before writing unit tests? And why not then automate those functional tests, just like the unit tests, and require all code to pass 100% of them before going into the repository and after each integration, just like unit tests?"

Differences between Functional Tests and Unit Tests That Affect Test-First

We have looked at this question during the last couple of years while working on XP projects for several companies, and found that unit and functional tests differ in some important ways that affect how a test-first approach can be carried out.

Unit tests are written in the same language as the code they test. They interact with the programming objects directly, often in the same address space. A pair can switch seamlessly between writing tests, writing code, refactoring, and running tests, without even getting out of the editor. And the unit tests correspond almost exactly to the programming objects. It is perfectly natural for a pair of programmers to write the tests for the object they are about to code, and then make sure they all run 100% when they are done. And from then on those tests are invaluable in validating that the code is still working after a change.

But functional tests almost always run completely separately from the system that they are testing and require specialized tools, and often different languages, that are designed to specify and recognize behavior visible to a user instead of to another programming object. And functional tests correspond to the programming objects only very loosely. A single functional test almost always relies on many different programming objects, and a single programming object often affects the outcome of several unrelated functional tests.

It is not possible to switch easily between functional tests and coding the way it is with unit tests, nor is it desirable, because the functional tests will not run until all of the required code is ready, even when the piece the pair is working on is completed. For the same reason, the functional tests cannot be expected to pass 100% in refactoring or at the end of an integration.

Adapting Test-First to Functional and Acceptance Testing

Because functional tests do not fit naturally into programmers' workflows, do not help programmers focus on the details of the code about to be written, and cannot be relied on to immediately find defects as the code is being written or during refactoring and integration, we found it to be difficult for programmers to write and automate these tests before writing the code. It would be easier if the functional test automation were trivial, but we found that it often required almost the same level of conceptual, design, and programming skill as developing the system.

For this reason, we have found it best to have a team member dedicated to or at the very least, focused on functional and acceptance test automation. This team member pairs with the other programmers as well as the customer to systematically transform the user stories into executable functional tests during release and iteration planning, and then automates and runs the tests during the iteration proper. The high degree of interaction between the test automation programmer and other team members provides the benefit of focusing attention on potential user-apparent defects before their introduction, just as with unit testing. And the automation of functional and acceptance tests provides similar benefits near the end of an iteration as the required code is completed and then, especially, in subsequent iterations.

The current scope precludes further details of the method we use for transforming user stories into tests or how that is integrated with the other XP activities. We describe briefly instead the way we have been actually automating functional and acceptance tests for Web applications. Additional details, as well as information on the other topics, can be found in the upcoming book from Addison-Wesley tentatively titled Testing for Extreme Programming, or on our Web site at http://www.xptester.org.