The Cleanroom Methodology

Cleanroom Software Engineering approaches software development as an engineering process with mathematical foundations rather than a trial-and-error programming process (Linger and Hausler, 1992). The Cleanroom process employs theory-based technologies such as box structure specification of user function and system object architecture, function- theoretic design and correctness verification, and statistical usage testing for quality certification. Cleanroom management is based on incremental development and certification of a pipeline of user-function increments that accumulate into the final product. Cleanroom operations are carried out by small, independent development and certification (test) teams, with teams of teams for large projects (Linger, 1993). Figure 2.5 shows the full implementation of the Cleanroom process (Linger, 1993).

Figure 2.5. The Cleanroom Process

From "Cleanroom Software Engineering for Zero-Defect Software," by R. C. Linger. Proceedings Fifteenth International Conference on Software Engineering , May 17 “21. 1993 IEEE. Reprinted with permission.

The Cleanroom process emphasizes the importance of the development team having intellectual control over the project. The bases of the process are proof of correctness (of design and code) and formal quality certification via statistical testing. Perhaps the most controversial aspect of Cleanroom is that team verification of correctness takes the place of individual unit testing. Once the code is developed, it is subject to statistical testing for quality assessment. Proponents argue that the intellectual control of a project afforded by team verification of correctness is the basis for prohibition of unit testing. This elimination also motivates tremendous determination by developers that the code they deliver for independent testing be error-free on first execution (Hausler and Trammell, 1993).

The Cleanroom process proclaims that statistical testing can replace coverage and path testing. In Cleanroom, all testing is based on anticipated customer usage. Test cases are designed to rehearse the more frequently used functions. Therefore, errors that are likely to cause frequent failures to the users are likely to be found first. In terms of measurement, software quality is certified in terms of mean time to failure (MTTF).

The Cleanroom process represents one of the formal approaches in software development that have begun to see application in industry. Other examples of formal approaches include the Vienna Development Method (VDM) and the Z notation (Smith and Wood, 1989; Wing, 1990). It appears that Z and VDM have been used primarily by developers in the United Kingdom and Europe; Cleanroom projects are conducted mostly in the United States.

Since the pilot projects in 1987 and 1988, a number of projects have been completed using the Cleanroom process. As reported by Linger (1993), the average defect rate in first-time execution was 2.9 defects per thousand lines of code (KLOC), which is significantly better than the industry average.

The adoption of Cleanroom thus far is mostly confined to small projects. Like other formal methods , the questions about its ability to be scaled up to large projects and the mathematical training required have been asked by many developers and project managers. Also, as discussed previously, the prohibition of unit testing is perhaps the most controversial concern. Whether statistical testing could completely replace range/limit testing and path testing remains a key question in many developers' minds. This is especially true when the software system is complex or when the system is a common-purpose system where a typical customer usage profile is itself in question. Not surprisingly, some Cleanroom projects do not preclude the traditional methods (such as unit test and limit test) while adopting Cleanroom's formal approaches. Hausler and Trammell (1993) even proposed a phased implementation approach in order to facilitate the acceptance of Cleanroom. The phased implementation framework includes three stages:

1. Introductory implementation involves the implementation of Cleanroom principles without the full formality of the methodology (e.g., box structure, statistical testing, and certification of reliability).
2. Full implementation involves the complete use of Cleanroom's formal methods (as illustrated in Figure 2.5).
3. Advanced implementation optimizes the process for the local environment (e.g., the use of an automated code generator, Markov modeling and analysis of system usage, and certification using a locally validated reliability model).

In their recent work, the Cleanroom experts elaborate in detail the development and certification process (Prowell et al., 1999). They also show that the Cleanroom software process is compatible with the Software Engineering Institute's capability maturity model (CMM).