Cost Effectiveness of Phase Defect Removal

In addition to the defect removal effectiveness by phase per se, the cost of defect removal must be considered for efficient quality planning. Defect removal at earlier development phases is generally less expensive. The closer the defects are found relative to where and when they are injected, the less the removal and rework effort. Fagan (1976) contends that rework done at the I0, I1, and I2 inspection levels can be 10 to 100 times less expensive than if work done in the last half of the process (formal testing phases after code integration). According to Freedman and Weinberg (1982, 1984), in large systems, reviews can reduce the number of errors that reach the testing phases by a factor of 10, and such reductions cut testing costs, including review costs, by 50% to 80%. Remus (1983) studied the cost of defect removal during the three major life-cycle phases of design and code inspection, testing, and customer use (maintenance phase) based on data from IBM's Santa Teresa (California) Laboratory. He found the cost ratio for the three phases to be 1 to 20 to 82.

Based on sample data from IBM Rochester, we found the defect removal ratio for the three phases for the AS/400 similar to Remus's, at 1 to 13 to 92. Caution: These numbers may not be interpreted straightforwardly because defects that escaped to the later testing phases and to the field are more difficult to find. When we invest and improve the front end of the development process to prevent these more difficult defects from escaping to the testing phases and to the field, the ratios may decrease. Nonetheless, as long as the marginal costs of additional front-end defect removal remains less than testing and field maintenance, additional investment in the front end is warranted.

Our sample study also revealed interesting but understandable findings. The cost of defect removal is slightly higher for I0 inspection than for I1 and I2 (Figure 6.6). The main reason for this is that external interfaces are being impacted and more personnel are involved in the I0 inspection meetings. The cost for creating and answering a problem trouble report during testing (i.e., problem determination cost) is correlated with the testing phase, defect origin, and defect severity (1 being the most severe and 4 the least) (Figure 6.7).

Figure 6.6. Cost of Defect Removal by Inspection Phase

Figure 6.7. Cost of Creating and Answering a Problem Trouble Report by Several Variables

In his work on software inspection, Gilb (1993, 1999) conducted thorough analysis with ample data. The findings corroborate with those discussed here and support the general argument that software inspection not only improves the quality of the product, but also is beneficial to the economics of the project and the organization.

Although front-end defect removal activities in the form of reviews, walk-throughs , and inspections are less expensive than testing, in general practice, these methods are not rigorous enough. Fagan's inspection method is a combination of a formal review, an inspection, and a walkthrough. It consists of five steps:

1. overview (for communications and education)
2. preparation (for education)
3. inspection (to find errors and to walk through every line of code)
4. rework (to fix errors), and
5. follow-up (to ensure all fixes are applied correctly)

Such a combination has made Fagan's method somewhat more formal and therefore more effective than earlier methods. The Active Design Reviews method, introduced by Parnas and Weiss (1985), represents an important advance. The approach involves conducting several brief reviews rather than one large review, thereby avoiding many of the difficulties of conventional reviews. Knight and Myers (1991) proposed the phased inspection method to improve the rigor of the process. It consists of a series of coordinated partial inspections called phases (therefore, the term is used differently). Each phase is designed to achieve a desirable property in the product (e.g., portability, reusability, or maintainability), and the responsibilities of each inspector are specified and tracked.

Knight and Myers defined two types of phase. The first type, referred to as a single-inspector phase, is a rigidly formatted process driven by a list of unambiguous checks, for examples, internal documentation, source code layout, source code readability, programming practices, and local semantics. The second type of phase, designed to check for those properties of the software that cannot be captured in a precise yes or no statement (such as functionality and freedom from defects), is called the multi-inspector phase. Multiple personnel conduct independent examinations and then compare findings to reach reconciliation. To facilitate and enforce the process, the phased inspection method also involves use of an online computer tool. The tool contains navigation facilities for displaying the work product, documentation display facilities, facilities for the inspector to record comments, and facilities to enforce the inspection process.

Advances such as the preceding offer organizations much promise for improving the front-end defect removal effectiveness. Beyond reviews and inspections, one can even adopt formal methods such as the Cleanroom functional verification (as discussed in Chapter 2).