Test Compression Factor

As the example and other cases in the literature illustrate (for example, Misra, 1983; Putnam and Myers, 1992), a fair degree of accuracy to project the remaining number of defects can be achieved by software reliability models, based on testing data. This approach works especially well if the project is large, where defect arrivals tend not to fluctuate much; if the system is not for safety-critical missions; and if environment-specific factors are taken into account when choosing a model. For safety-critical systems, the requirements for reliability and, therefore, for reliability models, are much more stringent.

Even though the projection of the total number of defects (or defect rates) may be reasonably accurate, it does not mean that one can extend the model density curve from the testing phase to the maintenance phase (customer usage) directly. The defect arrival patterns of the two phases may be quite different, especially for commercial projects. During testing the sole purpose is to find and remove defects; test cases are maximized for defect detection, therefore, the number of defect arrivals during testing is usually higher. In contrast, in customers' applications it takes time to encounter defects ”when the applications hit usual scenarios. Therefore, defect arrivals may tend to spread. Such a difference between testing-defect density and field-defect density is called the compression factor . The value of the compression factor varies, depending on the testing environments and the customer usage profiles. It is expected to be larger when the test strategy is based on partition and limit testing and smaller for random testing or customer-environment testing. In the assessment by Elbert and associates (1992) on three large-scale commercial projects, the com-pression factor was 5 for two projects and 30 for a third. For projects that have extensive customer beta tests, models based on the beta test data may be able to extrapolate to the field use phase.

Figure 8.4 shows a real-life example of compression of defect density between testing and initial data from the field. For the upper panel, the upper curve represents the extrapolated cumulative defect rate based on testing data. The lower curve is the actual cumulative field defect rate. Although the points of the two curves at four years in the life of product are close, the upper curve has a much faster buildup rate. The difference is even more drastic in the lower panel, in which the two defect density curves are contrasted. The extrapolated curve based on testing data is front loaded and declines much faster. In vivid contrast, the actual field defect arrival is much more spread out. It even follows a different density pattern (a delayed S or a Rayleigh-like pattern).

Figure 8.4. Compression Illustration ”Cumulative and Density Curves