Criteria for Model Evaluation

As discussed in the previous chapter, we contend that the most important criteria for evaluating reliability models are predictive validity, simplicity, and quality of assumptions, in that order of importance. With regard to quality management models, we propose that timeliness of quality indications , scope of coverage of the development process, and capability be the major criteria for evaluation.

The earlier a model can detect signs of quality problems or improvements, the more time is available for proactive planning. Furthermore, corrections in the early phases of the development process are much less expensive than those made at the back end. See Chapter 6 for discussion of the cost effectiveness of defect removal by development phase.

Model coverage of all phases of the development process is important. To have a good quality end product, quality in the intermediate deliverables at each phase, as defined by the phase entry and exit criteria, is a prerequisite. Each development phase must be managed and appropriate quality actions implemented. In case a single model cannot perform the tasks adequately, the use of multiple models is recommended.

While we contend that capability (other than predictability) is not an important criterion for software reliability models at the current state of the art, it is very important for management models. If "capability" refers to the model's ability to provide information for planning and managing software development projects, that is the purpose of quality management models. For instance, given the current in-process defect injection and removal pattern, will the project be likely to achieve its quality goal? If the effectiveness of the design review process improves by certain points, what is the possible impact on end-product quality? While quality management models may never reach the degree of accuracy and precision that reliability models have (or aim for), it is their capability to provide hints and findings for various in-process management questions that distinguish them as a separate category of software quality engineering model.