Linux Annoyances for Geeks: Getting the Most Flexible System in the World Just the Way You Want It

10.3 MONITORING AND CONTROL

The effectiveness of the review process depends on how well the process has been deployed. For example, if only two defects were found during the review of a 500-line program or a 20-page design document, clearly the review was not effective. The most common reason for a poor review is that it was not done with the proper focus and seriousness. Unless reviews are taken seriously, they will likely be a huge waste of time that does not give any due return, or they may be seen as a step to be checked off by performing it perfunctorily.

How does a project manager or a moderator evaluate whether a review has been effective so that she can decide the future course of action? One effective way of monitoring and controlling reviews is to use statistical process control (SPC) concepts implemented through control charts. Because the number of data points for reviews particularly for code reviews can be large, statistical techniques can be applied with confidence and rigor. This section discusses how Infosys monitors and controls reviews using statistical techniques.

10.3.1 The Review Capability Baseline

How can SPC be applied to monitoring reviews? To apply SPC, project managers must identify critical performance parameters, determine control limits for them, and then monitor the actual performance. They can build control charts by plotting the performance parameters of reviews and then use the plots to evaluate the effectiveness of a review. Another approach is to set the control limits for the various parameters and then use that range to determine the effectiveness. Although this latter approach has limitations because the run chart is not available, it is easy to apply. At Infosys, this latter approach is followed.

At Infosys, control limits have been determined for the following performance parameters: the coverage rate during preparation, the coverage rate during the group review meeting, the defect density for minor or cosmetic defects, and the defect density for serious or major defects (the overall defect density is simply the sum of the two preceding defect densities). These limits are determined from past data and from the review capability baseline. Creating and maintaining this baseline are important reasons for collecting summary data on reviews. Table 10.2 shows the group review capability baseline.

The group review baseline in Table 10.2 gives, for the various types of work products, the coverage rate during preparation, the coverage rate during review, and the defect density for minor and critical defects (the overall defect density is the sum of the two). The defect density is normalized with respect to size, where size is measured in the number of pages for all noncode work products and in lines of code for code work products. (For a design, size can also be measured in terms of the number of specification statements.) The coverage rate is stated in terms of size per unit effort, where effort is measured in person-hours. As you can see, for documents the coverage rates and defect densities are quite similar, but they are different for code (where the unit of size is also different).

The rates for one-person reviews can be expected to differ. Detailed design documents, test plans, and code undergo this form of review regularly. Hence, a one-person review baseline has been developed for these work products. In the baseline for one-person review of documents, the coverage rate per hour is about twice the corresponding coverage rate of group reviews; the defect detection rate per page is about half of that found with group reviews. For code, the coverage rate per hour is about the same, but the defect detection rate per LOC is about 30% less.

This baseline is the foundation for monitoring a review conducted in a project.

10.3.2 Analysis and Control Guidelines

The ranges given in the baseline are used to determine whether the performance of the review falls within acceptable limits. This check is specified as an exit criterion for the review process. Project managers can define the exit criteria as in-range checking of all the various parameters, but because detecting defects is the central purpose of reviews, the exit criterion is that the overall defect density should lie within the specified limits. (Another option is to check that the defect densities for the two types of defects are within the appropriate ranges.) If the number of defects found during the review is within the range given in the baseline, the review is considered effective, the exit criteria are satisfied, and no further action is needed for this review.

Instead of using the review capability baseline, a project manager can monitor reviews using an SPC tool developed in-house. This tool is essentially a spreadsheet that has the capability baseline data built into it. It also has data about the defect injection rate, defect removal efficiencies, organization-wide quality capability, and so on. From these data, the tool determines the performance specifications for a review that is, the range in which its outcome is expected to fall if the quality goal is to be met. The SPC tool gives warnings when a piece of review data falls outside either the control limits or the expected limits (defined shortly).

If the density of defects found in a review is not within the range given in the capability baseline, it does not automatically mean failure of the review. The project manager or the moderator critically evaluates the situation and decides on the next steps. The preparation rate and review rate become very useful here; if the review rate is "too fast" compared with that given in the baseline, the reason for the ineffectiveness of the review is relatively clear. The defect densities for minor and critical defects can also be useful in this analysis. Although the moderator or the project manager can use any technique to determine the cause of performance deviation and the corrective and preventive actions that should be taken, a set of guidelines, shown in Table 10.3, helps in this endeavor.

Table 10.3 includes two groups of guidelines: one set that is applicable when the defect density is below the range, and another set that is applicable when the defect density is above the range. Both cases suggest that something abnormal may have taken place, and the situation must be examined carefully. Table 10.3 lists some possible causes; the project leader or moderator can use this information to identify the cause and then decide on the corrective actions for this review and preventive actions for future reviews.

Static control limits work well when the process is operating in a steady state. However, if changes are consciously made to the process, control charts must be used with care because the process performance is expected to change. If changes in the process are likely to be regular, it is best to have dynamic control limits. One way to do this is to reset the control limits to new values based on the past n performance data points (n must be selected; it can be some number greater than about 10 to 15). With this approach, if the process is changed, its performance will change, and after a few data points, the control limits will reflect the performance capability of the changed process.

Another approach is to adjust the performance data or control limits based on the expected impact of the process changes. For example, if the review process is changed and if it is expected that the reviews will detect 10% more defects, then when a point falls outside the control limits, this fact should be taken into account during the analysis.

This latter approach is used at Infosys when defect prevention is employed during reviews. With DP, the defect injection rate is expected to decline. Consequently, the defect density detected in reviews is also likely to fall. If a project is using DP, then during project planning the expected impact of DP is also recorded. Using the defect injection rate, the expected reduction from using DP, and the defect detection rate, the expected limits for performance are established. If a review performance falls outside these limits, it is carefully examined for causes.

10.3.3 An Example

Consider the summary report for the group review of a project management plan given in Table 10.4. This summary covers a group review of a 14-page project management plan. The total number of minor and cosmetic defects found was 16, and the total number of major defects found was 3. Thus, the defect density found is 16/14 = 1.2 minor defects per page, and 3/14 = 0.2 major defects per page. Both rates are within the range given in the capability baseline, so the exit criteria are satisfied and it can be assumed that the review was conducted properly.

Although not needed for this review because the exit criteria are satisfied, other rates can also be checked. The review team had 4 members, each of whom spent 2.5 hours in individual review; the review meeting lasted 2.5 hours. Thus, the coverage rate during preparation and review was 14/2.5 = 5.6 pages per hour, which is within the range for preparation but somewhat higher for review.

When summary data is given to the SPC tool, it graphically displays the performance of this review, along with the control limits and expected limits.