Measuring Process Improvements at Activity Levels

For measuring process improvements over time, gross measures of entire projects are not granular enough to be effective. We have learned that it is necessary to get down to the level of specific activities in order for process improvements to become visible and measurable.

Activity-based costs can highlight the activities that process improvements benefit, and also illustrate activities where no tangible benefits are noted. This kind of analysis is not be possible using only project-level data. Software benchmarking companies have found that function point metrics are superior to the older LOC metrics for measuring activity-based costs and schedules. The reason is that function points can measure noncoding activities such as requirements, design, documentation, and management.

We use the function point metrics defined by IFPUG. The current standard for function points published by IFPUG is version 4.1, which is used here. For a general introduction to the topic of function point analysis, refer to Dreger (1989) or to Garmus and Herron (1995).

Note that while IFPUG function points are the most widely used metric in the United States and Western Europe, other forms of function point metric are in use too. For example, in the United Kingdom a function point metric called Mark II function points are very common. For a discussion of the Mark II function points, refer to Symons (1991).

Table 18.4 illustrates a hypothetical project of 1,000 function points ( roughly 125,000 C source statements). The organization producing this application would be a typical civilian organization at CMM level 1. Level 1 organizations are not very sophisticated in software development approaches. Thus, level 1 organizations often have missed schedules, cost overruns, and poor-quality products.

The most obvious characteristic of CMM level 1 organizations is that testing is both the most expensive and the most time-consuming activity. The root cause of this phenomenon is that CMM level 1 organizations usually have excessive defect levels and are deficient in two key quality factors: defect prevention and pretest reviews and inspections.

By contrast, Table 18.5 illustrates exactly the same size and kind of software project. However, Table 18.5 shows the results that have been noted in somewhat more mature level 3 organizations on the Software Engineering Institute capability maturity model scale. For the level 3 organization, testing has diminished significantly in terms of both timing and costs. This is because defect prevention methods have improved and pretest design reviews and code inspections have both been strengthened .

Table 18.4. Example of Activity-Based Cost Analysis for SEI CMM Level 1

[a.] FP = function points per IFPUG Counting Practices Manual 4.1.

[b.] LOC = lines of noncommentary code.

[c.] Burdened = basic compensation and overhead costs.

[d.] Staff = number of workers assigned to major activities. Decimal values indicate some part-time personnel.

[e.] Person month = 22 workdays of 8 hours each.

[f.] Calendar months.

As Table 18.6 shows, a side-by-side analysis of the costs indicates an overall reduction of about 20% between the level 1 and level 3 versions, but the activity costs illustrate that most of the savings occur during the testing phase.

Indeed, the costs of inspections are higher, rather than lower, for the CMM level 3 version. Also, some costs such as those for user documentation are the same in both scenarios. Activity-based cost analysis allows a detailed scrutiny of differences between processes, and opens up a way to perform rather sophisticated cost and schedule models which cannot be done using coarser project-level data.

Table 18.5. Example of Activity-Based Cost Analysis for SEI CMM Level 3

[a.] FP = function points per IFPUG Counting Practices Manual 4.1.

[b.] LOC = lines of noncommentary code.

If we turn now to quality, the results of improving both defect prevention and defect removal approaches in the level 3 example causes a very significant reduction in delivered defects. In turn , the reduced numbers of defects allow shorter and more cost-effective development cycles. When projects run late, it often happens that problems escape notice until testing begins. When major defects are found during testing, it is too late to bring the project back under control. The goal is to prevent defects or eliminate them before testing gets under way.

Table 18.6. Side-by-Side Comparison of Activity-Based Costs

[a.] FP = function points per IFPUG Counting Practices Manual 4.1.

[b.] LOC = lines of noncommentary code.

[c.] Burdened = basic compensation and overhead costs.

Table 18.7 illustrates the differences in defect potentials, defect removal efficiency levels, and delivered defects of the two cases. As can be seen, a combination of defect prevention and defect removal can yield significant reductions in delivered defect levels. Because finding and fixing defects is the most costly and time-consuming activity for software, projects that are successful in preventing defects or removing them via inspections will achieve shorter schedules and higher productivity as well as better quality.

If software process improvement is to become a mainstream technology it is important to demonstrate exactly what is being improved, and by how much.

Activity-based cost analysis illustrates that process improvement does not create a homogenous improvement of every activity equally. Improvements tend to be very significant for key activities such as testing but scarcely visible for other activities such as user documentation.

Table 18.7. SEI CMM Level 1 and Level 3 Defect Differences

[a.] Effects likely to be encountered from the start of requirements through at least one year of customer use.

[b.] Percentage of potential defects found before delivery of the software to customers.

[c.] Function points per IFPUG Counting Practices Manual 4.1.

[d.] KLOC = one thousand lines of code.

Indeed, for a number of important activities such as design and code inspections and quality assurance work, the costs will be higher for more mature organizations at level 3 on the CMM than for those at level 1 on the CMM.