Refactoring to Agility

5.5. Measuring the Effect of a Change

It is important to measure the effect of a process innovation for two reasons. First, any change we make can have a negative impact, even though we think that it will be a positive change. Before we decide to make the change permanent, we must ensure that its impact was positive. Second, selecting and making a change requires resources (i.e., impacts our efficiency), so we need to be sure that those resources are being used wisely. Quantifying the effects of a change verifies the quality of our improvement efforts and justifies continuing improvement activities.

Often the impact of a process innovation can be measured with the throughput and efficiency metrics, but in some cases, it is more insightful to define a metric specific to this change. In these cases, the index card from the Refactoring Planning Meeting translates almost directly into goals for the Goal/Question/ Metric (GQM) strategy discussed in Section 4.5.1. Recall that goals require four attributes: purpose, issue, object, and viewpoint. The purpose is generally "to improve." The card from the Refactoring Planning Meeting will provide the issue and usually the object. The only thing left to specify is the viewpoint from which this issue will be evaluated. Often, the viewpoint is the role played by the person who raised the issue. From there, you can use the GQM strategy to find metrics that will be insightful for this situation.

A single measurement cannot be used to measure the impact of a change. At the minimum, we need some measurement of the metric before the change and another measurement after the change to make a comparison. However, the conclusions that result from that analysis are very weak. All of these metrics have some inherent variability, so it will be difficult to know whether a change in the value of the metric is a random effect or a result of the process innovation being measured. Conclusions that are more robust depend on having data covering a longer period. Statistical process control can help with this analysis.