Design for Trustworthy Software: Tools, Techniques, and Methodology of Developing Robust Software

Numerous factors cause defects and determine product quality. Hinckley discusses a study to determine which elements of design have the greatest influence on product quality.^[3] The study showed that defect rates were strongly linked to assembly time and the number of assembly operations over a wide range of industries. This led to the conclusion that the major source of defects could not be variation, the primary focus of quality improvement initiatives. Some 60 to 90% of all defects are created in productionand a similar percentage of defects escape to customers due to mistakes, either human or technological (mechanical) in origin.^[4]

An often-overlooked reality is that a large number of organizations relying merely on variation-focused initiatives such as Six Sigma will fail to achieve the Six Sigma quality level unless they also find effective ways to address mistakes and complexities, the "mother of all defects." When seeking an extremely low rate of nonconformities, it is essential to eliminate mistakes in addition to controlling variation. Mistakes become even more crucial as organizations strive to achieve defects (nonconformities) lower than 1,000 defects per million opportunities (DPMO), which corresponds to 4.59σ. Both Six Sigma and Statistical Quality Control (SQC) are variation-focused and as such do not address mistake preventiona major, if not the major, source of defects in most processes that have already attained high level of capability. Mistakes are discrete and probabilistic compared to variations, which are random. They cannot be measured by distribution models that describe process variation. Part-to-part variations in component properties and dimensions have been historically conceived as the major cause of defects. Consequently, SQC has been promoted as an adequate and absolute quality control system. The grounds for SQC's inherent limitations as a guide to quality control are well documented and can be summarized as follows:^[5]

• Small sample sizes often fail to detect defects.

• A small number of readings obscures the skew of distribution.

• "Oddball" and "outlier" observations are often discarded.

• Traditional inspection methods are not perfectly reliable.

• Assumed distribution models may be invalid. (They may ignore the Pareto principle and the probability and impacts of the extremely rarethe "vital few.")

• SPC cannot deliver the highest levels of quality (lower than 50 DPMO, which corresponds to 5.39σ) by itself.

• Because the control mechanisms are downstream, many defects may still occur.

Both Juran and Gryna^[6] and Hinckley and Barkan^[7] state the inadequacy of the normal distribution. A Hinckley and Barkan study reveals that significant errors do occur in assessing the extreme limits of the (normal) distribution. As such, the normal distribution is of little help in predicting distributions 3σ beyond the mean.^[7] SQC has yet another limitation when applied to software development: the volume of data may often be small for any meaningful statistical analysis. As such, Six Sigma, Taguchi Methods, and other statistics-based methodologies must be supplemented. This helps with not only monitoring, control, and elimination of defects caused by variation, but also mistakes and complexities in manufacturing and, even more so, in software development. To summarize, trustworthy software must be robust vis-à-vis all three causes of defects:

• Variation: Statistical methods are commonly used to measure variations. These are discussed in Chapters 6 and 15. But as Hinckley and Barkan have stated, understanding rare events (particularly those beyond the three standard deviations from the mean) and the limitations of conventional statistical methods is particularly important when the goal is to achieve extremely low nonconformity (below 10 DPMO, which corresponds to 5.76σ).^[8] Variation-based defects (nonconformities) are discussed in Chapters 15 through 19. We will not discuss this topic in any depth in this chapter. Instead, we will focus on mistakes and complexity.

• Mistakes are one of the most common causes of defects in software development. In terms of severity, their impact is second only to defects caused by complexity. Mistakes are different from variations in that they are not characterized by the physical properties or dimensions of any attribute. They are conditions of being in one state or another. Mistakes and mistake-proofing (poka yoke) are discussed later in the book.

• Complexity is a major source of all kinds of defects, including variation- and mistake-based. The more complex a product, service, software, or system, the more opportunities it has for mistake and variation nonconformities. Complexity thus is the root cause of many nonconformities attributed to mistakes and variations. In a DFTS context, reducing both product- and process-related complexities should be an essential element of an overall defect-reduction strategy.

One important issue that comes out of the work of Hinckley and Barkan is identifying the tools that describe and manage these distinct sources of nonconformities. Variation is managed with SPC and statistics. Mistakes are tackled using poka yoke. Complexity is controlled during the design process.^[8]

Even processes with high process capability need poka yoke. Human errors, although dependent on complexity, are independent of variance-based nonconformities. Where software nonfunction can be catastrophic, poka yoke is an indispensable tool. This chapter discusses mistakes and complexities; variation is presented in Chapter 15. We first discuss poka yoke systems, beginning with where poka yoke can be used effectively as a mistake-proofing system.