Defect Taxonomies

'Failure' was simply not a word that would ever cross the lips of Miss Evelyn Duberry, mainly because Evelyn, a haughty socialite with fire-red hair and a coltish gate, could pronounce neither the letters 'f' nor 'r' as a result of an unfortunate kissing gesture made many years earlier toward her beloved childhood parrot, Snippy.

— David Kenyon

Introduction

What is a taxonomy? A taxonomy is a classification of things into ordered groups or categories that indicate natural, hierarchical relationships. The word taxonomy is derived from two Greek roots: "taxis" meaning arrangement and "onoma" meaning name. Taxonomies not only facilitate the orderly storage of information, they facilitate its retrieval and the discovery of new ideas. Taxonomies help you:

• Guide your testing by generating ideas for test design
• Audit your test plans to determine the coverage your test cases are providing
• Understand your defects, their types and severities
• Understand the process you currently use to produce those defects (Always remember, your current process is finely tuned to create the defects you're creating)
• Improve your development process
• Improve your testing process
• Train new testers regarding important areas that deserve testing
• Explain to management the complexities of software testing

In his book Testing Object-Oriented Systems, Robert Binder describes a "fault model" as a list of typical defects that occur in systems. Another phrase to describe such a list is a defect taxonomy. Binder then describes two approaches to testing. The first uses a "non-specific fault model." In other words, no defect taxonomy is used. Using this approach, the requirements and specifications guide the creation of all of our test cases. The second approach uses a "specific fault model." In this approach, a taxonomy of defects guides the creation of test cases. In other words, we create test cases to discover faults like the ones we have experienced before. We will consider two levels of taxonomies—project level and software defect level. Of most importance in test design are the software defect taxonomies. But it would be foolish to begin test design before evaluating the risks associated with both the product and its development process.

Note that none of the taxonomies presented below are complete. Each could be expanded. Each is subjective based on the experience of those who created the taxonomies.

Project Level Taxonomies

SEI Risk Identification Taxonomy

The Software Engineering Institute has published a "Taxonomy-Based Risk Identification" that can be used to identify, classify, and evaluate different risk factors found in the development of software systems.

If, as a tester, you had concerns with some of these elements and attributes, you would want to stress certain types of testing. For example:

ISO 9126 Quality Characteristics Taxonomy

The ISO 9126 Standard "Software Product Evaluation—Quality Characteristics and Guidelines" focuses on measuring the quality of software systems. This international standard defines software product quality in terms of six major characteristics and twenty-one subcharacteristics and defines a process to evaluate each of these. This taxonomy of quality attributes is:

Each of these characteristics and subcharacteristics suggest areas of risk and thus areas for which tests might be created. An evaluation of the importance of these characteristics should be undertaken first so that the appropriate level of testing is performed. A similar "if you are concerned about / you might want to emphasize" process could be used based on the ISO 9126 taxonomy.

These project level taxonomies can be used to guide our testing at a strategic level. For help in software test design we use software defect taxonomies.

Software Defect Taxonomies

In software test design we are primarily concerned with taxonomies of defects, ordered lists of common defects we expect to encounter in our testing.

Beizer s Taxonomy

One of the first defect taxonomies was defined by Boris Beizer in Software Testing Techniques. It defines a four-level classification of software defects. The top two levels are shown here.

Even considering only the top two levels, it is quite extensive. All four levels of the taxonomy constitute a fine-grained framework with which to categorize defects.

At the outset, a defect taxonomy acts as a checklist, reminding the tester so that no defect types are forgotten. Later, the taxonomy can be used as a framework to record defect data. Subsequent analysis of this data can help an organization understand the types of defects it creates, how many (in terms of raw numbers and percentages), and how and why these defects occur. Then, when faced with too many things to test and not enough time, you will have data that enables you to make risk-based, rather than random, test design decisions. In addition to taxonomies that suggest the types of defects that may occur, always evaluate the impact on the customer and ultimately on your organization if they do occur. Defects that have low impact may not be worth tracking down and repairing.

Kaner, Falk, and Nguyen s Taxonomy

The book Testing Computer Software contains a detailed taxonomy consisting of over 400 types of defects. Only a few excerpts from this taxonomy are listed here.

Binder s Object Oriented Taxonomy

Robert Binder notes that many defects in the object-oriented (OO) paradigm are problems using encapsulation, inheritance, polymorphism, message sequencing, and state-transitions. This is to be expected for two reasons. First, these are cornerstone concepts in OO. They form the basis of the paradigm and thus will be used extensively. Second, these basic concepts are very different from the procedural paradigm. Designers and programmers new to OO would be expected to find them foreign ideas. A small portion of Binder's OO taxonomy is given here to give you a sense of its contents:

Note how this taxonomy could be used to guide both inspections and test case design. Binder also references specific defect taxonomies for C++, Java, and Smalltalk.

Whittaker s How to Break Software Taxonomy

James Whittaker's book How to Break Software is a tester's delight. Proponents of exploratory testing exhort us to "explore." Whittaker tells us specifically "where to explore." Not only does he identify areas in which faults tend to occur, he defines specific testing attacks to locate these faults. Only a small portion of his taxonomy is presented:

Vijayaraghavan s eCommerce Taxonomy

Beizer's, Kaner's, and Whittaker's taxonomies catalog defects that can occur in any system. Binder's focuses on common defects in object-oriented systems. Giri Vijayaraghavan has chosen a much narrower focus—the eCommerce shopping cart. Using this familiar metaphor, an eCommerce Web site keeps track of the state of a user while shopping. Vijayaraghavan has investigated the many ways shopping carts can fail. He writes, "We developed the list of shopping cart failures to study the use of the outline as a test idea generator." This is one of the prime uses of any defect taxonomy. His taxonomy lists over sixty high-level defect categories, some of which are listed here:

• Performance
• Reliability
• Software upgrades
• User interface usability
• Maintainability
• Conformance
• Stability
• Operability
• Fault tolerance
• Accuracy
• Internationalization
• Recoverability
• Capacity planning
• Third-party software failure
• Memory leaks
• Browser problems
• System security
• Client privacy

After generating the list he concludes, "We think the list is a sufficiently broad and well-researched collection that it can be used as a starting point for testing other applications." His assertion is certainly correct.

A Final Observation

Note that each of these taxonomies is a list of possible defects without any guidance regarding the probability that these will occur in your systems and without any suggestion of the loss your organization would incur if these defects did occur. Taxonomies are useful starting points for our testing but they are certainly not a complete answer to the question of where to start testing.

Your Taxonomy

Now that we have examined a number of different defect taxonomies, the question arises—which is the correct one for you? The taxonomy that is most useful is your taxonomy, the one you create from your experience within your organization. Often the place to start is with an existing taxonomy. Then modify it to more accurately reflect your particular situation in terms of defects, their frequency of occurrence, and the loss you would incur if these defects were not detected and repaired.

Just as in other disciplines like biology, psychology, and medicine, there is no one, single, right way to categorize, there is no one right software defect taxonomy. Categories may be fuzzy and overlap. Defects may not correspond to just one category. Our list may not be complete, correct, or consistent. That matters very little. What matters is that we are collecting, analyzing, and categorizing our past experience and feeding it forward to improve our ability to detect defects. Taxonomies are merely models and, as George Box, the famous statistician, reminds us, "All models are wrong; some models are useful."

To create your own taxonomy, first start with a list of key concepts. Don't worry if your list becomes long. That may be just fine. Make sure the items in your taxonomy are short, descriptive phrases. Keep your users (that's you and other testers in your organization) in mind. Use terms that are common for them. Later, look for natural hierarchical relationships between items in the taxonomy. Combine these into a major category with subcategories underneath. Try not to duplicate or overlap categories and subcategories. Continue to add new categories as they are discovered. Revise the categories and subcategories when new items don't seem to fit well. Share your taxonomy with others and solicit their feedback. You are on your way to a taxonomy that will contribute to your testing success.

Summary

• Taxonomies help you:
  • Guide your testing by generating ideas for test case design
  • Audit your test plans to determine the coverage your test cases are providing
  • Understand your defects, their types and severities
  • Understand the process you currently use to produce those defects (Always remember, your current process is finely tuned to create the defects you're creating)
  • Improve your development process
  • Improve your testing process
  • Train new testers regarding important areas that deserve testing
  • Explain to management the complexities of software testing

• Testing can be done without the use of taxonomies (nonspecific fault model) or with a taxonomy (specific fault model) to guide the design of test cases.
• Taxonomies can be created at a number of levels: generic software system, development paradigm, type of application, and user interface metaphor.

References

Beizer, Boris (1990). Software Testing Techniques (Second Edition). Van Nostrand Reinhold.

Binder, Robert V. (2000). Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison-Wesley.

Carr, Marvin J., et al. (1993) "Taxonomy-Based Risk Identification." Technical Report CMU/SEI-93-TR-6, ESC-TR-93-183, June 1993. http://www.sei.cmu.edu/pub/documents/93.reports/pdf/tr06.93.pdf

ISO (1991). ISO/IEC Standard 9126-1. Software Engineering - Product Quality - Part 1: Quality Model, ISO Copyright Office, Geneva, June 2001.

Kaner, Cem,Jack Falk and Hung Quoc Nguyen (1999). Testing Computer Software (Second Edition). John Wiley & Sons.

Whittaker, James A. (2003). How to Break Software: A Practical Guide to Testing. Addison Wesley.

Vijayaraghavan, Giri and Cem Kaner. "Bugs in your shopping cart: A Taxonomy." http://www.testingeducation.org/articles/BISC_Final.pdf