Exploratory Testing

As she contemplated the setting sun, its dying rays casting the last of their brilliant purple light on the red-gold waters of the lake, Debbie realized that she should never again buy her sunglasses from a guy parked by the side of the road.

— Malinda Lingwall

Introduction

The term "exploratory testing," coined by Cem Kaner in his book Testing Computer Software, refers to an approach to testing that is very different from scripted testing. Rather than a sequential examination of requirements, followed by the design and documentation of test cases, followed by the execution of those test cases, exploratory testing, as defined by James Bach, is "simultaneous learning, test design, and test execution." The tester designs and executes tests while exploring the product.

In an article for StickyMinds.com entitled "Exploratory Testing and the Planning Myth," Bach wrote, "Exploratory Testing, as I practice it, usually proceeds according to a conscious plan. But not a rigorous plan ... it's not scripted in detail." James adds, "Rigor requires certainty and implies completeness, but I perform exploratory testing precisely because there's so much I don't know about the product and I know my testing can never be fully complete." James continues, "To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing. We become more exploratory when we can't tell what tests should be run, in advance of the test cycle."

Exploratory Testing

To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing. We become more exploratory when we can't tell what tests should be run, in advance of the test cycle.

In exploratory testing, the tester controls the design of test cases as they are performed rather than days, weeks, or even months before. In addition, the information the tester gains from executing a set of tests then guides the tester in designing and executing the next set of tests.

Note this process is called exploratory testing to distinguish it from ad hoc testing which (by my definition, although others may disagree) often denotes sloppy, careless, unfocused, random, and unskilled testing. Anyone, no matter what their experience or skill level, can do ad hoc testing. That kind of testing is ineffective against all but the most defect-ridden systems, and even then may not find a substantial portion of the defects.

Bach suggests that in today's topsy-turvy world of incomplete, rapidly changing requirements and minimal time for testing, the classical sequential approach of Test Analysis followed by Test Design followed by Test Creation followed by Test Execution is like playing the game of "Twenty Questions" by writing out all the questions in advance. Consider the following discussion from a testing seminar discussing exploratory testing:

How successful would we be at this game if we had to write out all the questions in advance? When we play this game well, each question depends on the previous questions and their answers. So it is in exploratory testing. Each test provides us with information about the product. We may see evidence of the product's correctness; we may see evidence of its defects. We may see things that are curious; we're not sure what they mean, things that we wonder about and want to explore further. So, as we practice exploratory testing, we concurrently learn the product, design the tests, and execute these tests.

Description

In his classic time management book, How to Get Control of Your Time and Your Life, Alan Lakein suggests we should constantly ask ourselves: What is the most important thing I can do with my time right now? Exploratory testers ask an equivalent question: What is the most important test I can perform right now?

Key Question

What is the most important test I can perform right now?

A possible exploratory testing process is:

Another process might be simply to explore and learn before forming conjectures of proper behavior.

Exploratory testing can be done within a "timebox," an uninterrupted block of time devoted to testing. These are typically between sixty and 120 minutes in length. This is long enough to perform solid testing but short enough so that the tester does not mentally wander. In addition, a timebox of this length is typically easier to schedule, easier to control, and easier to report.

When performing "chartered exploratory testing," a charter is first created to guide the tester within the timebox. This charter defines a clear mission for the testing session. The charter may define:

This charter is a guideline to be used, not a script to be followed. Because of this approach, exploratory testing makes full use of the skills of testers. Bach writes, "The more we can make testing intellectually rich and fluid, the more likely we will hit upon the right tests at the right time."

  Key Point

The charter is a guideline to be used, not a script to be followed.

Charters focus the exploratory tester's efforts within the timebox. Possible charters include:

It is possible to perform exploratory testing without a charter. This is called "freestyle exploratory testing." In this process testers use their skills to the utmost as they concurrently learn the product and design and execute tests.

Exploratory testers are skilled testers. (Of course, we want testers to be skilled no matter what testing process we are using!) The exploratory testing approach respects those skills and, in fact, depends on them. Good exploratory testers are:

Testers without these skills can still perform useful exploratory testing if they are properly supervised and coached.

In general, processes that have weak, slow, or nonexistent feedback mechanisms often do not perform well. Scripted testing is a prime example of a slow feedback loop. Exploratory testing provides a tight feedback loop between both test design and test execution. In addition, it provides tight feedback between testers and developers regarding the quality of the product being tested.

Advantages of Exploratory Testing

  1. Exploratory testing is valuable in situations where choosing the next test case to be run cannot be determined in advance, but should be based on previous tests and their results.

  2. Exploratory testing is useful when you are asked to provide rapid feedback on a product's quality on short notice, with little time, off the top of your head, when requirements are vague or even nonexistent, or early in the development process when the system may be unstable.

  3. Exploratory testing is useful when, once a defect is detected, we want to explore the size, scope, and variations of that defect to provide better feedback to our developers.

  4. Exploratory testing is a useful addition to scripted testing when the scripted tests become "tired," that is, they are not detecting many errors.

Disadvantages of Exploratory Testing

  1. Exploratory testing has no ability to prevent defects. Because the design of scripted test cases begins during the requirements gathering and design phases, defects can be identified and corrected earlier.

  2. If you are already sure exactly which tests must be executed, and in which order, there is no need to explore. Write and then execute scripted tests.

  3. If you are required by contract, rule, or regulation to use scripted testing then do so. Consider adding exploratory tests as a complementary technique.

Summary

References

Bach, James. "Exploratory Testing and the Planning Myth." http://www.stickyminds.com/r.asp?F=DART_2359, 19 March 2001.

Bach, James. "Exploratory Testing Explained." v.1.3 16 April 2003. http://www.satisfice.com/articles/et-article.pdf

Kaner, Cem,Jack Falk, and Hung Q. Nguyen (1999). Testing Computer Software. John Wiley & Sons.

Kaner, Cem,James Bach, and Bret Pettichord (2002). Lessons Learned in Software Testing: A Context-Driven Approach. John Wiley & Sons.

Weinberg, Gerald M. (1975). An Introduction to General Systems Thinking. John Wiley & Sons.

Related

Категории