Game Testing All in One (Game Development Series)

Test Kickoffs

Kickoffs are known to have a positive impact on software development, leading to better process definition, better problem solving, and cycle time reduction. On a team in which testers have various levels of testing and game project experience, individual needs are not likely to be addressed at the project kickoff. It benefits the team to have kickoffs at the next lowest level: a test kickoff for each "test" that is being created or executed by individual testers. The test kickoff illustrates the principle that increasing an organization's speed results from an iterative process of identifying obstacles, designing a new process that eliminates them, and ensuring that the new way is implemented.

Test kickoff activities are broken into two parts : tester preparation and the kickoff meeting, which is conducted according to the kickoff agenda. The tester's preparation steps and the kickoff agenda are documented on a test kickoff checklist, as shown in Figure 7.3.

Figure 7.3: Test kickoff checklist.

From the test kickoff checklist, the tester prepares in the following ways:

1. Reads the requirements and/or documentation for the game feature being tested

2. Gathers equipment, files, and programs needed for the test

3. Reads through the tests

The tester should consult with a "test expert" if there are any roadblocks or questions regarding the completion of any preparation activities. The test expert can be the original author of the test, a tester who already has much experience with the game feature, or the test lead. The expert should also be familiar with the recent defect history of the game and feature(s) to be tested. Experienced testers should not be exempt from this preparation process, and this process should be completed fully before conducting the kickoff meeting.

Once the tester has completed the preparation activities, a kickoff meeting is held. The test expert leads the kickoff meeting by doing the following:

1. Giving a feature overview

2. Addressing feature questions

3. Bringing up any special test instructions

4. Bringing up and soliciting any relevant test improvement suggestions

5. Addressing any test execution questions or issues

6. Recording important issues on the kickoff form and providing a copy to the tester after the meeting is completed

Following the preparation steps listed on the checklist and participating in the interactive meeting per the kickoff agenda benefits testing in the following ways:

• Prepares and equips the tester to run through the entire test without stopping for equipment or questions

• Familiarizes the tester with the expected behavior of the game or module during testing to increase tester awareness of right from wrong

• Resolves any test instruction conflicts prior to executing the test in order to eliminate retesting because of test ambiguities or errors

• Provides a forum for test improvement at the grassroots level, improving tester involvement and ownership

Each test kickoff is an opportunity to improve test understanding, test quality, and test execution. These opportunities would have been missed or identified much later in the test phase if the kickoff process was not used. The net result is that the test kickoff acts as a "pre-mortem" identifying important issues prior to performing the test, rather than waiting to identify them in a post-mortem after testing has already been done. As kickoff records are collected, systemic issues can be identified and addressed in the current test phase. Checklists, group meetings, and email are ways to communicate the lessons learned from the kickoffs and suggest remedies to implement on the current project.

By collecting and evaluating the results of kickoffs for each project, actions can be taken to prevent these problems in future test efforts. The preventative analysis of test kickoff results and the cycle time impacts achieved by the initial deployment point out the potential for improving the way hundreds of other tests will be conducted as you use this process going forward. The across-the-board use of test kickoffs will translate into further test execution cycle time improvements and uncover more defects, leading to better game quality.

The following behaviors, which are driven by the use of test kickoffs, can reduce the size of the testing critical path :

• Make fewer mistakes (reduce wasted effort)

  The test kickoff steps are designed to ensure that testing does not begin until the tester is equipped to test and understands the details and goals of the testing. Among other things, this results in quicker and more accurate measurement of results.

• Reduce cyclical efforts (shortest distance between two points is a line)

  As part of preparation, the tester reviews the test and requirements in their entirety. This reduces misunderstood and improperly performed steps, resulting in much less test effort spent on backing up and redoing test sections.

• All effort results in something that will be used

  The metrics show that the use of test kickoffs reduces the testing cycle time, even when accounting for the time it takes to hold the kickoffs.

• Truth telling is encouraged

  The one-on-one forum of a test kickoff is less intimidating than the group setting of a phase or release kickoff. The kickoff leader should make the tester comfortable and remind the tester of the kickoff goals. When testers see that their feedback results in improvements, they are more open about voicing their opinions and ideas.

• Have constructive discussions rather than unnecessary effort and debate

  The test kickoff meeting gets every tester involved in process improvement. It also gives the tester and kickoff leader shared responsibility to address the issues raised and recorded in the meeting. Sticking to the kickoff meeting agenda will keep the meeting focused on test- related issues.

I realize it's a foreign concept that having a meeting will actually save time. I needed to know this for myself when test kickoffs were just an "idea." In the project I was working on at the time, I held test kickoffs for a portion of tests while the rest were executed without a kickoff. The "kicked-off" tests were executed at 1.4 times the rate of the "non-kickoff" testing. Putting it another way, testers who had a kickoff completed 40% more tests that those who did not have a kickoff.

Test kickoffs can provide the same benefits for test creation as they can for test execution. All it takes is a slightly different agenda and checklist, as shown in Figure 7.4.

Figure 7.4: Test creation kickoff checklist.

Both test kickoff checklists shown in this chapter are available on the book's CD-ROM.