Game Testing All in One (Game Development Series)
It's possible to generate Cleanroom tests using any of the methods covered in this book. You can also create your own Cleanroom tests on-the-fly . A usage probability must be assigned to each step in the test. This can be done in writing or you can keep track in your head. Use the usage probability to select test steps, values, or branches and put them in sequence to produce tests that reflect your usages. For example, if you expect a simulation game player to develop residential property 50% of the time, commercial property 30% of the time, and industrial property 20% of the time, then your Cleanroom tests will reflect those same frequencies.
Cleanroom Combinatorial Tables
Cleanroom combinatorial tables will not necessarily be "pairwise" combinatorial tables. The number of tests to be created is determined by the test designer and the values for each test will be chosen on the basis of their frequency of use rather than whether or not they satisfy one or more necessary value pairs.
To produce Cleanroom combinatorial tables, assign usage probabilities to the test values of each parameter. The probabilities of the set of values associated with a single parameter must add up to 100%.
To illustrate how this is done, revisit the parameter and value choices for the HALO Advanced Controls table you completed in Figure 10.26. The test values for each parameter are listed below with the default values identified.
-
Look Sensitivity : 1, 3 (default), 10
-
Invert Thumbstick : Yes, No (default)
-
Controller Vibration : Yes (default), No
-
Invert Flight Control : Yes, No (default)
-
Auto Center : Yes, No (default)
Next , usage percentages need to be determined for each of the table's parameters. If you are considering testing against more than one player profile, you can make a separate usage table for each parameter with a column of usage percentages for each of the profiles you intend to test. Figures 12.3 through 12.7 show multiple profile usage tables for each of the five HALO Advanced Controls parameters you will incorporate in your Cleanroom combinatorial table.
Figure 12.3: Look Sensitivity values with usage percentages.
| Note ‚ | Please , don't send emails to explain why any of these percentages are wrong. We have only presented a variety of numbers to illustrate differences between user types based on personal experience. If you have better data gathered through scientific means, then that is what you should be using. If these numbers don't make sense to you, then consider them "for entertainment purposes only" as you continue through the examples in this chapter. |
Figure 12.4: Invert Thumbstick values with usage percentages.
Figure 12.5: Controller Vibration values with usage percentages.
Figure 12.6: Invert Flight Control values with usage percentages.
Figure 12.7: Auto-Center values with usage percentages.
Cleanroom Combinatorial Example
A Cleanroom combinatorial table can be constructed for any of the player usage profiles you define. For this example, you will create one such table for the "Casual" player. To decide which value to choose for each parameter, you need a random number source. You could think of a number in your head, write a program to generate a list of numbers, or roll electronic dice on your PDA. Microsoft Excel can generate random numbers for you with the RAND() function or the RANDBETWEEN() function if you install the Analysis ToolPak add-in. There is no wrong way as long as the number selection is not biased toward any range of numbers.
Start building the table with an empty table that has column headings for each of the parameters. Decide how many tests you want and leave room for them in the table. A Cleanroom combinatorial table "shell" for the HALO Advanced Controls is shown in Figure 12.8. It has room for six tests.
Figure 12.8: HALO Advanced Controls Cleanroom combinatorial table shell.
Since there are five parameters, get five random numbers in the range of 1 ‚ 100. These will be used one at a time to determine the values for each parameter in the first test. Construct the first test from the five numbers 30, 89, 77, 25, and 13.
Figure 12.9 shows the first test constructed from the random numbers 30, 89, 77, 25, and 13.
Figure 12.9: The first Advanced Controls Cleanroom combinatorial test.
A new set of five random numbers is required to produce the second test case. Use 79, 82, 27, 8, and 57.
The first number is 79, which is within the 11 ‚ 95 range for Look Sensitivity = 3. Put a "3" again in the first column for Test 2. The second usage number is 82. It falls within the 11 ‚ 100 range for Invert Thumbstick = No, so put a "NO" in the Invert Thumbstick column for Test 2. Your third number is 27. This is within the 1 ‚ 80 range for Controller Vibration = Yes. Add the "YES" to the third column of values for Test 2. The fourth random number is 8. This usage value corresponds to the Yes value range of 1 ‚ 25 for the Invert Flight Control parameter. Enter "YES" into that column for Test 2. For the last parameter of the second test, the random number is 57. This number is in the 31 ‚ 100 No range for Auto-Center. Complete Test 2 by entering "NO" in the last column. Figure 12.10 shows the first two completed rows for this Cleanroom combinatorial table.
Figure 12.10: Two Advanced Controls Cleanroom combinatorial tests.
The third test in this table is constructed from the random number sequence 32, 6, 64, 66, and 11. Once again, the first value corresponds to the default Look Sensitivity value of "3." The second usage number is 6, which results in the first Yes entry for the Invert Thumbstick parameter by virtue of being inside the 1 ‚ 10 range for that value. The number to use for determining the Controller Vibration test value is 64, which maps to the 1 ‚ 80 range for the Yes choice. The fifth number also provides another first ‚ the first No value for Invert Flight Control, because it falls within the 26 ‚ 100 range. The last random number for Test 3 is 11, which gives you a Yes value for the Auto-Center parameter. Figure 12.11 shows the first three tests entered in the table.
Figure 12.11: Three Advanced Controls Cleanroom combinatorial tests.
Continue by using the random numbers 86, 64, 95, 50, and 22 for Test 4. The 86 is within the 11 ‚ 95 range for Look Sensitivity =3, so put a "3" again in column one. A 64 is next in the usage number list. It maps to the No range for Invert Thumbstick. The next number, 95, provides the first No value for Controller Vibration in this set of tests. The 50 for Invert Flight Control puts a No in the table and the last usage number, 22, corresponds to a Yes for Auto-Center. Figure 12.12 shows the table with four of the six tests defined.
Figure 12.12: Four Advanced Controls Cleanroom combinatorial tests.
Your fifth set of random numbers is 33, 21, 63, 85, and 76. The 33 puts a "3" in the Look Sensitivity column. The 21 is in the No range for Invert Thumbstick. The 63 corresponds to a Yes value for Controller Vibration. An 85 is within the No range for Invert Flight Control and the 76 causes a No to be put in the last column for the Auto-Center parameter. Figure 12.13 shows the Cleanroom combinatorial table with five tests defined.
Figure 12.13: Five Advanced Controls Cleanroom combinatorial tests.
One more number set is needed to complete the table. Use 96, 36, 18, 48, and 12. The first usage number of 96 is high enough to be in the 96 ‚ 100 range for the "10" Look Sensitivity value. This marks the first time that value appears in the table. Moving through the rest of the numbers, the 36 puts a No in the Invert Thumbstick column, 18 corresponds to Controller Vibration = Yes, 48 is in the range for Invert Flight Control = No, and 12 completes the final test row with a Yes for Auto-Center. Figure 12.14 shows all six Cleanroom combinatorial test cases.
Figure 12.14: Completed Advanced Controls Cleanroom combinatorial table.
Your keen testing eye should have noticed that Look Sensitivity = 1 was never generated for this set of tests. That is a function of its relatively low probability (10%), the low number of test cases that you produced, and the particular random number set that was the basis for selecting the values for table. In fact, if you stopped generating tests after five test cases instead of six, the default value of "3" would have been the only value for Look Sensitivity that appeared in the table. This should not be considered a problem for a table of this size . If a value has a 5% or higher usage probability and you don't see it at all in a test set of 100 or more tests, then you may suspect that something is wrong with either your value selection process or your random number generation.
Also notice that some values appear more frequently or less frequently than their usage probability would suggest. Auto-Center = Yes only has a 30% usage for the Casual profile, but it appears in 67% (4/6) of the tests generated. This is mainly due to the low number of tests created for this table. With a test set of 50 or more you should see a better match between a value's usage probability and its frequency in the test set.
Just to reinforce the fact that the Cleanroom combinatorial table method doesn't guarantee it will provide all test value pairs that are required for a pairwise combinatorial table, confirm that the pair Controller Vibration = No and Auto-Center = No is absent from Figure 12.14. Now take a moment to see which other missing pairs you can find.
| Note ‚ | If you use any of the random number sequences in this chapter for your lottery ticket and you win, please remember to share with the authors of this book. |
You will recall that pairwise combinatorial tables are constructed vertically, one column at a time. Until you complete the process for building the table you don't know what the test cases will be or how many tests you will end up with. Because Cleanroom combinatorial tables are constructed horizontally ‚ one line at a time, you get a completely defined test on the very first row, and every row after that for as many Cleanroom combinatorial tests as you choose to produce.
TFD Cleanroom Paths
Cleanroom TFD tests come from the same diagram you use for creating minimum, baseline, and expert constructed paths. Cleanroom test paths travel from state to state by choosing each subsequent flow based on its usage probability.
A usage probability must be added to each flow if the TFD is going to be used for Cleanroom testing. The probabilities of the set of flows exiting each state must add up to 100%. Figure 12.15 shows a flow with the usage probability after the action. If there is no action on the flow, then the usage probability gets added after the event.
Figure 12.15: Example flow with usage probability.
Figure 12.16: Unlock Item TFD with usage probabilities added.
TFD Cleanroom Path Example
With the usage information added to the TFD, generate random numbers to guide you around the diagram from flow to flow until you reach the OUT terminator. The resulting path defines a single test. Continue generating as many paths as you like, using new random number sets each time. Experience has shown that it is a good practice to always assign a 10% value to the Exit flow. A larger value will result in paths that exit too soon and a smaller value will cause too many paths that seem to go on forever before finally exiting. The 10% value provides a nice mix of long, medium, and short paths in your Cleanroom test set.
Each Cleanroom test case is described by the sequence of flow numbers along the Cleanroom path. Since the path length can vary from one test to another, you will not know ahead of time how many random numbers you need to generate for all of your paths. You may end up exiting some tests after only a few flows or you could travel around the diagram quite a bit before reaching the OUT box. Normally you would generate the random numbers as you need them, but for your convenience the random number set for the example in this section is 30, 27, 35, 36, 82, 59, 92, 88, 80, 74, 42, and 13.
There are two flows exiting state HavePointsItemLocked . Flow 2 has a 10% usage and flow 4 has a 90% usage. The range for flow 2 is 1 ‚ 10 and for flow 4 it's 11 ‚ 100. Use 27 as the random number for this flow. That sends the test along flow 4 to HavePointsItemUnlocked . The test path at this point is 1, 3, 4.
HavePointsItemUnlocked is the most interesting state so far, with four flows to choose from for the next step in your test. Flow 5 has a 5% usage, flow 6 has 10%, flow 7 has 75%, and flow 12 has 10%. The corresponding number ranges are 1 ‚ 5 for flow 5, 6 ‚ 15 for flow 6, 16 ‚ 90 for flow 7, and 91 ‚ 100 for flow 12. You anxiously await the next random number ‚ and it's ‚ 35. Your test path now takes flow 7 to NoPointsHaveItem . The path is now 1, 3, 4, 7.
From NoPointsHaveItem there are two flow choices: flow 8 with a 10% usage and flow 9 with a 90% usage. You will take flow 8 if the random number is in the range 1 ‚ 10 and flow 9 if it's within 11 ‚ 100. Your new random number is 36, so take flow 9 to NoPointsItemUnlocked . The test path is currently 1, 3, 4, 7, 9.
Flows 10, 11, and 13 all leave NoPointsItemUnlocked . Flow 10's usage is 5% (1 ‚ 5), flow 11 has a 90% usage (6 ‚ 95), and flow 13 has a 5% (96 ‚ 100) usage. Another random number is generated and it's 82. That's within the range for flow 11, so take that flow to HavePointsItemUnlocked . The path has grown to 1, 3, 4, 7, 9, 11, but you're not done yet.
You're back at HavePointsItemUnlocked and the next random number is 59. That fits in the 16 ‚ 90 range for flow 7, taking you on another trip to NoPointsHaveItem . A usage of 92 here matches up with flow 9, going to NoPointsItemUnlocked . The test path is now 1, 3, 4, 7, 9, 11, 7, 9.
The next random number is 88. This takes you from NoPointsItemUnlocked to HavePointsItemUnlocked via flow 11. The 80 takes you along flow 7 for the third time in this path and the next number, 74, sends you to NoPointsItemUnlocked via flow 9. A 42 in the random number list chooses flow 11, which brings you once again to HavePointsItemUnlocked . These flows extend the path to 1, 3, 4, 7, 9, 11, 7, 9, 11, 7, 9, 11.
The next random number to use is 13. This falls within the 6 ‚ 15 range, which corresponds to flow 6. That's the Exit flow, which goes to the OUT terminator. This marks the end of this test path. The completed path is 1, 3, 4, 7, 9, 11, 7, 9, 11, 7, 9, 11, 6.
Once a path is defined, create the test cases using the data dictionary techniques described in Chapter 11. To create an overview of this test, list the flows, actions, and states in the order they appear along the path. List the flow number for each step in parentheses at the beginning of each line, as follows :
IN_PointsItem (1) Enter, NoPointsItemLocked (3) GetPoints, GetPointsEffects, HavePointsItemLocked (4) UnlockItem, UnlockEffects, HavePointsItemUnlocked (7) PurchaseItem, GetItemEffects, NoPointsHaveItem (9) DeleteItem, DeleteItemEffects, NoPointsItemUnlocked (11) GetPoints, GetPointsEffects, HavePointsItemUnlocked (7) PurchaseItem, GetItemEffects, NoPointsHaveItem (9) DeleteItem, DeleteItemEffects, NoPointsItemUnlocked (11) GetPoints, GetPointsEffects, HavePointsItemUnlocked (7) PurchaseItem, GetItemEffects, NoPointsHaveItem (9) DeleteItem, DeleteItemEffects, NoPointsItemUnlocked (11) GetPoints, GetPointsEffects, HavePointsItemUnlocked (6) Exit, OUT_PointsItem
Generating this path provided some expected results. The path starts with the IN and ends with the OUT, which is mandatory. Flows with large percentages were selected often, such as flows 9 and 11, which each have a 90% usage probability.
Did anything surprise you? Some flows and states didn't appear in this path at all. That's okay for a single path. When you create a set of paths you should expect to explore a wider variety of flows and states.
Was the flow longer than you expected? Flows 7, 9, and 11 appeared multiple times in this path. This is not what you would expect from minimum or baseline path sets. It's also interesting to note that those three flows form a loop. They were used three times in a row before finally exiting and ending the path.
Was the path longer than you wanted it to be? Is this a path you would have chosen on your own? Since this technique is based on a process rather than the ideas or pre-conceptions of a particular tester, the paths are free of bias or limitations. Cleanroom paths also highlight the fact that the game is not played one operation at a time and then turned off. These paths will test realistic game-use scenarios if your percentages are reasonably correct. As a result, your Cleanroom tests will have the ability to reveal defects that are likely to occur during extended or repeated game use.
Flow Usage Maintenance
There will come a time when you will need to move one or more flows around on your TFD. This may or may not affect your usage values. When a flow's destination (arrow-head end) changes, you are not required to change its usage. Conversely, if you change a flow to originate from a new state, you must re-evaluate the usage values for all flows coming from both the new state and the original one.
Figure 12.17: Unlock Item TFD with altered flow 9.
Figure 12.18: Unlock Item TFD with altered flows 6 and 9.
For HavePointsItemUnlocked , one or more percentages need to increase since that state lost a flow. You can give flow 12 the 10% that used to be allocated to flow 6. That would not overly inflate the usage for flow 7 and it keeps flow 5's usage small. As Figure 12.18 shows, flow 12 now has a 20% usage instead of its original 10% value.
Additionally, one or more flows coming from NoPointsHaveItem must now be reduced to make room for the new flow. Since flow 6 is an Exit flow, it must have a 10% usage. Two other flows come from NoPointsHaveItem : flow 8 with a 10% usage and flow 9 with a 90% usage. Reducing flow 8 by 10% will put it at 0%, meaning it will never be selected for any Cleanroom paths for this TFD. Instead, take away 5% from flow 8 and 5% from flow 9. The new percentages for these flows are reflected in Figure 12.18. Alternatively, you could have taken 10% away from flow 9 and left flow 8 at 10%. Your choice depends on what distribution you think best reflects the expected relative usage of these flows according to the game player, mode, or data you are trying to model.
Flow Usage Profiles
You may want to have multiple usage profiles to choose from when you create TFD Cleanroom paths. One way to accomplish this is to create copies of the TFD and change the usage numbers to match each profile. Another solution is to do what you did for combinatorial profiles: produce a mapping between each test element and its usage probability for one or more game users, types, or modes. In this case, usage numbers should not appear on the TFD. Figure 12.19 shows the Unlock Item TFD without usage percentages on the flows.
Figure 12.19: Unlock Item TFD without usage probabilities.
Figure 12.20: Casual player usage table for Unlocked Item TFD flows.
The total at the bottom of the flow probability table is a good way to check that your percentages add up right. The total should be equal to 100 (for the Enter flow) plus 100 times the number of states on the diagram (flows exiting each state must add up to 100%). The TFD in Figure 12.19 has five states, so 600 is the correct total.
Generate your Cleanroom tests from the flow usage table similar to the way you do when the flow usage is on the diagram. The only difference is the extra step to look up the flow's range in the table. If you are creating an automated process or tool to construct TFD Cleanroom paths, this table could be stored in a database or exported to a text file.
Admittedly, keeping track of flow usage in a table presents some problems. Because the flow numbering does not have to be related to the way flows appear on the diagram, it takes a little more work to identify the flows coming from each individual state. For example, the flows coming out from NoPointsItemLocked ‚ 13 and 14 ‚ are at opposite ends of the flow list. This wrinkle can become more of a problem when many flows are added, moved, or removed to adapt to changes in the game software. Just be careful and check your numbers when you are faced with this situation.