Game Testing All in One (Game Development Series)

Once you've found the problem and can describe all the ways in which it affects the game, you need to record that information and notify the developers about it. Typically, project teams will use software tools to help with this. DevTrack is one of the defect tracking and management tools used in the game industry. Although you don't have to be concerned about the installation and management of this kind of tool, you should become familiar with how to best use it to record your defects and get them closed. This is not meant to be a full-blown tutorial ‚just an initial exposure and discussion of the following:

• Using a defect tracking system

• Information essential to good defect reports

• Some typical mistakes and omissions

• Extra things you can do to help get your bug looked at and fixed

Figure 2.3 shows the new defect entry window of the DevTrack tool. The main elements of this window are the function selection along the top, the view selection at the left, and the data viewing/entry screen on the right.

Figure 2.3: DevTrack's new defect window.

In general, this function and the view selection choices are very similar to using an email program like Microsoft Outlook. You can see which things need your attention, browse through them, and add information to them if you need to.

The data entry screen is where you make your contribution, so the following sections explore some of the key fields you need to work with. To learn more about using other functions of DevTrack, you can explore the demo at www.techexcel.com.

Describe

Start your entry with a descriptive title. Generic or broad descriptions like "Had to restart game" or "Problem in training hall" do not sufficiently describe the problem to get the attention of the people who need to go after the problem and fix it. Imagine picking up a newspaper and reading headlines like "Some Crime Happened " or "A Team Won" ‚it might leave you scratching your head. Instead, provide one or two details that help narrow down the problem.

Take a cue from the Sports page. If a team beats another under no special circumstances, you might see a headline like "Yankees Beat Red Sox." But if something else noteworthy happens, there might be more detail, like "Marlins Shutout Yankees to Win Series." Think of your bug as a noteworthy event that will be competing for the reader's attention.

Figure 2.4 has a title and a description for a problem found while playing Neverwinter Nights Gold . In this case the title mentions what happened and where it happened. Always include the "what" and then add one or two of the distinctive "who," "where," "when," or "how" factors.

Figure 2.4: Defect title and description.

In the description, be sure to include all of these details: who (Fighter character), what (on top of interior wall), where (in training hall), when (after visiting spell trainer), how (jump). Then, describe how you were able to remedy the situation, if at all, and add any things that you tried to do that would not reverse or undo the effects of the problem. This serves two purposes. First, it helps the project leaders evaluate the importance of fixing the bug. Second, it gives the developers clues about how the problem happened and how they might go about fixing it. It also establishes the minimum criteria for your testing later on when you need to verify that this bug was properly fixed.

Another way to describe your defect in detail would be to provide a step-by-step description of how you found it. Don't start from when you turned the computer on, but include the steps that are relevant to reproducing the problem. So, an alternative description for the Neverwinter Nights bug would be:

• Create a Fighter character. Go to the training hall and visit the spell trainer. Leave the spell trainer's room and jump up onto the wall across from his door. The character can move around along the wall but cannot jump down to resume playing the game.

You should also include information about any additional places you looked for the problem where it didn't show up.

Prioritize

Depending on the "rules of engagement" for your project, you may also be required to classify the defect priority (or "severity") and/or type. Figure 2.5 shows an example pull-down menu of choices for assigning an initial priority to this defect.

Figure 2.5: Defect priority selection.

The names and meanings of the priority choices may be different for your project, but the concept is the same. Rank the defect according to its importance, as defined for each choice. For example, "Urgent" may be defined as a defect that stops or aborts the game in progress without any way to recover and continue the game. This kind of bug probably also causes some of the player's progress to be lost, including newly won or discovered items. If your character was frozen in a multiplayer game, this could also result in player death and its associated penalties after the evil horde you were fighting continued happily pummeling you until your health reached 0.

A "High" priority bug may be a problem that causes some severe consequence to the player, such as not getting a quest item after successfully completing the quest. This priority could also be used for an "Urgent" type of defect that happens under obscure circumstances. You should be stingy about making that kind of a downgrade when you first log the bug, especially in a multiplayer game, because nefarious players may be able exploit that bug to their advantage or your disadvantage if the obscure circumstances are discovered in the released game and subsequently publicized. An example of this kind of misuse happened in the Asheron's Call PC-based online game where players would kill their character and then intentionally crash the game server. Once the server was back up, they were able to retrieve a duplicate of a rare item from their corpse. See the following sidebar for the developers' response to this defect when it occurred in January of 2001.

"Medium" defects cause noticeable problems, but probably do not impact the player in terms of rewards or progress. The difference between "High" and "Medium" may be the difference between getting your bug looked at and fixed, put aside to be fixed in a post-release patch, or left in the game as it is. When in doubt, unless otherwise directed by your project leads, assign the "High" priority so it will be fairly evaluated before being downgraded. Be careful not to abuse this tactic, or the defects you find will not be taken as seriously as they should.

The "Low" priority is normally for very minute defects that don't affect gameplay, those that occur under impossible conditions, or those that are a matter of personal taste. For example, in the GBA game Yu-Gi-Oh! The Eternal Duelist Soul , when Yami Bakura is defeated for the 10th time, the dialog box refers to the "Great Sheme of Things" instead of the "Great Scheme of Things."

January 23, 2001

We wanted to thoroughly explain the cause of today's hotfix, and what impact it will have on you, the players.

Late Monday night, a bug was discovered that allowed players to intentionally crash the server their characters were on. Additionally, a person could use this bug and the resulting time warp ( reverting back to the last time your character was saved to the database) to duplicate items. By intentionally crashing the servers, this also caused every other player on that server to crash and time warp, thus losing progress.

We were able to track down this bug and we turned off the servers to prevent additional people from crashing the servers and/or duplicating items.

The good news is that we were able to track down all the players who were exploiting this bug and crashing the servers. As we have stated in the past: Since Asheron's Callwas commercially released, it has been our policy that if players make use of a bug that we did not catch or did not have time to fix before releasing the game, we would not punish them for our mistake, instead directing our efforts toward fixing those bugs as soon as possible. The exceptions to this are with those bugs that significantly affect the performance or stability of the game .

The players who were discovered repeatedly abusing this bug to bring down the servers are being removed from the game. While we dislike taking this type of action, we feel it is important that other players know that it is unacceptable to disrupt another player's game in such a fashion.

We deeply regret this bug, and sincerely apologize for the consequences this has had on our players.

‚ The Asheron's Call Team

http://classic.zone.msn.com/asheronscall/news/ASHEletterJan2.asp

In many game companies, an additional "severity" rating is used in conjunction with a "priority." In these cases, the severity field describes the potential impact of the bug on the player, while the priority field is used by the team to establish which defects are determined to be the most important to fix. These categories can differ when a low impact (severity) defect is very conspicuous, such as misspelling the game's name on the Main Menu, or when a very severe defect is not expected to occur in a player's lifetime, such as a crash that is triggered when the console's date rolls over to the year 3000. The ability of a player to recover from a defect and the risk or difficulty associated with fixing a defect are also factors in determining the priority apart from the severity. In this arrangement, severities are typically assigned by the person who logs the defect and the priority gets assigned by the CCB or the project manager.

Pick a Type

The defect Type field is also important for routing and handling of your defect. Figure 2.6 shows the list provided in the DevTrack demo.

Figure 2.6: Defect type selection.

Not everything you find as a tester is a bug in the sense that something doesn't work as planned. You will find things that could be improved or added to the game to make it better. These kinds of issues would be classified as "Minor Improvement" or "New Feature," respectively.

Likewise, you can enter a "Future Enhancement" for such things as

• An idea for the sequel

• An optimization to make for the next platform you port the game to

• Adding support for a brand new type of controller

• A feature or item to make available for download after the game ships

Another function of the game tester is to check the documentation. You could be looking for consistency with how the actual game functions, important things that are left out, or production errors such as missing pages, mislabeled diagrams, and so on. These would fall into the "Documentation Problem" type.

The "Third Party" type could be problems introduced by software or hardware that your team does not produce. For example, a particular brand of steering wheel controller doesn't give force feedback to the user , whereas three other brands work fine.

For the kinds of defects where the game is simply not working the way it is supposed to, the "Discrepancy Report" type would be specified, while the "Change Request" choice might be used for something that has a larger scope, such as redoing the collision detection mechanism in the game.

Be Helpful

Finally, make sure you include any other artifacts or information that might be of help to anyone trying to assess or repair the problem. In addition to adding details to the description, with DevTrack you can use the Notes function to add helpful files to the defect record. Attach or provide links to any of the following items you can get your hands on:

• Server logs

• Screen shots

• Transcripts from the character's journal

• Sound files

• Saved character file

• Videotape recording (including audio) of the events leading up to and including the bug

• Traces of code in a debugger

• Log files kept by the game platform, middleware, or hardware

• Operating system pop-ups and error codes

• Data captured by simulators for mobile game environments, such as BREW ‚, Java, Palm OS , or Microsoft ‚ Windows ‚ CE

Pass or Fail

Tests that fail are good from your perspective, but it's also important for the project to know which tests passed. You will probably be expected to record the completion and results of your tests by indicating which passed and which failed. Other status types might include "Blocked" or "Not Available," where "Blocked" indicates that an existing problem is preventing you from executing one or more steps in the tests, and "Not Available" indicates that a feature or capability that is part of the test is not available in the version of the game you are testing. For example, a multiplayer scenario test is "Blocked" if a defect in the game prevents you from connecting to the multiplayer server. A test that can't be run because the level hasn't been added to the game yet would be classified as "Not Available."

Usually this information goes to the test lead and it's important to provide it in a timely manner. It might just be a physical sheet of paper you fill out and put on a pile, or an electronic spreadsheet or form you send in. This information can affect planning which tests get run for the next release, as well as the day-to-day assignments of each tester.