The Art and Business of Speech Recognition: Creating the Noble Voice

To get the best test results, you need to do a few things ahead of time.

Test the test. Before getting subjects to test an application, always perform a dry run to ensure that the subjects will be able to use it and that the application hasn't changed (for the worse ) since the last time you used it.

Create a pre-test questionnaire. A questionnaire helps designers and testers get a better understanding of the test subjects and their interactions with the system. In addition to such personal data as age and education level, the questionnaire should include application- related questions. For example, if we were testing a brokerage application, we might ask, "Currently, how many times a month do you make trades?" This questionnaire mainly serves as reference material to help the tester analyze and interpret the subject's test experience. Figure 8.1 is an example of a pre-test questionnaire that might be used when testing a flight reservation system.

Develop a list of tasks to test. Designers and testers need to decide which tasks the test subjects will be asked to perform during the test. When testing a home banking application, tasks would probably include checking balances and transferring funds. This set of top- tier tasks should be derived from the Requirements Specification. The tasks need to be representative of the most common tasks that users will perform, as well as some of the critical but less used features of the application. Ideally, the entire application would be tested , but sometimes this isn't possible, usually due to time constraints. Most often, the tester chooses a set of three to five tasks from those most commonly used in the system. For a home banking application, the test subjects' tasks might be to check their balances , transfer funds between accounts, and determine if a particular check has been cashed.

Some common tasks can be omitted from the test because they're similar to other tasks being tested. For example, in a brokerage application, there'd be little reason to ask the subject to check a stock's price/earnings ratio if they've already demonstrated their ability to check other information, such as the stock's 52-week high price.

Testers are also unlikely to include tasks that only a small fraction of the target population would ever perform ”unless the task is very different in style from the rest of the application or uses some unusual terminology or may be critical for that small population. In those cases, testing would be warranted. If, however, the task is quite standard and the designer and tester agree that it has no esoteric content or style, then usually it won't need to be tested.

Organize a logging method. There are several ways to record the test subject's testing experience ”usually videotape, audio tapes of the phone line, and perhaps a tick-sheet that the tester will use to note how many times particular events occur. Making a videotape of the test is best because it can be reviewed and reanalyzed later if new issues present themselves . Once the recording method has been established, the testing environment needs to be prepared.

Create a post-test questionnaire. This questionnaire is given to test subjects after they've completed the test. It enables testers and designers to get the test subjects' written answers to questions such as "How well did you like the voice of the system? Rate this on a scale of 1 (not at all) to 5 (very much)" or "List three things that could be changed to improve this system."

Figure 8.2 shows what a sample post-test questionnaire might look like.

When all these preparations are complete, it's time to test. Only one question remains: How do we get the right test subjects?