Real-World .NET Applications

The fastest kind of analysis is simply reading the comments. I print out large groups of comments (again, 100–200 should be sufficient) and go to a quiet cafe for a couple of hours, marking and pulling ones that have good quotations or clear examples of behavior. When reading them, I keep a couple of things in mind:

• Read from the user's perspective. They don't know everything I know about the product. They don't use the same words. They don't have the same emotional attachment. Try to read feedback as it was written.

• Focus on the facts. What is wrong? How is it wrong? What part of the product is being referenced? Some comments will be made in the throes of anger or euphoria. When they're angry, they should not be discounted, nor should they be given extra weight because they're laudatory. However, what makes people happy or angry provides insight into a powerful part of their experience.

• Don't jump to conclusions. A couple of messages doesn't necessarily define a trend, but even a single message can contain a useful idea. Just because a couple of people have said that they like or don't like something doesn't mean that they're representative of a sizable segment of the population.

• Don't make the list of common problems into a list of must-have fixes. Complaints are pointers to problems, not problem descriptions. Complaints from people not finding the "fork" link don't mean that there immediately needs to be a bigger "fork" link. It could mean that the way silverware is divided is incompatible with the target audience's expectations, or that they don't understand how that structure is being communicated. Resist the urge to solve problems as you learn about them.

• Take comments with a grain of salt. There's a reason why the term silent majority exists. There are many people who do not feel or behave the same way as those who send comments. Don't trivialize the feedback, but don't treat it as gospel either. Sometimes gripes are amplified versions of common problems, and sometimes they're just unjustified gripes.

As for analysis, I keep track of several facets of the audience's experience and try to notice patterns in them.

• Who the users are. People can give a lot of information about themselves while they're explaining a problem or a request. Different groups of people may have different kinds of comments.

• What they're trying to do. Are there common goals they're trying to reach? What are the intermediate goals (as described by the users, not as part of the interaction model)?

• How they approach problems. Despite having different specific problems, there may be common strategies that help people understand how to use the system. Similar descriptions and problems at analogous places in the process provide insight into the mental model people build.

• The problems they're having. Are there similarities to the kinds of problems people are having? Are there obvious bottlenecks?