These guidelines are especially applicable to qualitative studies, where the goal is to understand whether participants are able to achieve a goal or perform a task they have in mind, see what problems they encounter, and identify the mental processes behind their behaviors—why they do what they do. Some of these guidelines might be less applicable if you need to collect strict performance data and statistical measures of usability—for example, success rates or times to complete a task—or if you need to resolve very simple usability issues. These guidelines are also better adapted to formative studies than summative ones.
However, whatever type of usability testing you are doing or type of product you are testing, it is always important to consider whether the artificiality of the testing conditions might bias your findings and to what extent you should ground usability testing in participants’ real lives.
1. Recruit passionate users.
By recruiting passionate users, I mean recruiting participants who
- use the type of product or content you are testing in their real lives
- are truly interested in the product or content
- have an emotional connection to it
Recruiting participants who typically use the type of product or content you are testing in their real lives may seem obvious, but it is easy to overlook this when you are absorbed in the recruiting process and confronted with external deadlines and constraints on recruiting. This mistake is especially easy to make when testing a product or Web site that is somewhat specialized, but pretty easy for anyone to understand—for example, a dating, movie, or travel Web site.
Example: Testing a Dating Web Site
When testing a dating Web site, one might think: Oh, pretty much everybody knows about dating Web sites or has used one at some point, so let’s just ask the participants we’ve recruited for this other Web site to pretend they’re looking for a mate. If you’re testing common usability issues like the discoverability of a data refinement menu, this approach can work. The personal engagement and interest of participants in the activity of dating online would have very little impact on the discoverability of the menu.
However, if you are testing user perceptions of the subscription process—does it take too long to subscribe and create a profile—involving participants who are not interested in dating could be misleading. The emotional involvement a participant has in the process of finding a mate online can have a big influence on a participant’s perception of the process. If finding a mate is crucial to a participant, he might be ready to invest a lot of time and energy in it. His perception of the length of the process might be very different from that of someone who is not emotionally involved in the process. What one person might find unendurably long, the emotionally involved participant might find okay—if the process is so important to him, he’s ready to go through it.
It is simply impossible to put yourself in the shoes of somebody who is emotionally involved in the process of looking for a mate if you are not. You can’t predict how a person who is emotionally involved would react. Participants don’t need merely to understand the content; they need to be emotionally involved in the process.
This is what Jared Spool calls “passion and its effect” [2]: “Passion on a subject changes how participants invest in usability test tasks. That change can have profound effects on the results and the recommendations produced by the team.” Passion is about a participant’s emotional connection and engagement with a topic of interest. It is a user’s emotional connection to and knowledge of a topic that changes his perception of it. You can’t predict how a person who is passionate and knowledgeable about a topic will react.
Therefore, you must recruit people who are actually involved in the process you are testing and are passionate about it. This is especially true if you want to test how users respond to specific content or functionality, how they navigate and search for information, or whether certain content is useful and valuable to users.
For simple, general usability issues—such as the discoverability of a menu item or content on a page, role-playing might not be an issue. The degree of a participant’s engagement with a site or topic might not have much effect on its discoverability. Likely, you would be able to assess such issues by doing testing with any Internet users, regardless of whether they are into dating.
However, it is always important is to ask yourself to what extent having participants role-play might affect the reliability of your test findings. Sometimes, it won’t; but other times, it will.
2. Have participants perform realistic tasks.
The tasks participants perform during usability testing should match what they do with a product in their real lives. This is another guideline that sounds straightforward to usability experts, but it’s easy to overlook its importance when you are occupied with designing well-crafted role-playing scenarios that can answer your client’s or stakeholder’s questions about a Web site.
Example: Testing a News Release Search Engine
I once participated in usability testing for a corporate Web site, in which one of the goals was to test the effectiveness of a search engine in searching for the company’s news releases. We crafted well-written tasks in which we asked participants to find specific news releases. Everything went smoothly and the client was happy. The issues we identified allowed us to provide actionable recommendations for improving the presentation of the search results.
One of the tasks we gave to participants was to search for a news release the company had published about a specific product, on a specific date two years before the test. The search engine provided many pages of results. The only way to find the particular news release was to guess what page of results would include the news release from that specific date and randomly click a results page number at the bottom of the page. Finding the correct news release was a long and difficult process.
Our recommendation was obvious: We needed to let participants browse the results by year and month! Therefore, we designed a well-crafted, chronological sorting menu. Everything seemed perfect. However, six months later, an external firm evaluated the Web site and pointed out that “the navigation by month and year was heavy and doesn’t necessarily match users’ needs.” Then, I asked myself: What if that firm is right and searching by month is not one of our users’ primary goals? I realized we had based our new design entirely on a single task the client had provided, and I didn’t know whether this task was representative of the way users look for news releases. It might be—but it might not. Maybe, our conclusions weren’t far from the truth, but we hadn’t checked our assumption and had no way of knowing exactly. We could have been totally wrong and based the design on a false assumption, which nobody had ever challenged.
In this case, we had made the mistake of not checking whether the tasks our client gave us were actually the tasks the key users of the Web site performed in real life. While our mistake appeared obvious afterward, when you have to craft a usability test for a client on a tight schedule, it can be easy to overlook this step. While the consequences of neglecting to determine the appropriate tasks to test might be less critical in the case of an informational Web site, for a shopping Web site, this kind of mistake can be really damaging to the bottom line. Jared Spool gives a striking example: “The design recommendation seemed solid, yet sales had dropped 23% immediately after the changes were made.” [2]
Devising Realistic Tasks
There are several possible solutions to this problem, as follows:
- investigate users’ needs and tasks
- gather data about users’ real tasks
- tailor task scenarios to participants’ real tasks
- ask participants to actually purchase products during a usability test
The following sections describe them in detail. You can use these approaches alone or in combination with each other.
Investigating Users’ Needs and Tasks
Ideally, you should thoroughly investigate the needs and tasks of users beforehand, through extensive interviews or field research, and devise your tasks based on the knowledge you gain. Then, you can be sure your tasks represent users’ real behaviors and needs.
Gathering Data About Users’ Real Tasks
If you can’t conduct extensive user research, you can gather data about users’ real tasks through alternative means, then use this data to adapt your tasks to users’ real needs. To obtain this data, you can do any of the following:
- Ask your stakeholders or clients. They sometimes know more than they think they know.
- Get data from Customer Service.
- Determine whether the company has conducted any previous studies, including market research or surveys. It is amazing the number of reports that lay dormant within organizations, and we reinvent the wheel again and again, because we don’t know these reports exist.
- Look at some external literature. Others might have conducted studies on what users look for in a similar context.
- Talk to some people you know who might fit user profiles—for example, coworkers, friends, or family members. This is not ideal, but in situations where you badly lack data, doing this might be helpful.
Tailoring Task Scenarios to Participants’ Real Tasks
You can collect information about participants’ real tasks during test sessions and tailor the task scenarios to each participant as you go. Ask participants to do something they actually need to do with your product—or to do something they’ve done recently—while you observe what they do.
Example: Testing a Yellow Pages Web Site
When testing a Yellow Pages Web site, I included a free-form task at the beginning of each test session—which I’d tailored to the needs of specific participants—before asking participants to perform the tasks I had prepared for them in advance. I asked participants to think of something for which they needed to search using the Yellow Pages and to search for it. Because participants were searching for something they really needed, they were totally immersed in the task—they did not have to pretend. The task was real, not artificial. I obtained very rich and unexpected insights from what I observed—about the quality and relevance of the search results and the discoverability and usefulness of the results refinement menu.
Depending on your situation, you can use variants of this approach to make tasks more realistic, as follows:
- Mix free-form and predefined tasks.
- Ask participants to think of some tasks they actually need to perform on a Web site before the session, as a homework assignment.
- Follow Jared Spool’s Interview-Based Tasks approach [2], in which you add 30 minutes at the beginning of a test session for interviewing a participant about his passions, then create the test tasks with him, based on his passions and interests.
If project constraints don't let you tailor your task scenarios to each participant, you can either
- dedicate the first five minutes of a session to investigating participants’ current goals and tasks on a site—If appropriate, you can try to adapt your predefined tasks slightly. (See Guideline 6.)
- investigate the realism of each predefined task—For example, when presenting a task to a participant, ask: “What is your experience doing this kind of thing?” “Is that something you would do?” “What do you do in general?” If you can’t do more in-depth user research, this can, at least, help you evaluate a participant’s engagement with the task and, therefore, the reliability of your findings, based on each participant’s behavior or reaction when performing a task.
Asking Participants to Purchase Products During a Usability Test
If you are testing an online shopping site, you can ask participants to actually shop for a product. Ask participants to search for and buy a product they need or want during the test session. You can give them credit card information to use when making the purchase. Each participant’s incentive is the product he or she will buy. If a task is one a participant would typically perform using the site, this technique helps add more realism to the way the participant engages with the site.
3. Test using content that’s meaningful to participants.
Sometimes—because of either development constraints or the unfamiliarity of your stakeholders with usability testing—you might end up in a situation where you are asked to perform a test using content participants wouldn’t use in real life or even comprehend. For example, they might want you to ask participants to evaluate a description of an IT position, when participants don’t know anything about IT and would never look for such a position in real life.
It’s common to encounter such situations—particularly when testing mockups or prototypes for which only a small sample of content or data is available. Sometimes, the only content you have isn’t the type of content participants would use in real life. However, it’s important to ensure the content you use when testing is as relevant to participants as you can make it.
Example: A Job Description That Doesn’t Apply to Participants
When I was testing a job search Web site, I needed to test the relevance of the search results and the job descriptions to participants. The primary stakeholder had prepared a series of fake mockups, presenting an IT position, and wanted participants to give feedback on them. He wanted answers to the following questions: How would participants choose a position they were interested in on the search results page? What kind of content would be useful to them on the description page?
Since our participants matched a very general profile and did not work in IT, it was impossible to ask these questions of participants who were unfamiliar with IT positions and would never look for this type of position in real life. How could somebody who would never look for such a position put himself in the shoes of somebody who would and select an appropriate job? You can’t simulate knowledge you don’t have.
However, the stakeholder was anxious to have feedback on specific listings and descriptions. To minimize bias and answer the stakeholder’s questions, I mixed free-form searches on the live Web site—where a participant could search for a position in which he was really interested—with predefined searches using the mockups. For the predefined searches, we chose a type of position anybody could understand—like a writer—rather than choosing a specialized position that required very specific knowledge like IT. Conscious of the limitations of our approach, after conducting the study, I interpreted the results carefully.
Tips for Choosing Meaningful Content
How can you ensure content is meaningful to participants?
- Don’t conduct usability tests using content participants would not use at all in their real lives.
- Adapt your test tasks to specific participants on the fly, showing them content they understand and are familiar with. Either at the beginning of a test session or before a specific task, ask participants what content they’re interested in, then tailor the predefined task by using that content.
- If you absolutely must test specific content or pages, mix free-form tasks that are tailored to each participant with predefined tasks to minimize bias. Ensure a page you’re testing is at least understandable to participants and be aware of its limitations.
