Wrangling Difficult Usability Test Participants

Untalkative Participants

While test participants vary in terms of how talkative they are, some just don’t talk much at all. When you ask them questions, they give you one-word answers. Despite all your attempts to draw them out by asking additional questions, they resist.

How to Handle Untalkative Participants

The best way to draw out untalkative people is by asking open-ended, follow-up questions. For example, ask “What do you mean by….” or “Why do you say it’s good? What, in particular, do you like about it?” When you respond to their brief answers by asking follow-up questions, they’ll eventually get the point that you’re going to keep asking them follow-up questions. Sometimes that gets them to provide more in-depth answers.

If they have difficulty answering your open-ended questions or elaborating on their original answers, ask more specific questions. If they still don’t open up, simply accept that these participants are reserved. Allow them to focus on performing the tasks and observe their behavior. There are quiet people in most user groups, so you shouldn’t eliminate them from your studies.

Overly Talkative Participants

At the other extreme are participants who talk too much. At first, they may seem like great participants. They answer your questions and give you a lot of detail. However, if they go on and on beyond what you need to hear, their answers will take up too much time, eventually making it necessary to skip tasks or questions.

How to Handle Participants Who Are Too Talkative

If participants are talking a lot and giving you really good information, you might just let them keep talking, even though that means you’ll have to skip some tasks or questions later. Otherwise, you should try to steer them to the next task or question. Subtly remind them of the time remaining, “We have only about 25 minutes left, and we have a few more tasks to get through. I don’t want to take up too much of your time.” Be careful not to make them feel bad about talking too much. They might be offended and stop talking completely.

Participants Who Ramble Off-Topic

Closely related to the person who talks too much is the person who talks a lot, but frequently rambles off topic. When someone talks too much, but stays on topic, at least the information they provide is relevant. However, when a participant rambles on about unrelated topics, that’s a waste of time and isn’t productive.

How to Handle Rambling Participants

Carefully, try to steer such participants back to a relevant topic. One way of doing this is to ask them a question that’s a natural transition back to the original topic. Another approach is to remind participants that you have limited time and need to get on to the next task. If appropriate, tell them that you’ll have some time at the end of the session to get their comments about additional topics.

Inarticulate Participants

People vary in their ability to articulately express their thoughts. Ideal participants can effortlessly express their thoughts in succinct sound bites. They give you perfect quotations, without any wasted words, and natural video clips. At the other end of the spectrum, there are people who have difficulty expressing themselves. They need to take more time to gather their thoughts. When they do speak, they take longer to get each of their points across—and they may also take a while to get to the point of interest to you.

How to Handle Inarticulate Participants

When dealing with inarticulate participants, be patient and get what you can from them. If their answers to your questions take too long and don’t provide enough value, focus more on their behavior and task performance. Limit the number of open-ended questions that you ask, instead asking more direct questions to help you understand their behavior.

Participants Who Struggle to Think Aloud

Having participants think aloud as they perform tasks during a test session is a great way to understand what they’re thinking and trying to accomplish at each moment. However, some people have difficulty thinking aloud, finding it unnatural and feeling uncomfortable about it. For other participants, thinking aloud interferes with their ability to focus on a task. Some people repeatedly forget to think aloud, despite frequent reminders.

How to Handle Participants Who Can’t Think Aloud

At the beginning of test sessions, demonstrate to participants what you mean by thinking aloud. Then, try to encourage them to think aloud by giving them reminders if they forget. But, if you find that participants still have a hard time thinking aloud, despite your giving them a few reminders, allow them to skip thinking aloud and just focus on the test tasks. You can ask questions at specific points to get a sense of what they’re thinking or trying to do. However, if you find that asking questions distracts them, save the rest of your questions till they’ve completed a task. Some people just can’t easily think aloud, so it’s better to let them simply perform the tasks and ask your questions afterward.

Participants Who Have No Opinions

Although usability testing primarily concerns gathering usability metrics such as task-completion rates, error rates, and time on task, you’ll usually want to get participants’ opinions of a user interface, the tasks, and the problems they encounter as well. Although most participants don’t hold back on sharing their opinions, some participants don’t seem to have any opinions at all. From people who have no real opinions, you might get responses such as the following: “It’s okay, I guess.” “It’s fine.” “It’s pretty good.”

How to Handle Participants with No Opinions

When people give brief, lackluster opinions such as these, ask them open-ended, follow-up questions to get them to expand on their remarks. Here are some examples of follow-up questions for particular responses:

• “It’s okay.”—Why do you say that?
• “It’s pretty good.”—What do you think is good about it?

If asking follow-up questions doesn’t work, try soliciting these participants’ opinions closer to when the corresponding interactions take place. For example, ask their opinion of a problem they encountered right after it occurs. People are more likely to express their true opinions in the midst of an event than when reflecting back on it later in a test session.

To head off this problem earlier in your process, make sure you recruit people who are likely to have opinions about the product you’re testing. For example, if you’re testing a car manufacturer’s Web site, recruit people who are currently shopping for a car. People in that situation can relate better to the task and will be more opinionated than someone who hasn’t bought a car in years.

Uncritical Participants

Some people just don’t feel comfortable providing criticism. Perhaps they want to get along with and please everyone, or maybe they don’t feel comfortable complaining to a stranger. Others may be uncritical because they don’t want to admit they had difficulties performing a task.

You can easily recognize these participants because, when you ask them how they felt about a task—after observing them struggling through the task, experiencing all kinds of problems—they’ll say something like: “That was good.” “It’s fine.” “It was pretty easy.” If you probe further by asking them about the problems they had, they might say something like: “Yeah there were a few problems, but it wasn’t that bad. Once you get used to it, it’s fine.”

How to Handle Uncritical Participants

To prevent this behavior, at the beginning of a test session, emphasize that the purpose of the test is to find problems with the design. You want to find problems. Then, you’ll fix them. Tell participants that you want their feedback—both positive and negative. If you aren’t the designer, point that out so they’ll feel more comfortable providing criticism.

If participants start giving you uncritical answers, try asking them specific questions about the problems they encountered. If you still can’t get them to be honest about their difficulties, focus more on observing what they do rather than on what they say. When you do your analysis of the session, put more emphasis on the actual problems they experienced than on their uncritical assessment of those problems.

Participants Who Blame Themselves

Some participants blame themselves for the problems they encounter rather than blaming the system you’re testing. They may get flustered and embarrassed. They may not express this self-blame aloud, but you can often sense it.

How to Handle Participants Who Blame Themselves

Up front, tell participants that the whole purpose of usability testing is to find design problems. Immediately after they encounter problems, reassure them by telling them that the problems they’re experiencing reflect faults in the design. Emphasize your user-experience philosophy: technology should be designed around the needs of the people who use it.

Sometimes I find myself wanting to say, “That’s okay. Most people have had the same problems.” I want to reassure them that their experience is normal if nearly everyone has had exactly the same problems. But doing this can be tricky because you don’t want to introduce bias. However, I think it’s okay, once someone has completed a task, to give a flustered participant this kind of reassurance. That’s preferable to a participant’s thinking he or she is a failure.

Uncooperative Participants

Most participants are friendly and cooperative. They put effort into performing the test tasks and try to answer your questions. They’re helpful and have a good attitude. After all, they volunteered to participate, they’re often getting paid, and sometimes they really want to help improve the product.

But occasionally, you’ll encounter uncooperative participants who don’t cooperate with you, perform tasks half-heartedly, and don’t have a good attitude. It seems as if they don’t really want to be there. They just want to get the session over with as soon as possible. When you ask such participants questions, they’ll give the shortest answers possible. For example, if you give them a task to find something on a Web site, they might say, “I wouldn’t normally do that.” Or, after putting very little effort into the task, they’ll quickly give up and say, “I’d just search for it.” Or “I’d call support.”

How to Handle Uncooperative Participants

When participants don’t want to play along, show them that you won’t let them off the hook that easily. For example, if you give them a task and they say, “I would just search for it,” in response, say, “It’s good to know what you would normally do, but just for the purposes of this test, show me what you’d do if you couldn’t search.”

If participants quickly give up on a task, ask them nicely to continue, saying, “Let’s say you wanted to keep looking, show me what you’d do next.” If you persist in gently prodding them along, you’ll show them that they can’t get away with just giving up on the task. They’ll see that it’s quicker and easier to try to accomplish the task rather than give up and have you pester them.

Just as with uncommunicative participants, follow up their brief answers by asking open-ended, follow-up questions such as: “Why do you say that?” or “What do you think is good about it?” They’ll get the point that, if they keep providing superficial answers, you’ll keep prodding them with follow-up questions.

Set a Positive Example

While perfect usability test participants are very rare, so fortunately, are difficult participants. You’ll encounter difficult participants only infrequently. If you recruit well, most of your test participants will be very helpful and friendly.

Nevertheless, as a user researcher, you need to be able to work well with all kinds of people. The quality of any usability-test session is mostly up to you. Even when working with difficult participants, you need to get the best possible data from each session. Although I’ve categorized different types of difficult test participants in this column, don’t put real people in categories. Try to approach each test session as a new experience with a unique individual. Ideally, the positive attitude you bring to a session will bring out the best in each test participant.