Several years ago, I wrote a UXmatters column titled “Wrangling Difficult Usability Testing Participants,” which described ten types of challenging participant behaviors and how to handle them. Since then I’ve come across enough additional types of difficult participants to write more about this topic. Now, in Part 2, I’ll cover nine more types of difficult participants and describe how to handle them.
As I wrote in my earlier column, most usability-testing participants are typical people who find themselves in the unusual situation of participating in usability testing—something that is usually a new experience for them. They really try to be helpful, and you can easily work with them to get the information you need. Unfortunately, you might sometimes come across some difficult participants. However, there are fortunately ways to prevent or overcome the problems that you encounter with them. Now, in Part 2, I’ll provide more tips about how to wrangle difficult participants effectively and get the most out of their usability-testing sessions.
Champion Advertisement
Continue Reading…
Happy Clickers
Happy Clickers—or Happy Tappers, for those on mobile devices—are all too ready to click anything they see on the screen. However, in usability testing, you might not want participants to click until you’ve finished asking them questions about the current screen. For example, you might want to ask them what they would expect to happen if they clicked a particular link or button. When Happy Clickers click something before you want them to, they see what happens and discover what really happens before you can ask them the question. So you can’t learn about what they expect would happen.
For example, if you were to ask, “Where would you go if you wanted to change your email address?” the Happy Clicker would immediately click the Profile icon and see what’s on the subsequent page. Therefore, you no longer have the opportunity to ask Happy Clickers your questions about their expectations of the Profile icon or what they would expect to find on that page.
How to Handle Happy Clickers
When it’s important to prevent participants from clicking things until you’re ready for them to do so, include a reminder at the beginning of your question. For example, instead of asking, “Where would you go if you wanted to change your email address?” say, “Don’t click anything yet, but please tell me: Where would you go if you wanted to change your email address?” Once you’ve said, “Don’t click anything yet, but…” many times, most Happy Clickers realize that they should wait before clicking anything. While you could initially give the instruction not to click anything until you tell them to, most participants won’t remember to refrain from doing that. So, when it’s important that participants not yet click something, build that instruction into each of your questions.
Talkers, Not Doers
The opposite of Happy Clickers are Talkers, Not Doers, who whenever you ask them questions tell you what they would do instead of doing it. Although you sometimes do want participants just to tell you what they would do, at other times, you want them to show you what they would do. For example, you might ask, “Let’s say you want to cancel your order. What would you do?” Talkers, Not Doers would say, “I would click the Cancel link and that would cancel my order.” So you then have to ask them to show you how they would do that.
How to Handle Talkers, Not Doers
When you want participants to show you how they would do something, include that instruction in your question. For example, “Show me what you would do if you wanted to cancel your order.” As with Happy Clickers, participants can’t read your mind. They don’t know when you want them just to talk about something or actually click something—unless you tell them. So make your questions and task instructions clear regardless of what you want participants to do—whether that means telling them, “Don’t click anything yet, but….” or “Show me what you would do….”
Givers of Facts, Not Opinions
When you ask some participants for their opinions—those who are Givers of Facts, Not Opinions—they’ll just give you facts about what’s on the screen instead of providing their opinions. For example, if you were to ask, “What’s your opinion of the information on this screen?” the Giver of Facts, Not Opinions might respond, “Well, it has the current balance and the statement balance. It tells you the due date. It has the button to make a payment. You can click to see transactions….” They go on like that just describing the facts about what is on the screen, but never giving you their opinions about it.
I’m not sure why some participants do this, but I’ve seen it many times. I suspect that some people do this when they don’t have a strong opinion about something, one way or the other. They begin by talking through what they see on the screen as a way of thinking aloud, as they try to figure out how they feel about it, then eventually forget to provide their opinions.
How to Handle Participants Who Give You Facts, Not Opinions
Allow these participants to talk about what they see on the screen. Doing that might help them formulate their opinions. Then, if they finish their thoughts without providing an opinion, follow up with a reminder such as, “So how do you feel about the information on this screen?” With some participants, you might need to ask a more specific question such as, “How helpful or unhelpful is the information on this screen to you?” If they give you a halfhearted answer such as, “It’s pretty helpful,” probe further by asking, “What do you think is helpful about it?” Although you may have to press these participants to get their opinions, remember that people don’t always have a strong opinion about something—positive or negative. So, if that seems to be the case, let them be.
Representatives of the Business
During usability-testing sessions, some participants step out of their role as the user or customer and become Representatives of the Business, providing advice about what’s best for the business behind the user interface they’re evaluating. I saw this recently when testing online ads: Instead of talking about the noticeability of the ads, whether they found the ads annoying or acceptable, or whether they found them relevant or irrelevant to their needs, some participants talked about how the designer could make them even more noticeable and, thus, effective for the advertiser. They recommended changes to the ads that would make them work better for the advertiser, even though, as a customer, they would find the resulting ads extremely annoying.
How to Handle Participants Who Represent the Business
When participants begin to give recommendations for the business, remind them that you want to hear about their opinions and perspectives as users or customers. If they’ve given you business-focused recommendations, ask them how those changes would affect them as a customer and how they would feel about them.
Participants Who Take Prototypes Too Literally
Although you’ve probably explained, at the beginning of a session, that a prototype is just an early and incomplete version of a design, some Overly Literal Participants won’t understand that or might get hung up on insignificant aspects of the prototype. Some don’t understand the concept of a low-fidelity prototype and might assume, sometimes with horror, that they’re seeing the final product. You’ve probably heard people say, “It’s not going to be in black and white, is it?” Even though you’ve told them that not all links or other aspects of the prototype are currently working, they might click things, then not understand why they’re not working, giving them a negative impression of the user interface. Some participants get hung up on insignificant details of a prototype such as Lorem Ipsum text, generic placeholder images, or unrealistic data.
How to Handle Participants Who Take Prototypes Too Literally
Instead of explaining the nature of a prototype at the beginning of the session, give that explanation just before you show the prototype to the participant. Often participants either won’t listen closely or won’t remember an introduction from the beginning of the session. Your explanation of the prototype would be just one of many things you go over in the introduction, so it’s better to save this explanation for later in the session, just before you show the prototype. Tell participants that it’s an early design so some elements and links are active, while others are not and that the prototype has some placeholder elements in it. Then, as they begin using the prototype, if they notice or try to use those nonfunctional elements, remind them: “Yes, that’s just placeholder text.” Or, if they click something that isn’t yet working, tell them that’s one of the things that hasn’t yet been built into the prototype, but would normally work in the final product.
Professional Research Participants
Some people have participated in so much user research that they’ve almost become Professional Research Participants, who actively look for studies in which they can participate to supplement their income. That’s not necessarily a bad thing—as long as those participants are honest and actually meet your recruiting criteria. However, some people deliberately lie their way through screeners to get into as many research studies as possible.
Sometimes you can tell they’re Professional Participants because they obviously don’t fit your criteria. Or you might detect that they already know your research lingo and what to expect. For example, I had participants ask me, “Do you want me to think aloud?” I know one researcher who had a participant stop the introduction and offer to give it.
How to Handle Professional Research Participants
Include a question in every screener that asks potential participants about the last time they participated in a UX or market research study. Recruiting companies should keep track of this information themselves to avoid contacting people who have recently participated in studies and double-check participants’ answers to ensure they’re being truthful.
Write the questions in your screener in a way that doesn’t reveal the right answer for the type of participant for which you’re looking. This makes it more difficult for people to lie to get into a study. For example, instead of asking, “When do you think you might begin shopping for a new car?” and giving them date ranges from which to choose, ask a question that disguises what your research is about. For example, “Which of the following items are you considering purchasing within the next six months?” Then provide a long list of products, including a new car, as answers to disguise which type of customers you want. Disqualify people who unrealistically choose all or nearly all of the items in the list to try to get into the study.
Uncomfortable, Nervous Participants
Participating in user research can be a strange experience for most people, and it’s natural for participants to feel somewhat uncomfortable or nervous. Going into a lab or focus-group facility and being observed and questioned can feel awkward and intimidating. Having several people come into your home or workplace is not the most natural experience either. So participants’ feeling some nervousness is understandable. However, you might sometimes encounter especially Nervous Participants who never seem to become comfortable.
I once had a participant who seemed slightly nervous when I greeted him and escorted him to the lab. By the time he saw the usability lab with its one-way mirror, he was sweating profusely and seemed almost ready to panic. He apologized and said he just couldn’t do the session. I think he may have had a social-anxiety disorder. I knew this wasn’t the typical, slightly nervous participant, so I told him it was no problem, took him back to the front desk, gave him his incentive check for coming to the session, and used a floater participant instead.
How to Handle Uncomfortable, Nervous Participants
For all participants, it’s important to begin their session with a friendly introduction, explain what they’ll be doing, and make them feel comfortable. It’s helpful if you can make yourself seem like a friendly person who is similar to them. If participants are especially nervous, it sometimes helps to acknowledge the strangeness and awkwardness of a research session. Then ease them into the session with easy, get-to-know-you types of questions to help put them at ease.
However, if participants are still extremely nervous and uncomfortable despite all your efforts, you may need to let them go—for their own benefit and yours. End the session, thank them for their time, and use a floater or alternate participant instead.
Participants Who Are Too Relaxed
With nearly all UX research being conducted remotely since the beginning of the COVID-19 pandemic and participants joining usability-testing sessions from home, some participants might be far too relaxed. Most participants still take their sessions seriously, but sometimes you’ll get Overly Relaxed Participants who do things they would never do in person. For example, I’ve done sessions with participants who were lying down in bed or on a couch, wearing inappropriate clothing, eating during the session, and even smoking pot.
How to Handle Participants Who Are Too Relaxed
To help ensure remote-research participants take their session seriously, near the end of the screener, include a question asking whether they are able to participate in the study in a professional manner—seated rather than lying down, wearing appropriate clothing, not eating during the session, and not being under the influence of alcohol or drugs. You would think you wouldn’t need to ask such a question and, for at least 95% of participants, that’s not necessary, but including this language in your screener helps set their expectations for a professional session.
If you still get participants who are too relaxed, don’t be afraid to ask them to sit up or refrain from ingesting anything during the session. Yes, it can feel awkward to have to mention this, but you can use the excuse that it’s hard to see them or hard to hear them. If they’re doing something truly inappropriate or they won’t modify their behavior, end the session and use a floater or alternate participant instead.
Harassers
Lastly, the worst kind of participants are those who harass the researcher. Fortunately, I haven’t experienced any Harassers myself, but I’ve heard from female colleagues that harassment by male participants happens occasionally. In addition to sexual harassment, there are many other forms of harassment—for example, those that are based on race, ethnicity, or sexual orientation. Fortunately, this is very rare, but it does sometimes happen.
How to Handle Harassers
No one should have to deal with harassment. In this situation, promptly end the session, and use a floater or alternate participant instead. If necessary, you can blame your ending the session early on a technical problem. But, if you feel uncomfortable doing that, you can just disconnect a remote session without any explanation. During an in-person session, you can leave the room with the excuse that you need to check on something, then if you don’t feel comfortable dismissing the participant yourself, get the help of someone else who can come in and do that for you.
Most Participants Are Good People Trying to Do Their Best
When you do occasionally come across especially bad participants—for example, professional research participants who lied to get through the screener or participants who behave inappropriately or harass you—let the recruiting company know in detail what has happened so they can put those people on a do-not-recruit list. This would prevent your encountering these same bad participants again and help other researchers avoid getting these bad participants.
The other types of participants I’ve described in this column are not bad participants; they just need guidance in terms of what kinds of behavior and feedback you need. Fortunately, most participants are good people who are trying to be helpful and provide good feedback. Difficult participants are rare and, with experience, you can learn how to work with them effectively to get what you need. For most people, participating in usability testing is something new and unfamiliar. So it’s up to you to put your participants at ease and guide them to do what you need to them to do. If you’ve found this column helpful and want to learn about some other types of difficult participants, check out Part 1, “Wrangling Difficult Usability Testing Participants.”
Jim has spent most of the 21st Century researching and designing intuitive and satisfying user experiences. As a UX consultant, he has worked on Web sites, mobile apps, intranets, Web applications, software, and business applications for financial, pharmaceutical, medical, entertainment, retail, technology, and government clients. He has a Masters of Science degree in Human-Computer Interaction from DePaul University. Read More
