The Moderator’s Survival Guide: Handling Common, Tricky, and Sticky Situations in User Research

2.2 Resolve any threats to physical safety

In the middle of a contextual inquiry, you hear a low rumbling sound. You take a moment to evaluate the situation, and quickly realize that you’re experiencing an earthquake. What do you do next? (For more about how to handle earthquakes and other natural disasters, see section 14.2.)

When a situation involves a physical threat, you need to respond to and address the threat as soon as you’ve taken a couple of seconds to understand what’s going on. Respond quickly and appropriately, even if that means that you’re unable to continue the session. The safety of you and the participant takes priority over your data collection needs!

Keep in mind that the participant may not know what to do, especially if he is in your facility. You must take responsibility for his safety as well as your own. For example, if a fire alarm goes off or your building is evacuated, bring the participant with you as you follow standard evacuation procedures. While the participant may end up seeing your observers, your top priority is to keep him from physical harm instead of maintaining the integrity of the session.

In the example of an earthquake, you should instruct the participant about where to go—for example, under a desk or in a doorway. If you’re moderating a usability study from an adjacent control room, and you think it’s safe to spend another couple of seconds using the microphone, tell the participant that you’re experiencing an earthquake and instruct him to go under the desk or in the doorway immediately.

If you’re in the participant’s space, follow his lead to exit the building safely or follow the appropriate safety protocol for the event that you’re experiencing.

Depending on the severity of the situation and safety of the environment, you may want to cancel or end the session rather than asking the participant to wait around until the situation is resolved. If you do this, be sure to give him his full compensation and thank him for the feedback provided. If the environment is safe, and the participant is not visibly shaken and willing to continue, offer to take a short break before resuming so you and the participant can both regain focus.

2.3 Verify that you’re not causing or magnifying the situation

During an interview, the participant starts making snide remarks after each question you ask. You’re not quite sure why he is responding that way. The issue does’t resolve itself, and taking a break does’t help. There is no physical danger to resolve but you’re worried about the participant’s attitude and its effect on the feedback he is providing. What, if anything, should you do?

Let’s step into some awkward territory for a minute, but we promise, it’s for your own good! Sometimes a situation is magnified, if not created, by something that you’re doing. Part of being a good moderator is being humble enough to constantly evaluate and reflect on how you’re interacting with participants. Before you jump to a conclusion about why the participant is behaving in a certain way, try to detect anything you may have done that led to this behavior.

Table 2.3 shows some examples of problematic moderator behavior that we’ve observed, the moderating styles (as identified by Table 1.1) that may encourage this behavior, and the effect it can have on the participant’s comfort level.

Table 2.3—Examples of Problematic Moderator Behavior and Its Effects

Moderator Behavior (Moderating Style)		How the Participant May React
Over-the-top fake (Friendly Face)	You’re trying to be very kind and respectful toward the participant while maintaining a friendly tone. However, you don’t realize that you come across as patronizing or disingenuous.	The participant becomes irritated and curt with his feedback. He may feel like you’re not taking his responses seriously.
Buddy-buddy (Friendly Face)	You want the participant to be comfortable, so you start acting like the two of you are old pals. You laugh together and find yourself making jokes. He tries to make you laugh and looks to you for affirmation, as with any natural friendly exchange. For an example of what this may look like, see Figure 2.1.	The participant begins to try to please you, unknowingly reflecting your attitudes and biases. His efforts to please you throw him off his natural course of interaction and keep him from answering questions fully and honestly.
Obliviously abrasive (Down to Business)	You maintain a serious, expectant tone as you ask questions. However, your tone comes across as overly stern. You also stare intently at the participant without looking away or blinking while he talks. For an example of what this may look like, see Figure 2.1.	If the participant is naturally talkative and high profile, he may feel insulted or put off by your serious tone. If he started out nervous, your tone may only increase his discomfort. The intent staring may unnerve him to the point of asking to leave.
Death by silence (Down to Business; By the Book)	You’re trying to be as neutral as possible so you take a very minimalistic approach to moderating. You’re completely silent most of the time, using only brusque probes—for example, “And…?”—even though the participant is obviously nervous and uncomfortable with the silence.	The participant starts to blame himself and apologizes for “doing a bad job.” Because you don’t respond, he becomes more uncomfortable.
Clock watcher (By the Book)	class="last-row" You’re trying to stay on schedule, and look at the clock or your watch at the most inopportune times. You ask a question from your study plan and, while the participant answers, look at the clock, look ahead in the study plan, or obviously pay no attention to the participant’s answer.	The participant gets frustrated and makes sarcastic or snide remarks because he does’t feel like you’re listening to him.

As discussed in Chapter 1, there is a delicate balance between the goals and method of your study, the type of participant you have, his current emotional state, and the appropriate moderating style to use. These last two elements in particular can change multiple times throughout the session. You should always carefully monitor how the participant responds to avoid making him uncomfortable. Again, part of the art of moderating is in evaluating your technique throughout the session and adapting it as necessary.

© 2013 Mark Ainscow, used with permission.

To find out if you’re creating or contributing to a problem:

1. Check your body language:

• Are you too close to or too far from the participant?
• Are you maintaining eye contact, or are you making too much eye contact?
• Have you been yawning or glancing at your watch?
• Are you sitting upright, or are you slouched down in your chair?
• Have you been frowning or grimacing instead of maintaining a friendly, neutral expression?

2. Replay what you’ve said during the session:

• Have you neglected to reassure the participant that you’re listening to him? Have you been chiming in with the occasional “mm-hm” to encourage him to continue talking?
• Have you been talking too much, and not letting the participant talk?
• If running a usability study with a think-aloud protocol, did you give the participant enough time to respond before reminding him to think aloud? (See the sidebar “Learning to Enjoy the Silence” for more on this.)
• Did you stop using the participant’s language and start using language from the product’s user interface, domain, or any other jargon he may not understand?
• Are you using close-ended or leading questions that “give away” the answer you want or expect to hear?
• Have you been laughing with or making jokes with the participant?

3. Replay your paralanguage—how you said what you said:

• Did you sound judgmental about an opinion or detail that the participant shared?
• Did you insinuate that the participant did something incorrectly?
• Could your tone be heard as patronizing or disingenuous?

4. Evaluate your note-taking behavior:

• Are you shuffling papers while the participant attempts tasks or answers your questions?
• Have you been taking notes only when the participant says or does something “wrong” or unexpected?

5. Check the physical setup of the room:

• Is the temperature too hot or too cold?
• Is there anything in your equipment setup that the participant seems to be struggling with—for example, a monitor that is pushed too far back on the table, or a chair that is set too high or too low for the participant?

We’ll talk more about how to build awareness of your own behaviors while moderating in Chapter 16.

2.4 Check the participant’s comfort level

Once you’ve verified that the situation you’re seeing hasn’t been caused or exacerbated by your own behavior, check to see if the participant’s degree of comfort has declined to an unacceptable level.

Most user research starts out being somewhat uncomfortable for the participant. He may arrive without knowing exactly what he’ll be asked to do, or he may worry that he is being tested in some way. If you are coming in to his space, he’ll be unable to get away from the typical hassles of his environment, which may become magnified by the uncertainty of what he’ll be asked to do during the session.

Unless you’re doing very specific research that deliberately puts users in uncomfortable situations, you want to minimize, if not altogether eliminate, anything that compromises the comfort or safety of the participant. Part of your job as a moderator is helping the participant feel comfortable participating and providing his feedback, and ensuring that he remains comfortable—physically and psychologically—during the session.

Ensuring the participant’s physical comfort is usually fairly easy to accomplish. Let him know that he is welcome to take a break at any time and, if possible, offer him a beverage. If he seems uncomfortable in the research space—for example, because the chairs are at an awkward height or size for him or the air conditioning is running high—make whatever adjustments you can to keep the environment as welcoming and stress-free as possible.

The participant’s psychological comfort can be more challenging to maintain, especially when he is asked to interact with a product that has major usability issues or he is being interviewed about a sensitive or private situation. However, keep in mind that if a participant is struggling or growing frustrated with a usability study task, his comfort is not necessarily being threatened. In fact, that struggle often fulfills a common goal of user research, which is to identify problem areas. The participant’s struggle serves the purpose of making the product better for future users.

If, however, the participant berates himself for being stupid or unable to figure something out, or exhibits more than one of the signs of an uncomfortable participant (Table 2.4), you need to alleviate his discomfort. Remember that a participant who leaves a session feeling disgruntled or unhappy is an unhappy customer or potential customer. Given how easy it is to share negative reviews and experiences via social media, you want to avoid any potential damage to the credibility of your research or your organization.

If the participant seems uncomfortable, be careful not to assign any blame to him. Remember that he is likely in an unfamiliar environment (to him), or may be stressed to show a stranger (you) around his comfortable environment. He may be under a lot of stress unrelated to the session, such as childcare issues or tight work deadlines. Even if there is nothing external causing his discomfort, the research session itself can create some stress no matter what you do to avoid it.

Table 2.4—Signs a Participant May Be Uncomfortable

He increases his physical distance from what’s being evaluated—for example, chair pushed away from the table or keeping a physical product at arm’s length. He crosses his arms. He blames himself or apologizes for being “stupid” or inexperienced. He has difficulty speaking clearly. He exhibits signs of emotional distress, such as tremors, crying, or excessive sweating. He constantly glances around the room or at the cameras. He asks to end the session early. He pays excessive attention to distractions—for example, his email is open in the background and he keeps checking it.

Watch Video 1 to see an example of a moderator interacting with a self-blaming participant during a usability study. The participant is very nervous about the session and is looking for affirmation.

Visit our Web site.

To help the participant feel more comfortable:

• Reassure him that he is not being tested, and that anything that frustrates him is valuable information for the product team. Remind him that his feedback will make the product easier to use for future users. For a contextual inquiry or interview, you can reassure him that his feedback is confidential and will ultimately be used to make his own life/work better.
• If you’re running a usability study, avoid using the term test when referring to your research. A test is something that can be failed, and may place additional pressure on the participant. He may not understand that you’re testing the product and not him. Instead, use study, session, or evaluation of <product>. Likewise, you may consider calling each usability task a scenario instead of task.
• If the participant crosses his arms or moves away from you—or the product being evaluated—ask him to tell you what he is thinking. This question gives him permission to express the reasons for his discomfort, and may reveal if he is uncomfortable because of something you asked him to do.
• Ask the participant if he would like to take a short break, or go ahead and suggest a break.

If the participant reaches a point where he asks to leave or end the session early, you’re obligated to wrap things up. Be sure to give him his agreed-upon compensation and thank him for his time and help. You can check on him afterwards—using whatever means you originally scheduled him for the session; if he was scheduled by a recruiter, ask the recruiter to follow up. If appropriate—for example, if he was distraught due to external circumstances—you can even offer to reschedule for a more convenient time.

2.5 Use careful language and tone to probe on the situation and begin to resolve it

Once you’ve established that you’re not causing or magnifying an issue, you’ll probably need to talk with the participant to try to understand more about, and hopefully resolve, the situation. One of the most important principles in moderating is to let the participant do most of the talking, but when you do speak—whether it’s to provide direction, probe for more detail, or give neutral reassurance—you need to choose your words, and your tone, carefully.

To choose the appropriate language, we suggest reviewing and memorizing some prefabricated responses that represent moderating best practices. Prefabricated responses are valuable for interacting with participants while maintaining neutrality and consistency. By memorizing at least some of these phrases and reviewing a list before your study, you save the cognitive effort of improvising appropriate responses on the spot. As we start reviewing specific situations in Part 2, you’ll be amazed at how flexible these short phrases can be.

The phrases we use most often include:

• “What would you expect?”
• “What are you trying to do right now?”
• “What are you thinking right now?”
• “How did this compare to your expectations?”
• “What are your thoughts on how that worked?”
• “That link isn’t working for today’s session. What would you expect to happen if you clicked it and it was working?”
• “Thank you for that feedback, but for the sake of time, please move on to….”

(For more examples of these phrases, see Appendix A.)

Since you need to be careful about providing positive feedback to participants, we’ve found the following phrases to be relatively neutral. When used judiciously, the phrases are capable of acknowledging to the participant that you’ve heard his feedback and are giving him a small amount of reassurance that the feedback was helpful:

• “Okay.”
• “Okay, thank you,” or “Okay, thanks for that.”
• “That is good to know,” or “That is helpful for us to know.”

Careful use of these phrases will address the majority of typical issues you see during your research. Just as important as what to say is what not to say, which we give examples of in the sidebar “What Not to Say.” Of course, you’ll encounter situations where these neutral phrases may not cut it, or if a participant does’t act as expected when given one of those phrases. In these cases, keep in mind these three guidelines:

• Be as unbiased and nonjudgmental in your response as possible. No matter how inflammatory or provoking the situation is, or how much you agree—or disagree—with what the participant is saying, it is your responsibility to remain neutral. Remember that it’s not just what you say, but also how you say it, both in intonation and body language. For example, if the participant is describing a personal or cultural belief that you find repugnant or inappropriate in response to your interview question, try to keep your face still and your voice at an even neutral tone as you ask your next question. (We discuss this kind of situation where a participant makes inappropriate comments in section 13.1.)
• Build on what the participant says rather than negating it. As mentioned in Chapter 1, one of the fundamental principles of improv is to never disagree with something that has been said, but instead to say “Yes, and…” This principle means that you’re always building on what someone else is doing rather than negating it. In a research session, your “Yes, and…” is less about agreeing with what the participant has said and more about acknowledging that he is been heard. For example, if the participant asks if he has performed a task correctly, respond with a prefabricated phrase instead of ignoring the questions. For example, “So I’ll actually turn that question around on you: Do you feel that you’ve found the information that you’re looking for?”
• Be as graceful as possible when redirecting or reacting. When the participant wanders off track or no longer provides useful feedback, your paralanguage will set the stage for his reaction. You don’t want to imply that he has done something wrong. Here’s one of our favorite examples of a moderator-participant exchange that we’ve seen, showing how your word choices and intonation can have a negative effect:

Moderator: How would you do <task>?

Participant: Oh, I would do <this>.

Moderator: (pause, intonation rising) Oh. You would?

The rising intonation implies a questioning of the participant, and may cause him to feel judged or that he should rethink his answer. Instead, a graceful reaction would be to say “okay” with a neutral tone and move on, or ask him if there are other things he would try if the first attempt didn’t work. If he is interacting with a prototype, you could also let him attempt the task, ask how it compares to what he expected, and then ask if there are any other ways he might try to do the same thing.

Table 2.5—Examples of What You Should Not Say While Moderating

What Not to Say	Why
“Let me give you a hint.”	Reinforces that the participant needs help.
“Why would you want to go there?” or “What made you do it that way?”	Implies that the place the participant wants to go, or did go, is not correct.
“Talk to me about… ”	Depending on how it’s said, or how much it’s overused, can sound commanding or condescending.
“No, don’t <do/click> that.”	Reinforces that the participant is doing something he shouldn’t.
“We/they did that because <special case could happen or special constraint limited us>.”	Implies that you were part of the design team, and/or are defending the design choices.
“That is great feedback.” or“That is good feedback.”	Implies that other feedback is not great.

2.6 Regain control to bring the session back on track

Hopefully, what you’ve done so far has jumpstarted the session. However, if the participant feels out of control or continues in a direction that is not useful to your research goal, it’s your job to steer him back. As the moderator, you’re always responsible for maintaining control of the session. This means accepting responsibility for changing the tone or providing whatever support is necessary to bring the session back on track.

Remember that you’re not the participant’s buddy. You didn’t set up this time with him for a gripe or therapy session. It’s your job to keep him from treating it that way. At the same time, some of the art of moderating includes redirecting a session and treating the participant’s feedback with grace and respect.

So how do you reestablish control of the session? For most one-on-one methods, a specific restatement or refocusing of a task or question is enough to get the participant back on track:

• “Thank you for going into that level of detail. For the sake of time though, I’d like you to return to <attempting/answering task/question>….”
• “Thank you for explaining that, it’s helpful for us to hear. We’re a bit pressed for time so I’d like to direct your attention to….”

You can also bring the participant to a different task or question if you think he may be getting stuck or hung up on something in particular. (The pattern for redirecting the participant is more fully explained in section 3.3.)

If your first attempts to regain control don’t work, you may need to use stronger language to rein the participant back in:

• “I apologize for interrupting, but it’s really important that I get your feedback on this <task/topic>. If you don’t mind, let’s return to that, and if there is time at the end of the session, we can return to this discussion in more detail.”

Also, check that your body language projects an appropriate authority level. Maintain eye contact with the participant and angle your body toward him—this will help draw his attention and increase the odds that he’ll pay attention to you. However, never touch the participant, even as a way to get his attention or give him comfort. Remember that many people do not like being touched by strangers and the inherent power imbalance within a moderating session can make any touch completely inappropriate. (We talk more about how to use your body language in section 3.5.)

If the session is veering out of control through no fault of the participant—for example, technical problems—avoid sharing your frustration with him and do your best to keep that emotion out of your voice. The participant may feel like you’re directing that irritation at him.

Another reason you may need to regain control is when the participant does not meet the desired criteria and you must adapt accordingly. One approach to this situation is to transition the session into a different type of research. This works especially well if the participant is an important customer, or part of a user group, who you want to feel validated and listened to. For example, a usability study with a customer who does’t match your target user profile may be transitioned into an interview that will provide more information about his profile. Be sure to discuss these options with your stakeholders so they know what’s happening and can provide input on other types of feedback they might be interested in receiving. If, however, the participant will be unable to provide any kind of useful input—for example, if his role is entirely unrelated to your target—you may need to politely end the session. This technique is also referred to as shifting the focus, which we discuss more in section 3.8.

No matter what situations you encounter, the steps discussed in this section should help you figure out the best approach to address and resolve them. When in doubt, think back to what you would want done if your positions were reversed—how would you want your moderator to handle the situation? Chapter 3 details some of the specific interaction patterns that you can apply to help with this resolution.

Discount for UXmatters Readers—Buy The Moderator’s Survival Guide: Handling Common, Tricky, and Sticky Situations in User Research and other Morgan Kaufmann titles on the Morgan Kaufmann site, using the discount code PBTY14, and save 25% off the retail price.