User Research Is Unnatural, Part II: Making User Research More Natural

Practical Usability

Moving toward a more usable world

A column by Jim Ross

April 18, 2011

Short of espionage or spending years living with a group of people conducting true ethnography, user research will always be somewhat unnatural. In Part I of this series, I discussed unnatural aspects of user research that can prevent us from observing realistic user behavior, including the

representativeness of the people we recruit
effect of incentives
location of the research
way we explain user research to participants
realism and meaningfulness of tasks
fidelity of prototypes
effect of thinking aloud
presence and behavior of a facilitator
presence of observers
prominence of recording technology

To minimize the negative effects of these unnatural aspects of user research and get more realistic results, there are many things we can do to keep user research as natural as possible. Now, in Part II of this series, I’ll discuss some of the things you can do to make your user research seem more natural.

Champion Advertisement

Be a Participant Yourself

There’s no better way to gain empathy for what participants go through during user research than to experience being a participant yourself. Volunteer for focus groups, usability testing, online studies, and any other research activities for which you can qualify. Of course, our profession often disqualifies us from being real study participants, so participating in a pilot test—or dress rehearsal for a research session—is a great way to help out our fellow researchers and get beneficial experience as a participant ourselves. You’ll be surprised at the insights you’ll get from being on the other side of the research table. Apply the lessons you learn to your studies.

Recruit Real People

Recruit representative participants from existing sources of actual users rather than professional research participants from recruiting company databases. Either recruit customers or employees from existing lists or recruit customers from your Web site, stores, or other organizations.

Don’t Recruit for Entertainment

Recruit participants who are representative of actual users, not just those who will impress observers by speaking in perfect sound bites. In his article, “User Testing Is Not Entertainment,” [1] Jakob Nielsen makes a good point about how screening for participant eloquence can eliminate quiet, introverted people who might be representative of a significant part of your user base. Including some of each type of participant gives you a more representative group and still provides enough interesting video clips.

Minimize the Formality

Establish an informal rapport with each participant at the beginning of a session before launching into the details of the study. When you do describe the study, continue in an informal tone rather shifting abruptly to legalese.

Dress Appropriately

What you wear sets the tone for the degree of formality for a research session and how participants perceive you. Dress appropriately according to the type of participant and the situation. Most often, it’s best to dress slightly more formally than participants. However, if you dress much more formally than participants, it sets a more serious tone, and they may feel self-conscious and uncomfortable. If you dress much more casually than participants, they may not take you as seriously and may feel uncomfortable.

Set the Right Expectations

The way you set participants’ expectations at the beginning of a session affects everything that comes afterward.

Explain Why You’re Doing the Research

Participants don’t naturally understand the purpose of our research. Make it clear that your philosophical allegiances lie on the side of supporting and empowering users of technology. It’s important to let participants know you aren’t testing or evaluating their performance. Especially in work situations, research can arouse suspicion that you’re observing employees for nefarious purposes on behalf of management.

Acknowledge the Unnaturalness

Openly acknowledge to your participants that certain aspects of your study might seem unnatural or awkward. For example, “Okay, yes as you can imagine, there are people behind that mirror. But they’re normal people like you and me.” Doing this shows you empathize with their potential discomfort. Humor and informality are great ways of defusing uncomfortable situations.

Use a Plain Language Consent Form

Consent forms should be brief, easy to read, and written in plain language, with a friendly, informal tone. Focus on the rights of the participant, not on their signing away their rights, and don’t let your legal department turn a consent form into a legal document.

Include a brief description of who you are, what your company does, and an introduction to your study. This minimizes the need for an explanatory introduction at the beginning of each session. You might summarize this information again in person, but you’ll have less to say if you’ve already introduced your study in the consent form.

Explain How You’ll Use What You Learn

Clearly explain how you’ll use the information you obtain from each participant, whether you’ll report their comments anonymously, and who will see the study results. Be especially careful to protect participants from peers or supervisors who are either observing the research or will read the results. This becomes especially important when your participants are employees, and co-workers will view the research results. You should take care to protect your participants from retaliation or any other negative consequences of their comments.

Be Clear About What Will Happen to Recordings

Explain that recordings are for notetaking purposes and only people working on the project will see them. It’s best to explain this both in the consent form and in your initial introduction to participants.

Make It Clear You Want Honest Feedback

Because some people are uncomfortable providing negative feedback, it’s important to reassure participants that you’re looking for their honest opinions—both positive and negative. If you are not the designer or your company did not design what you are evaluating, emphasize that fact, so participants know they can speak freely without offending you. If you are the designer, don’t emphasize that fact. Either way, make it clear that you are planning to make changes to the design, and the purpose of the research is to learn what changes to make.

Use Think Aloud When Appropriate

Although observing a participant thinking aloud while performing tasks doesn’t reflect natural behavior, it’s a great way of understanding what the participant is doing and thinking. While this does affect the way people perform tasks, in many cases, the benefits outweigh the disadvantages. If you aren’t recording time on task, completion rates, eyetracking data, or other metrics, having participants think aloud is an excellent way of gathering qualitative information.

Don’t Use Think Aloud When It is Important to Observe Natural Behavior

When it’s more important to observe realistic task performance and gather metrics such as time on task, error rates, and completion rates or record eyetracking data, don’t use the think-aloud method. Instead, let participants complete tasks on their own, without interruption. You can ask follow up questions afterward or use a retrospective think-aloud approach by having participants talk about what they were doing after completing a task.

Stop Bothering Participants When They Can’t Think Aloud

One of the most important things you’ll learn from being a participant yourself is how difficult it can be to perform a task, think aloud, and answer questions all at once. When a task is cognitively difficult, bothering a participant with questions or requests to think aloud becomes extremely distracting and annoying. Allow a participant to concentrate on the task, then follow up with questions once things get easier.

Stop Bothering Those Who Can’t Easily Think Aloud

It’s also important to recognize when you have a participant who is not very good at thinking aloud or who is naturally quiet. When participants think aloud, the quantity and quality of their remarks don’t have to be the same. Recognize the silent types and let them be silent if that’s their preference. With these people, focus instead on their behavior and use follow-up questions to gain insights.

Use Realistic Tasks

Don’t ask participants to do things they wouldn’t normally do. Ensure that tasks are realistic for the representative users you’ve recruited.

Personalize Tasks

In addition to predefined tasks all participants perform during usability testing, it’s often valuable to let participants choose their own tasks or add meaningful context to tasks. One option is to use basically the same predefined tasks for all participants, but allow them to personalize those tasks. For example, if a task is to make a purchase on an ecommerce site, let participants choose what they want to purchase. Another possibility is to ask participants to describe what they would normally do on a site, then build a task around that. Both of these methods enable you to gather additional insights by having participants perform meaningful tasks.

Don’t Embarrass Participants

Don’t require your participants to do anything embarrassing or uncomfortable. Most people don’t know much about what you’ll ask them to do when they volunteer for a study. Even though you explain that they can end the session at any time, few people have the guts to do that. Most just silently suffer through an embarrassing study, like I did when I sat in Heathrow Airport for several hours with a big number 4 hanging around my neck. If you find that you’ve recruited people who aren’t embarrassed by things that would embarrass most people, you should wonder how representative those participants are.

Properly Explain Prototypes

Lower-fidelity prototypes require a lot of facilitator explanation. Although we understand the nature of a prototype, most participants do not know what to expect or how to interact with our prototypes. For lower-fidelity prototypes, clearly explain to participants that they do not represent the final design, show them examples of placeholder text and images, and set the expectation that not all links and functionality are working. When appropriate, step in and explain what would happen in response to a participant’s actions.

Provide the Right Level of Detail in Prototypes

In low-fidelity prototypes, it’s important to provide enough detail to enable participants to understand what they are looking at. Do not include lorem ipsum text, humorous or familiar names, or unrealistic data. Instead, use realistic sample text. If an image is necessary for participants to understand what they are looking at, provide a realistic sample image instead of a placeholder image box.

Minimize the Effect of Observers

The ideal place for observers is out of sight in another room where they cannot communicate with or distract participants. Make sure the observation room is well soundproofed and, if it’s not, that observers keep quiet. If you have people observing through a Web conference or over a speakerphone, make sure they keep their phones muted.

Set Strict Rules for Observers

If you do have observers in the same room with participants, introduce them to participants and explain their role in the research, then make sure they remain quiet and neutral. Ensure that they don’t make sounds or use body language that gives away their feelings—for example, sighs, head-shaking, or defensive body postures such as arm crossing. The primary researcher should take the lead and ask most of the questions. Observers will likely want to ask questions, which is often useful, but establish a time at end of the session for their questions.

Limit the Number of Researchers and Observers

The more people who are directly observing a session, the more awkward and uncomfortable it can be for a participant. One facilitator is ideal for usability testing and other lab-based research. For field studies, two is an ideal number—in terms of safety and comfort for both the participant and the researchers. Having more than two people becomes awkward—particularly in the limited spaces available when observing people in their work or home environments.

Do Not Include Observers Who Might Make Participants Uncomfortable

Observers who might make a participant hesitant about speak openly should observe from another room. For example, participants may not feel free to express negative opinions when the designer, developer, or owner of an application is in the same room; and employees may be hesitant to speak freely when people higher up in their organization are observing.

Minimize the Effect of Technology

Unobtrusive recording and observing technology can do a lot to make participants feel more comfortable.

Avoid Rooms with Two-Way Mirrors

Participants are far more comfortable in usability labs and focus-group facilities in which discreet cameras that feed to a large video screen in an observation room have replaced the traditional, two-way mirrors. If you must use a room that has a two-way mirror, acknowledge it immediately and admit that you understand it may make participants feel awkward. To minimize its distraction, place participants so they do not face the mirror. If there are no observers, tell the participant there are none and close any blinds or curtains that cover the mirror.

Use Hidden or Unobtrusive Cameras

Of course, you should always get a participant’s permission before recording a session, but participants do not need a constant visible reminder that you’re recording their session. The more unobtrusive your recording equipment, the more participants will be able to relax.

Minimize the Distractions of Eyetracking

If you don’t really need to do eyetracking, don’t run it in the background just for the heck of it. If you do need to do eyetracking, describe how it works briefly at the beginning of a session, when you calibrate the eyetracker to the participant’s eyes, but then try not to draw any attention to it afterward. For example, don’t stare at the facilitator’s monitor showing the eyetracking data instead of paying attention to the participant, and don’t make comments that refer to eyetracking data you’ve seen. Conduct the session as you would a regular usability test.

Keep Notetaking to a Minimum

Ideally, have someone take notes for you, so you can focus on participants. If you do take notes yourself, record the session—at least recording audio—and take minimal notes. The more you become immersed in your notetaking, the less you’ll observe and the weaker your personal connection with a participant will be. If you do take notes when you’re in the same room with a participant, take them on paper, not on a laptop, which can be very distracting.

Conduct More Field Studies

Obviously, field studies let you observe behavior in a much more natural context than when conducting user research in a usability lab or focus-group facility. Participants usually feel more at ease and are better able to demonstrate their natural behavior in their own environment.

Spend More Time Observing

Observing people in their own natural environment with no interruptions is about as unobtrusive as research gets. However, despite the benefits, most researchers spend very little time doing pure observation and tend to rely more on contextual inquiries, which involve much more interaction between participant and researcher.

While contextual inquiries are very valuable, it’s best to spend some time before or afterward just observing a participant’s natural behavior, without interrupting. Even if the tasks you want to observe are only a small part of a typical day, observation over a long period provides greater insights into a participant’s overall situation and where the tasks fit into their day. This gives you the best of both worlds: observation as well as the additional insights a contextual inquiry can provide.

Respect Participants’ Privacy

When doing research in people’s homes, behave as a guest would. Respect where they want you to go and where they may not want you to go. If there’s no real benefit to your seeing more private rooms such as a bedroom or bathroom, don’t ask to see them. Let the participants know in advance who will be coming along with you, and don’t bring too many people.

Do Usability Testing in the Field

Researchers often go to participants’ workplaces and homes to conduct contextual inquiries and ethnographic observations, but why not do the same with usability testing, interviews, and other traditionally lab-based studies? Conducting usability testing with people at their own desks at work or at home is far more natural than doing testing in a lab. Participants can use their own computer and materials, in their own environment; it’s usually not possible to bring along observers; and there is no intimidating recording technology.

Use Remote Techniques

In remote research, people participate from their own location with few of the distractions of being in a lab or having researchers come to their location.

Conduct Remote, Moderated Research

Remote, moderated research—such as usability testing, user interviews, or contextual inquiries—allows people to participate in studies at work or at home, in their own natural environment, with their own computer, tools, and supplemental materials, and with typical distractions. Although participants realize you are recording what they do and other people may be listening in, they are usually much more comfortable without observers in the same room with them. Thinking aloud tends to come more naturally to participants over the phone. Since they realize you can’t see them directly, explaining what they’re doing and thinking makes sense and is less awkward.

Conduct Remote, Unmoderated Research

A variety of activities such as card sorting, unmoderated usability testing, and diary studies, in which participants write about their experiences, let people participate without any direct interaction with a researcher or observers.

Conduct Natural Observation

There’s no law—at least in the United States—or ethical rule against simply observing people in public. What could be more natural than simply observing what people do in real situations, without any interference from a researcher? In the poorly planned study I suffered through at Heathrow Airport 20 years ago, researchers’ just observing what people did in the terminal would have been easier and far more realistic than asking certain people to walk around with large numbers hung around their necks. As a participant, I would have preferred that, and the researchers would certainly have seen more realistic behavior from me.

Conclusion

Like any group of professionals, as user researchers, we can get so immersed in our research techniques and our usual ways of doing things that we take them for granted and overlook possible alternatives. We may fail to consider how participants might experience our studies from their own perspective. Therefore, the most important recommendation is the first one I gave: be a participant yourself. There is no better way to spot the strengths and flaws of a research technique than to experience it from the other side of the research table.

Note

[1] Nielsen, Jakob. “User Testing Is Not Entertainment.” Alertbox, September 11, 2006. Retrieved March 2, 2011.

In Remote UX Research | Usability Testing | User Research