The Biggest Mistakes in User Research, Part 2

Failing to Prepare Participants in Advance

For those of who work in User Experience, it’s all too easy to forget that most people have no idea what to expect when they agree to participate in user research. The recruiting company, client, or whoever else may have contacted participants might not have painted a clear picture of what to expect. Most people aren’t familiar with the idea of demonstrating their tasks while talking about what they’re doing.

Compounding this problem is the fact that it’s very difficult to clearly and concisely describe the field-study process in a way that participants can understand or would take the time to read about. When you’re trying to recruit people, it’s easier to be brief and speak in generalities. Wait to tell them the specifics until you’re there in person and can introduce and explain your method.

Most participants expect a traditional interview or a focus group, in which they’ll respond to your questions, instead of being prepared to take a more active role in the session by actually performing their tasks and explaining what they’re doing. If they’re not prepared to show you their tasks, they may not have the actual materials they need to demonstrate those tasks. For example, if you need to observe a manager processing employee expense reports, the manager might already have approved all open expense reports earlier in the day, before you arrived, and have nothing to show you.

Thinking you’re there to interview them, participants might insist on meeting with you in a conference room, where there’s more room and they can avoid disturbing their coworkers. Or they might have invited some coworkers to the conference room to share their opinions.

Solutions

To ensure that participants are prepared and know what to expect, follow these tips:

• When you first contact participants to recruit them, briefly describe your research more generally. Once you’ve recruited participants, send them an email message to introduce yourself and explain in more detail what their session will involve.
• In introducing yourself, explain that your role is to represent users, understand their needs, and ensure that the product design takes their needs into account. This helps to dispel participants’ misconceptions and worries about your motives—especially when you’re observing employees and how they do their work. 
• Explain that, to enable you to understand their tasks, the session will be more than a typical interview. You’ll need to observe participants as they actually perform their tasks in full, as they would normally perform them. Verify that the participants do typically perform the tasks that you want to observe.
• Explain that, because you want to see participants performing their usual tasks, the session should take place at the location in which they typically perform those tasks, and it should be a one-on-one session.
• If necessary, ask participants to save work to show you during the session. For example, you could ask that manager to save some employee expense reports to approve during your session.
• This is a lot to introduce in an email message so try to be as brief as possible. Although explaining all of this in your message means you’ll have less to explain at the beginning of the session, you’ll still need to provide a recap at the beginning of the session, describing who you are and what you’ll be doing during the session.

Getting Distracted by Notetaking

It’s extremely difficult to simultaneously observe, listen, assess what you’re seeing and hearing, ask appropriate questions in a neutral manner, maintain eye contact and rapport with the participant, keep track of the direction a session is taking, keep the participant on track in performing the tasks, remember which tasks you’ve already observed and which you still need to see, remember what questions you still need to ask, and monitor the time remaining in the session. If you try to take detailed notes in addition to all these other activities, one or more of the activities would likely suffer. Usually, you won’t be able to devote as much close attention to observing or listening, or your rapport with the participant suffers.

Solutions

Consider these other approaches to avoid getting distracted by trying to take detailed notes:

• Take only high-level notes, capturing your insights and observations and noting what questions you’ll need to ask later.
• Delegate detailed notetaking to someone who is observing the session. Since that person doesn’t have to focus on facilitating the session, he or she can fully focus on observing, listening, and taking detailed notes.
• Make video or audio recordings of the sessions. Later, you can either review the recordings in their entirety or review only the more complicated or especially interesting segments to capture more detail.
• If you don’t have time to review the recordings, have someone transcribe them.
• Take time after each session to type up your overall impressions and high-level themes that you remember.

Failing to Record the Research Sessions

It’s a rare luxury to have sufficient time during analysis to go back and review user-research recordings. If you already know you’re not going to have enough time to review them, why record the sessions at all? Because not recording the sessions is usually a mistake. If you don’t record them, you’ll have to rely on your memory and your notes, which can capture a lot, but not everything. Even with the best notetaking, you’ll sometimes find that you need to rely on faulty memory in interpreting your fragmentary notes.

Of course, there are times when you shouldn’t record sessions. If recording participants would influence how they behave or make them feel uncomfortable, you might want to avoid recording. Plus, some companies forbid recording on their premises. However, whenever you can, record sessions at least as a backup to your notes and memories.

Solutions

To ensure you capture the best information for your team, do the following:

• If participants and their organization grant permission, and you think recording won’t negatively affect participants’ behavior or present privacy or security concerns, always record user-research sessions—creating either video or audio recordings.
• If you think recording video might make participants feel uncomfortable or change their behavior or video recording is not allowed, record only audio.
• If you don’t have time to review the full recordings, review particular tasks that need clarification.
• If you have time, review the recordings to capture participant quotations and video clips to share with others on the project. This can be a great way to convey your research findings to those who couldn’t observe the sessions. Video and audio clips can make your presentation of your research findings more interesting and impactful for your audience.

Interrupting Participants Too Often

When you observe participants during user research, there are two approaches you might take:

1. You might want participants to act as naturally as possible and perform their tasks as they normally would, without interruption.
2. You might want them to demonstrate their tasks and tell you what they’re doing as they perform each step.

Getting participants to do exactly what you want—either demonstrating their tasks in full or speaking aloud in detail—takes a lot of effort and skill. Once they’re doing what you want, the last thing you should do is interrupt them unnecessarily and throw them off track. For example, a common problem in contextual inquiries occurs when participants are walking you through their tasks, step by step, and you ask a question that causes them to stop what they’re doing and turn toward you to answer. Then, once they answer your question, they sit there waiting for your next question instead of going back to performing their tasks. They’ve slipped back into interview mode, and you’ll have to get them back into task-demonstration mode.

User researchers might fail to consider how difficult it is for participants to simultaneously perform their tasks and talk about what they’re doing. Each time you interrupt them with a question, you risk breaking their concentration and interrupting their flow.

Solutions

To avoid interrupting participants unnecessarily, follow these tips:

• If your research technique involves observing participants’ natural behavior, without interruption, try to remain unobtrusive. Write down your questions as they arise. Then, either ask your questions at the end of the session or schedule time to interview the participant later.
• During contextual inquiries, carefully consider whether and when to jump in with a question. Do you need to ask the question immediately, or can it wait until later? Sometimes, if you delay asking a question, you’ll get the answer anyway by observing the participant’s actions or through their subsequent explanations.
• If a participant is doing a good job of demonstrating and explaining a task, err on the side of not interrupting. Instead, write down your questions and ask them later, when there is a natural pause in the action.

Failing to Manage Observers Effectively

You might forget that not all observers know how to behave during a research session, assuming that they’ll just know to sit back and let you take the lead. But amateur, first-time observers can ruin a research session by jumping in with questions at the wrong time, asking inappropriate or leading questions, changing the direction of a session, breaking a participant’s concentration, or making the participant feel uncomfortable.

Solutions

To avoid problems with your observers, be sure to do the following:

• Give observers clear directions about what they should and should not do during research sessions. Make it clear that you’ll be leading the sessions, and observers should remain quiet and mostly observe.
• Specify a time when observers can ask questions—perhaps at the end of a session or when you give them a signal that it’s their turn. For example, you might ask, “Bob, do you have any questions?”
• Coach observers in how to ask neutral, non-leading questions.
• Give observers a task to perform—such as managing video or audio recordings, taking photos, or taking notes. You might ask them to focus on noting specific things such as environmental factors.

Rushing Through the Analysis and Deliverable Phase

User research can seem to take a long time—especially for those who aren’t aware of all the activities taking place throughout the process. Recruitment of participants can take longer than expected. Plus, having to schedule around participant availability can result in further delays. Therefore, when it’s finally time for analysis, the client or project team might often be antsy—wanting to get the results of your research quickly and move on to design. So there is often pressure to rush through analysis and quickly create a high-level deliverable. While the desire to speed things up might be understandable, it’s a very bad idea to spend so much time and effort gathering research data, then not take the time to analyze it properly.

Solutions

To avoid rushing through your analysis and deliverable phase, do the following:

• At the beginning of the project, be sure to allocate enough time in the project plan for both analysis and creating your deliverable.
• If anything delays the schedule, adjust the dates in the project plan accordingly. Don’t allow anyone to take time away from the analysis or deliverable-creation tasks.
• Involve your clients and project team in the research, and provide frequent updates about what’s going on. The more they know about the research, the less they’ll wonder: What’s going on? Why is it taking so long?
• At the beginning of the analysis phase, tell the team you’re starting your analysis and give them a sense of how much data you’ll be analyzing. Showing them the number of participants in the study and photos, artifacts, and hours of recordings you need to review in a huge affinity diagram can seem impressive and helps them to understand the magnitude of your task, justifying why it’s going to take some time.
• If you can, tide the team over by providing some preliminary findings—such as some interesting insights you’ve already discovered. That can pique their interest and make them anticipate all the other interesting information that comes from careful analysis of the data.
• Involve observers in analyzing the data. Ask them to help you out by creating and organizing an affinity diagram.
• For the deliverable, you don’t have to create a long, detailed report, but you should at least create a presentation in which you discuss the findings and document the research for future readers.

Doing Nothing About the Findings

Crazy as this might seem, companies often conduct user research, seem to appreciate the results, then do nothing about the findings. We’ve all seen this happen. You conducted the research and presented your findings. Everyone seemed to agree with your recommendations, then they never got implemented. Change is difficult. There are a number of reasons why recommendations don’t get implemented. For more details, see my column, Why Don’t Usability Problems Get Fixed?

Solutions

Do the following to make it more likely that the product team will implement your findings and recommendations:

• Involve powerful people in the research, who have the power to demand that your recommendations get implemented. Involve them in planning the research by asking them what they want to learn from it, inviting them to observe sessions, and getting their opinions about what they saw.
• Run your recommendations by your own technical resource to determine whether they’re feasible before including them in your report. If a recommendation might be difficult to implement, provide an alternative solution that would be more feasible in the short term.
• Prioritize your recommendations and assist the team in planning how to implement them.
• Remain involved throughout the design and development process to collaborate, monitor progress, and answer the team’s questions.

It’s Okay to Make Mistakes

While user research can be very difficult, and it’s easy to make mistakes, that’s okay. User researchers are only human. We all make mistakes. Even veteran user researchers have made their share of these mistakes, as well as others. The key is to learn from your mistakes and improve your skills as a user researcher so you’ll have more successful studies in the future. Remember, this series doesn’t provide a comprehensive list of all possible user-research mistakes. If you have others you’d like to share, please feel free to leave a comment.