Recruiting the Right Participants for User Research

Screening Participants

“As researchers, it’s our obligation to tap the right participants to enable us to get insights that are meaningful and actionable,” responds Gavin. “This makes our participants—our data—the Achilles heel of research. The first step is to get recruiters you can trust. At GfK, we have a lot of them because we use recruiters with particular specialties.

“The second step is to prepare a recruiting screener that asks the right questions in a way that would prevent potential participants from being able to discern what answers would get them into a study. Let’s face it, there are professional participants out there, and we want to avoid them. As part of our training for researchers, a whole section covers writing screeners. One example of how to do this is to limit the number of yes/no questions. For example, if you are targeting a group of participants who haven’t done X in the last six months, instead of asking, ‘Have you done X in the last six months?’ you should ask them the date on which they last did X. They won’t know the answer that would get them through.

“Third, when asking warm-up questions at the beginning of a session or as a question in the pre-session questionnaire, ask your core questions again to verify that you have the right participants. We have found that people sometimes change their mind. Participants who lie often don’t remember their lies. Finally, if we ask participants about their experience with something, we ask them to bring that thing into the session and check their product IDs. Despite all of this, you’ll still sometimes get a bad participant or two. Tag them in your database to filter them out of future studies, and be sure to provide feedback about them to your recruiter.”

Relying on User Profiles and Prior User Research

“Recruiting the right participants is the most important part of conducting effective user research—especially when the number of participants is small,” says Carol, “If you don’t recruit actual target users, you won’t be able to confirm that the research findings are relevant. The way to determine whether you have recruited the right participants is to create a user profile that you base as much as possible on information about real users that you’ve gathered from prior research and other data sources. Sources of real information about your users can come from any available market research, customer service phone logs, sales information, and other internal resources.

“If no data is available, your best option, if budget permits, is to conduct your own research and create formal or informal personas of your users. If no data is available and there is no budget for conducting your own research about users, you’ll have to do the best you can on the basis of anecdotal information. Gather what information you can by talking informally to the sales and marketing team, the customer service team, and any others who have access to target users. Then, schedule a planning meeting with the core team to determine who they think the users are and what they think users want to do with the product or service you are testing.

“Don’t let the answer that ‘everyone is a potential user’ stand. You have to break down ‘everyone’ into subgroups of potential users, and each subgroup needs its own list of characteristics that make up that subgroup’s user profile. Depending on the size of your research budget, you should recruit participants from one or more of these subgroups of users. For each subgroup, write a detailed screener based on what you have determined to be necessary characteristics, as well as other characteristics that are nice to know about, but not essential to determining the eligibility of a participant for your study.

“Be prepared to screen many more people than you will need for your research. Working with a detailed screener, you’ll find that you have to eliminate many prospects who do not qualify for your study. Even prospects who meet many of your criteria may not make good participants if they do not have a real and current need for your product or service. Above all else, you must identify motivated participants who share the goals of real users or customers, so their actions and feelings reflect your target audience in a realistic way.

“The challenge of understanding who your users actually are is huge. The challenge of successfully recruiting such users for your studies is another big one. So think about this activity as a two-stage process:

1. Understanding your users and creating a screener to match them
2. Recruiting and scheduling participants that match them for your research studies

“If you complete both of these activities, you’ll be able to manage and control the whole recruiting process to ensure that you get the participants you want and need. However, be prepared to invest a lot time in recruiting and scheduling participants. If you will not be doing the recruiting yourself, see whether you can build some sort of applicant-review process into the screening process to provide quality control. If such a review is not possible, make sure that your screener is as detailed as you can make it, so your recruiting agency understands how to make a good match in selecting study participants.”

Understanding Users and Their Motivations

“This question really calls for a two-part answer,” replies Dana. “The first part is about making sure that participants are the kind of people who would normally use the product or service that you’re designing. This means they would practice the behavior you’re trying to understand and support. The second part is to ensure that they’re truly motivated to participate before you get them in a room with you.

“One way to know whether you’ve recruited the right people for your study is to focus on behavior. What behavior do you want participants to take in your study? Have they done this in real life—or are they about to do it in real life? For example, one terrific method of research for ecommerce Web sites is to conduct compelled shopping studies. User Interface Engineering (UIE) pioneered this approach in the 1990s. The idea is that you find people who are in the market for what you’re selling—cars, digital cameras, business cards, or whatever—and give them the money to buy it, then watch them use the Web site to make the purchase. They can either use the money to pay for the thing that they came to the site to buy, or they can keep the money if the design of the site fails to help them find what they want, then purchase it. There’s no pretending. This is not a hypothetical situation.

“The important thing is that participants have to be people who are in the market for the thing you’re selling. You can learn about this through careful sourcing and interviewing of candidates. At UsabilityWorks, we don’t use screening questionnaires. We do voice-to-voice interviews on the phone and start with open questions. For example, for an example ecommerce study, we might begin by asking about the last time the person bought a digital camera—if that’s what the study is about—and why they’re looking for a new digital camera now. This tells us a lot about what their experience was like, whether they’re truly in the market now, and whether they’ve done any research about the product yet. It’s harder for people to game their way into a study when they can’t try to pick the right answer from a list of multiple-choice options.”

Understanding Participants’ Feelings

“When recruiting participants, you must first and foremost understand that they are people with their own lives, emotions, frustrations, and needs,” answer Dan and Jo. “Often, clients give us a set of criteria for participants that describes people as belonging to particular market segments. But this has little to do with people’s willingness to participate or their interest in the domain you are studying. Participants need to feel that they can make an important and active contribution toward improving a product’s design. So our first step in determining whether we have recruited the right people for our user research is to stop describing them or treating them as lab rats in experiments.”