Avoiding Hard-to-Answer Questions in User Interviews

Practical Usability

Moving toward a more usable world

A column by Jim Ross

January 9, 2017

A funny thing happens when you interview people—they answer your questions even if they don’t really know the answer. That’s why it’s so important to know what types of questions people can and cannot answer correctly.

There’s a good reason why UX research focuses more on observing people’s behavior in their natural context than on interviews, surveys, and focus groups. Although all of these techniques can be useful, what people say doesn’t always match what they actually do. Observing and interviewing people in the context of their tasks gives you a much more accurate understanding of their characteristics, their tasks, the tools they use, and their environment.

Of course, talking with people is helpful because observation alone often isn’t enough. So almost every user-research method includes some kind of interview or discussion. While observing user-research participants shows you what they do, it also raises questions. Unless you interview participants, too, you’ll have to make assumptions to understand the motivations behind their actions. Talking with people is essential for you to understand their behavior.

Champion Advertisement

But some questions are easier for people to answer than others. In this column, I’ll discuss some types of questions people have a hard time answering accurately.

Asking People to Remember the Past in Detail

When you ask someone about their experiences with a product or their opinions about it, they can provide their overall impression and tell you about the types of activities they performed using it, but it’s usually difficult for them to remember specific details. For example, when conducting a usability test for an auto-manufacturer’s Web site, one might ask participants about online research they’ve done recently, as part of their car-buying process. They would probably remember some of the Web sites they visited, the general tasks they performed, and their overall opinions of those sites, but it’s usually much more difficult for them to remember details about the sites. Human memory is fallible and inaccurate. If some participants can remember a few details about a particularly good or bad experience, that’s great. But you shouldn’t push them to try to remember more or you might actually cause them to make up details they can’t recall.

How to Ask Questions About the Past

It’s okay to ask questions about past experiences, as long as you recognize the limitations of people’s memory. Here are some tips for asking questions about the past:

Ask questions to understand participants’ general experiences and opinions. Don’t expect or press participants for details they can’t provide.
Ask participants to talk about memorable incidents, but expect that they may not remember much detail.
Display the Web site, application, or product a participant is talking about. Letting participants walk through a user interface makes it much easier for them to describe what they’re experiencing. They can demonstrate a task or point to something concrete rather than relying completely on memory.

Asking People to Predict the Future

People aren’t very good at predicting what they would or wouldn’t do in the future—or in a hypothetical situation. If you ask them to do this, they may answer your question, but their comments might not accurately reflect what they would actually do. For example, if you ask people whether they would use a new feature, some might say Yes because they wouldn’t want to rule out the possibility that they might actually use it. Others might say No because they can’t yet imagine why they might use it. It’s especially difficult for people to provide an opinion on something that doesn’t yet exist.

How to Learn About the Future

Wait to ask participants for their opinions about specific features or design elements until they’ve tried using them in a prototype. It’s much easier for people to provide opinions about something tangible rather than to comment on an abstract concept. This still won’t predict whether people will use a particular feature in the future, but it can give you a good sense of whether a design is successful.

Discussing Details Out of Context

In traditional user interviews, people can tell you about their tasks in general, but it’s difficult for them to accurately remember each step. People perform many tasks automatically, without thinking about each individual step, which makes it especially difficult for them to talk about their tasks out of context. For example, think about teaching a child how to tie his shoelaces. Could you tell him what to do without having a shoe with which you could demonstrate how to do it?

How to Learn About Users’ Tasks

It’s much easier for people to demonstrate their tasks than to talk about them in the abstract. So visit participants in their natural environment and observe them performing their actual tasks. You can either observe them without interrupting their work, then ask them questions afterward, or you can conduct a contextual inquiry, during which you ask participants to talk about each step as they perform their tasks. The key is to observe people’s natural behavior and get them to talk about their tasks in the context of performing them.

Interviewing People to Identify Their Needs

It’s a mistake simply to ask people what they want or accept what they tell you without questioning it. People can tell you about the problems they face and the things that annoy them, but they have a difficult time articulating what they want or need. While research participants sometimes offer suggestions for what they think would solve their problems, their ideas don’t always lead to the best solutions.

How to Understand Users’ Needs

Observing and talking with people while they perform tasks in their natural context is the best way to discover their true needs. When people suggest solutions, don’t simply accept them at their face value, but don’t dismiss them without considering them either. Find out more about people’s problems and the reasons behind their requests. Ask them about how specific problems affect them, what they’re trying to achieve, and how they think their solution would solve their problem. Having that information will help you to develop a design that solves their problems and meets their needs.

Asking People to Imagine How Something Would Work

Don’t ask people to envision or comment on design concepts they cannot see or use. People usually have difficulty imagining a design solution, feature, or interaction simply from a verbal description. Even if you’re able to provide a good description, how can you know that participants have the same understanding as you do? Showing participants something concrete is always better than a description, but their simply seeing a design is not enough. To give you the best feedback, participants need to be able to use a prototype.

How to Gather Feedback on a Design

Give people a prototype they can see, use, and react to. It’s much easier for people to provide feedback on something concrete with which they can interact. But consider the limited fidelity of prototypes when assessing participants’ feedback. Some complex or novel interactions are difficult to simulate with lower-fidelity prototypes, making it harder for participants to get a realistic sense of how certain elements work.

Asking People to Envision an Improved Design

Very few research participants are designers. So they can tell you what seems to make sense, what confuses them, and why a particular design is frustrating. They can tell you what works and what doesn’t work, but they’re usually unable to provide specific design solutions. While they may be able to suggest some simple improvements, asking them to suggest big improvements usually leads to silence.

How to Discover Design Improvements

Since users aren’t designers, you shouldn’t expect them to make design decisions for you. The purpose of usability testing is to gather information about what works and doesn’t work, then use that information to improve the design. Don’t ask participants for solutions. Focus on finding out about the problems they encounter, then use your own judgment about what design changes to make. Finally, test your design changes with participants to see how well they work. Validate that your changes were actually improvements.

Asking People to Distinguish Unimportant Design Differences

Some project teams try to solve every design dispute by saying, “Let’s test that.” Of course, it’s great when your teammates believe in the value of usability testing, but some teams take this too far and try to place the burden of every design decision on test participants. The purpose of usability testing is not for participants to tell you what to design or to resolve every design dispute.

In fact, such disputes are sometimes about differences that are so minuscule that test participants can’t tell the difference between them and don’t really have a preference. Sometimes you may even sense that participants are thinking, I don’t really see much difference between these designs. Why are you even asking me this?

How to Test Multiple Designs

You can conduct comparative usability testing to get feedback on alternative designs. When participants see only one design, they may have a hard time envisioning how it could be different. But when they can compare two or more designs, they’re usually able to provide much more useful feedback.

However, don’t test more than two or three versions of a design during the same usability study. Participants would likely find trying to compare too many different versions difficult and confusing. Ensure that design differences are meaningful. Again, don’t use comparative testing to solve every design dispute. Make design decisions first, then test your designs with users.

Asking People to Explain Their Behavior

Observation lets you see what people do, but it often doesn’t answer the more intriguing question: why? While people don’t typically know the reasons for their actions, if you ask about them, they’ll rationalize to come up with a plausible answer. They aren’t consciously trying to deceive you; they just aren’t aware of all the subconscious reasons for their actions.

How to Understand Participant Behavior

Because participants can’t accurately explain their actions, some UX professionals have advised paying attention only to what research participants actually do and not what they say. I think this is a little extreme. Observing behavior gets you only so far. You must try to discover the reasons for people’s actions.

Careful questioning and having participants think aloud as they perform tasks during a usability-test session can help you understand their actions. When participants think aloud, this gives you immediate insights into what they’re thinking as they perform the tasks. Later, you can ask follow-up questions to get additional insights. I don’t think it hurts to ask people why. Even if participants don’t have any insights into why they’ve done something, their answers can still help reveal their mental models. Just remember, don’t blindly accept their explanations as the truth. Instead, consider their answers as a means of getting insights into their thinking.

Things People Can Easily Discuss

In user research, participants provide the most accurate and useful information while they’re performing their typical tasks in their usual context. Observing participants in their own context lets them show you exactly what they normally do. In usability testing, participants provide the best feedback when trying out and comparing design solutions rather than simply looking at designs and providing their general opinions. By asking specific questions immediately after participants perform tasks, you’ll get the best responses.

When doing user research, put more emphasis on observing what people actually do rather than on what they say. Of course, you should listen to what they say, too, but take what they say with a grain of salt. By understanding what types of questions people have trouble answering, as well as those that elicit the best information, you can avoid wasting time asking unproductive or misleading questions and focus instead on asking the right questions.

In Interviewing Users | User Research