Although we can’t always spend as much time and money as we’d like to conduct user research and there are times when we need to take shortcuts, there’s a fine line between discount user research and half-assed user research. UX professionals have always had to fight to get user research included on projects. Because of time and money pressures, we may have felt justified in cutting corners to fit in whatever user research we could. After all, even a little user research is better than none at all. Isn’t it?
Yes, taking clever shortcuts can reduce the time and cost of doing user research—and, sometimes, conducting at least some user research is better than doing none at all. However, if you sacrifice in the wrong areas, you can end up gathering incorrect or incomplete information that can lead to poor design decisions and, ultimately, waste far more time and money than the time and money you originally saved by conducting discount user research.
Champion Advertisement
Continue Reading…
Looking beyond the negative impacts that doing half-assed user research can have on an individual project, it’s important to be aware that long-term damage can occur if an organization accepts half-assed user research as the norm. If cutting corners appears to work once, you may have normalized the cutting of corners on all projects—rather than devoting enough time and money to a research project to conduct it properly. By allowing an organization to avoid facing the consequences of not conducting user research properly, you become an enabler, allowing your company to feel good about checking off the user-research checkbox: Yes, we conduct user research. Unfortunately, when half-assed user research provides poor results, it can give all user research a bad reputation.
In this column, I’ll discuss the various shortcuts that teams try to take when walking the fine line between discount user research and half-assed user research. I’ll also explain what you should do instead of taking half-assed shortcuts.
Conducting User Research Without Users
When your research doesn’t involve any actual users, that’s a good sign you’re conducting half-assed research. Some companies rely on other people to represent the users—such as stakeholders or subject-matter experts—instead of doing research with actual users. For example, companies that create medical software often use formerly practicing doctors as subject-matter experts to represent the physician’s point of view.
Why This Is Half-Assed
Stakeholders and subject-matter experts can provide useful information about business requirements, the subject matter, and general information about users and their tasks, but they can’t substitute for actual users. Once they join the project team, they become insiders who are too close to the company and its products, and they are too far removed from the context the actual users’ experience.
What You Should Do Instead
Interview stakeholders and subject-matter experts first to learn about business goals and the subject matter and to gather initial information about the users and their tasks. Use this information to learn enough about the subject matter to plan the user research. Then conduct research in the field, with representative users. To avoid people misunderstanding the differences between these two types of research, call your interviews with stakeholders and subject-matter experts stakeholder research, distinguishing it from user research.
Using the Wrong Participants
User research is useless if you conduct it with the wrong participants. Nevertheless, when it’s difficult to find or recruit just the right participants, some companies include people they think can substitute for representative users. Examples of using the wrong participants include using employees in your company, doing research with family and friends, or simply grabbing people off the street.
Why This Is Half-Assed
When you do research with the wrong participants, you end up gathering information about the wrong people. Family, friends, and coworkers might not have the right subject-matter knowledge or perform the same tasks. They might have too much inside knowledge about the product or company to be objective. Their relationship with you or your company may bias their actions and responses.
What You Should Do Instead
Recruit participants who represent actual user groups—either by using lists of existing users or recruiting people who match important user characteristics and behaviors.
Using friends, family, coworkers, or strangers for user research can be okay if they meet the following criteria:
They do fit the user profile.
The target audience is a general, consumer audience.
The participants don’t need to have any specific subject-matter knowledge.
They aren’t already biased toward or against your organization or product.
Excluding Important User Groups
For some products, there are multiple user groups, each with different needs, tasks, and characteristics. Ideally, you should include multiple participants from each user group in your research. Because this increases the total number of participants you’ll need to include, this also increases both the time and cost of your research. If you have limited time and budget, you might either include fewer participants per user group or disregard certain, less important user groups altogether.
Why This Is Half-Assed
To save time and money, you could try focusing your research on one or two key user groups. However, at the beginning of the research process, you might not know which groups it’s most important to focus on. You might not discover that there are differences between certain user groups until you’re in the middle of the research. When you exclude important user groups, who have different needs or may use the product in different ways, you may end up basing your design decisions toward the needs of only the people you did include. But those decisions might not work well for the user groups you didn’t include in your research.
What You Should Do Instead
Work with your clients and stakeholders to define the main user groups, and try to include all of the most important user groups in your research. Even if you can include only two or three participants from certain user groups, that should at least provide enough information for you to determine whether you need to add more participants from those groups. If you determine that one or two user groups are far more important than the others, you may be able to focus your research on them. This can be similar to setting the goal of meeting the needs of a primary persona because, in doing so, you’ll also meet most of the needs of the secondary personas.
Not Having Enough Participants
When your time is limited, you might not be able to include enough participants. There’s no magic formula for the right number of participants to include in user research. It depends on the number of user groups, the number of tasks you need to observe, the length and complexity of the tasks, the length of the research sessions, and the complexity of the subject matter.
Why This Is Half-Assed
When you don’t have enough participants, they may not represent every user group, you probably won’t observe as many different tasks, and you might not see enough variation or repetition between people to conclude that you’ve witnessed common patterns. You’ll end up with incomplete information, which requires you to make assumptions, often leading to poor design decisions.
What You Should Do Instead
When your time is limited and you know you won’t have an ideal number of participants, focus on research goals that are narrower and more specific. Define the questions that it is most important for the research to answer. With a smaller, better targeted scope, you’ll get more detailed answers rather than ending up with a vague and incomplete understanding of an overly broad range of users.
Using the Wrong Research Methods
Ideally, user-research methods involve visiting people to observe them performing their tasks in their usual environment. When it would be too difficult, expensive, or time consuming to travel to individual participants’ locations, you might rely on methods that are easier to perform—such as interviews, focus groups, and surveys.
Why This Is Half-Assed
Interviews, focus groups, and surveys focus on what people say and don’t show you what people actually do. People aren’t able to accurately describe their tasks out of the context in which they normally perform them. For UX design, preferences and opinions aren’t as useful as understanding users’ tasks and behaviors. Of course, such methods are fine when you use them inaddition to observational user research, but they cannot substitute for direct observation.
What You Should Do Instead
Whenever possible, conduct field studies, visiting participants in their normal locations and interviewing and observing them as they perform their typical tasks. Interviews, focus groups, and surveys can provide supplementary information, but don’t allow them to become a substitute for field studies.
Conducting User Research Remotely
Unfortunately, it’s not always possible to travel to participants’ locations. Travel increases the time and cost of research, especially when you need to include participants in many different locations. A common cost-saving alternative is to conduct research remotely—by doing either phone interviews or remote meetings in which participants can share their computer screens.
Why This Is Half-Assed
When you conduct research remotely, you can talk with participants and see their on-screen actions, but you’ll miss much of their context. You can’t directly observe their behavior, so you’ll miss seeing their expressions and body language. You can’t see all of the tools and technology they use. You won’t see the context of their environment. Plus, although it’s easy for participants to share a computer screen, it’s more technically challenging for people to share their mobile-device screen.
What You Should Do Instead
While remote research isn’t ideal, it can work fairly well for tasks that take place primarily on a computer screen. With screen sharing, you can see most aspects of the task. Plus, you can combine remote sessions with in-person sessions with participants who are nearby in your area. This lets you observe some participants in person, while including participants who are too far away to see in person.
Remote research works well for usability testing, which focuses narrowly on finding usability problems in a user interface. You can easily observe a usability test through remote sharing. Eliminating the need for travel lets you include more participants—even people working in very demanding professions, who may be anywhere in the world, allowing you to reach people you wouldn’t normally get to participate. However, you should try to avoid conducting remote user research in situations where you need to see tasks that occur off a computer screen.
Relying Only on Usability Testing Instead of Doing Generative User Research
Some organizations feel that they don’t need to conduct generative user research because they conduct usability testing. Others don’t realize that there actually are methods other than usability testing. Some people feel that they can’t really learn anything concrete until they have a prototype to which users can react. When there is a limited research budget, some people feel that it’s best to use it on usability testing.
Why This Is Half-Assed
These organizations are getting only half of the value of user research. Usability testing is very narrowly targeted toward specific tasks and questions and evaluating the usability of an existing product or design. Although testing is a great way to evaluate a design, it doesn’t reveal much information about the users, their tasks, the tools and information they use, or their environment. If you don’t conduct generative user research at the beginning of a project, you’ll have to rely on assumptions when designing a solution. You won’t learn whether those assumptions are correct until you conduct usability testing. By that point, you’ll already be far into the design process.
What You Should Do Instead
For an organization that’s just beginning to incorporate user involvement in product development, usability testing is a good first step. It will provide immediate, actionable information that you can convey to management to get them to see the value of involving users in the design process. But eventually, your process should evolve to incorporate user research at the beginning of projects to learn about the users, their tasks, their tools and technology, and their environment. Use information about actual users to inform your design, then conduct iterations of usability testing and design at later stages of the design process to evaluate and improve on your original design.
Not Taking Enough Time to Analyze the Results
The step of user research that requires the most time, but always seems to get the least time is analysis. Research often gets delayed because of difficulties in finding and recruiting participants, then working around their schedules. Because the due dates of design deliverables rarely change, the time between the end of the research sessions and completing the design deliverables is what always gets squeezed.
Why This Is Half-Assed
It’s a waste of time to conduct user research, then not properly analyze the results. Rushing through analysis often requires you to skip over valuable details in favor of making superficial findings. When suboptimal research findings are the result, your stakeholders may doubt the value of user research rather than realizing that a rushed analysis phase is the reason for the poor findings.
What You Should Do Instead
When planning user research, provide a realistic amount of time for analysis. Define a specific research-analysis task in the project plan. This task should be separate from the time you allocate to preparing the research-findings deliverable. If the project schedule changes because of delays in recruiting and scheduling, make sure the research deliverable date also changes. Try to prevent the analysis time from getting cut. When you know up front that you won’t have enough time to do detailed analysis, scope the research more narrowly so there is less data to analyze.
Don’t Enable Half-Assed User Research
Yes, adding user research at the beginning of a project involves extra time and money. However, taking the time to conduct user research properly saves time and money in the long run by providing the information you need to make the right design decisions more quickly, producing better designs that require fewer changes, and delivering a more successful product.
Ironically, half-assed user research actually costs more in the long run. Plus, it gives user research a bad name. Sure, we all have to cut corners sometimes, but if you find that you’re always cutting corners, refuse to be an enabler of half-assed user research. Instead of allowing your organization to get away with doing half-assed user research, either convince them to conduct proper user research or practice tough love by skipping user research altogether, allowing them to see the consequences of their actions.
Thank you for the article, I enjoyed the read and agree with the sentiment on the whole. Particularly with the idea that half-assed research can harm the perception of user research, which can be very damaging when we have to prove the value of research even at the best of times. That said, I think it’s a bit reductive to say that conducting user research remotely is half-assed.
I understand that you qualify this statement, but my opinion is that most types of research can be conducted effectively remotely. I can only think of contextual inquiry as something that is impractical to conduct remotely; it is the exception, not the rule, IMO. However, I’ve conducted usability testing, depth interviews, and card sorts remotely to good effect.
Sure, it’d be awesome to have the user in front of me, but the benefits of cost and turnaround far outweigh those (also, I’ve never encountered technical problems with a user sharing a mobile screen, at least none that didn’t also occur on a desktop).
Great article with good reminders for all of us working in environments where we’re constantly making tradeoffs around time/cost/quality. However, I disagree with the idea that remote research is inherently half-assed. When the ability to conduct research remotely came on the scene it opened up so much opportunity for me in terms of getting feedback from users who were geographically distant. For example, I had a project for an Intranet where the budget did not allow me to travel and yet we were able to conduct interviews and usability tests with end users in Brazil, India, and China. It radically enriched my understanding.
I would argue that it is exclusively conducting remote research coupled with a lack of generative research that creates the “half-assed” dynamic, but I think that we should embrace remote research and the opportunities it presents to extend our reach. I think there is a difference between not reaching an ideal and being half-assed and it is important for practitioners to be able to make this distinction.
I agree that usability testing, card sorting, and interviews work well remotely. Contextual inquiries, on tasks that primarily take place in a computer interface, can sometimes also work well remotely. However, when you need to see people performing their current tasks, in their natural environment, it’s important to be there in person.
So I suppose I would further qualify my opinion to state: when it’s necessary to observe people in person, but you are only allowed to do user research remotely (because that’s what’s always worked for every other project), then that’s when it becomes half-assed. Instead of being able to observe people in person, you have to rely on phone interviews, in which you only talk with people about their current tasks.
Jim has spent most of the 21st Century researching and designing intuitive and satisfying user experiences. As a UX consultant, he has worked on Web sites, mobile apps, intranets, Web applications, software, and business applications for financial, pharmaceutical, medical, entertainment, retail, technology, and government clients. He has a Masters of Science degree in Human-Computer Interaction from DePaul University. Read More
