In my 19 years of involvement in User Experience, I’ve heard some really bad ideas about how to conduct user research. Don’t get me wrong—I’ve had a few bad ideas myself. However, in this column, I’ll write about the very worst ideas I’ve ever encountered.
Most of these ideas originated as attempts to shortcut the difficulties, time, effort, and cost of user research. The results were far from optimal.
Bad Idea: Let’s Offshore Our UX Work
“We’re offshoring all of our development work. Why don’t we offshore the UX work, too?”
Back in the early 2000s, I worked for a large corporation that was thinking about adopting the fad of offshoring as much work as possible to India, and our management team decided to experiment with offshoring the work of our UX team.
Champion Advertisement
Continue Reading…
Why This Was a Bad Idea
Having UX professionals in India would be ideal for conducting user research with and designing for Indian users and clients. However, our clients and most of our users were in the United States. Our projects involved designing enterprise applications and the company intranet. At that time, we had the advantage of being physically close to the users—our fellow employees. This gave us the luxury of easy access to research participants, at little or no cost, for activities such as contextual inquiries, interviews, usability testing, and card sorting. We were also physically close to our clients, as part of the same corporate culture.
How This Worked Out
The company hired one or two UX professionals in India, who could perform activities such as heuristic evaluations, remote usability testing, and UX design. However, their physical and cultural distance from our users and clients was a great disadvantage that overshadowed the cost savings of offshoring. The idea that the company could offshore our work demoralized many members of our UX team, and most of them left the company over the next year or two.
Lessons Learned
Because User Experience requires an in-depth understanding of both the business and the users, UX teams should ideally be collocated with business leaders and users. Therefore, User Experience is not the best profession to offshore to people who are in locations far away from the business and users. Although it is possible to do some UX activities at a distance, it is important for UX professionals to be physically and culturally close to clients, stakeholders, and users.
Bad Idea: Let’s Hire Someone Off the Street to Moderate Usability Tests
“Why do we need all these expensive researchers? Anybody can read a discussion guide and take notes.”
At a design firm where I worked, someone in upper management proposed our hiring inexperienced, low-paid employees to moderate usability tests as a way to save money, rather than using experienced, higher-paid UX researchers. Our researchers would still create the discussion guides, analyze the results, and write the reports, so the low-paid moderators would only need to read the guides and take notes.
Why This Was a Bad Idea
Obviously, moderating usability tests involves far more than reading a discussion guide and taking notes. It requires a great deal of skill and experience to create the right rapport with research participants, pose the right questions, probe with additional questions, make changes to a session when necessary, observe participants’ behavior, and note the most important findings. Even if off-the-street moderators were able to do an effective job, the act of personally moderating and observing sessions provides a depth of knowledge that you can never get from simply reading someone else’s notes.
How This Worked Out
Luckily this was simply a proposal that never went past the bad-idea stage. However, learning that management had such a poor understanding of the value of their work demoralized the UX research team.
Lessons Learned
Conducting generative user research requires a lot of skill and experience. It’s not possible to cut corners by giving part of the work to inexperienced, lower-paid employees. However, usability testing can be a good activity to assign to newer UX researchers. Conducting usability testing is often easier than using more complicated, unpredictable user-research techniques such as field studies. But it’s still best to have the same UX researcher involved in all phases of a study. It doesn’t make sense to have someone else moderate sessions for another researcher.
Bad Idea: Let’s Use Bob from Accounting as a Participant. He Uses the Software.
“I don’t think we can get actual customers. Let’s use our own employees as participants.”
At another company, in addition to a lack of time and money for user research, we had difficulty getting actual customers to participate in our studies. So someone proposed using people from our company’s Accounting team, who they assumed were similar to our users.
Why This Was a Bad Idea
Using your company’s employees as participants in your user research or usability testing is a bad idea because they don’t represent real users. Even if they have some similarities to your actual users, their association with your company and their knowledge of the company’s products can affect their behavior and bias their opinions. This, obviously, can provide misleading results.
How This Worked Out
The few times I’ve had to use this approach, it did not provide very good results. Nevertheless, this can sometimes work okay for usability testing under two conditions:
If the target user for the product you’re testing is part of a general audience that your fellow employees actually match
If you’re not testing your company’s products
Lessons Learned
Recruit participants who are representative of your actual users. Resist the temptation to use internal employees as research participants—unless they’re representative of your users, and they don’t have insider knowledge of the product you’re testing.
Bad Idea: Let’s Not Tell Participants Who I Am
“If the participants might be biased or uncomfortable because I’m there, let’s just not tell them who I am.”
At a company where I previously worked, I had a client who wanted to observe in-home, field studies. We felt that the presence of such a high-level executive from the company we were discussing with participants would make them feel uncomfortable and be unwilling to speak freely and honestly about their experiences with the company. Plus, we wanted to limit the number of people coming into the participants’ homes to avoid overwhelming them. So the executive proposed not telling participants who he was or the company for which he worked.
Why This Was a Bad Idea
We thought it would be misleading and unethical to either lie or fail to mention who we were or what companies we were from. The participants had a right to know who we were—especially because we were going into their homes.
How This Worked Out
We compromised and allowed one client to attend each session, so we had a maximum of three observers, including myself, as the user researcher, and the UX designer. This would prevent our overwhelming participants by bringing too many people into their homes. However, we did not compromise on identifying ourselves. We gave our names and company affiliations to each participant, and knowing this seemed to have less effect on participants than we had thought it would.
Lessons Learned
First, I learned that my bad idea was thinking that clients shouldn’t attend field studies. I feared clients might disrupt the sessions and make participants feel uncomfortable, thereby preventing our seeing participants’ natural behavior. However, I learned that having clients observe sessions can work well, as long as you coach your clients on how to observe properly, without disrupting the session. Plus, directly observing field studies is an excellent way for clients to learn the value of user research.
Bad Idea: Read the Introduction to Usability Testing to Participants Verbatim
“Read the introduction to participants verbatim, from a card, so all participants get the same instructions.”
When I was in graduate school, getting my Human-Computer Interaction degree, our professors advised us to read usability-test instructions verbatim to each participant, so everyone receives exactly the same information. Their intention was to prevent various participants from getting different instructions, which might affect the results. They even told us to begin the instructions with an explanation of why we were so strangely reading the instructions off a card: “Hello, my name is Jim Ross. I am reading you these instructions off a card so each participant receives the same introduction….”
Why This Was a Bad Idea
I’m sure reading the instructions verbatim would ensure that each participant gets the same introduction, but it also produces a very stilted, unnatural initial impression. This makes it difficult to establish a good rapport with the participant. Plus, it adds to the unnatural feeling of being part of a formal study or test.
How This Worked Out
Although I used this method in graduate school, I found that it felt extremely unnatural in the real world. So I quickly abandoned this practice in favor of a more relaxed, natural style.
Lessons Learned
While it’s important to give each participant a similar introduction to their user-research session, it’s more natural and seems more human to give each of them a more improvised introduction. So I create a bulleted list of the items I need to mention, which I can glance at to ensure I remember all the points I need to talk about. Then I can relay this information in a conversational manner, while looking participants in the eye. This seems much more natural and puts participants at ease.
Bad Idea: Let’s Do the Research in a Conference Room
“We don’t have a lot of room here at my desk. Let’s move this to a conference room. We’ll have more space there.”
Several times in the past, I’ve had clients or participants try to move a contextual inquiry from the location where participants normally performs their tasks to a conference room. They said, “There will be more space there, and we won’t disturb other employees.” When I explained the importance of seeing participants in their typical work environment, they told me they would bring their notebook computer and paperwork to the conference room to show me.
Why This Was a Bad Idea
Ideally, field studies and contextual research should involve observing users performing their tasks in their usual environment. Moving a contextual inquiry to a conference room moves participants’ tasks to an artificial environment, which defeats the purpose of traveling to participants’ workplaces to observe their natural behavior and environment.
How This Worked Out
Sometimes in such situations, I’ve been able to convince clients and participants of the importance of conducting the research in their natural environment. However, in other situations, they’ve resisted, and I’ve had to go along with moving research to a conference room. While the user research that I conducted outside the context of the participants’ natural environment didn’t lose all of its value, it did become less useful.
Lessons Learned
At the beginning of a project, make sure your clients understand the importance of observing user-research participants’ performing tasks in their usual context and environment. After recruiting participants, send each one an email message, introducing yourself and explaining what you’ll be doing. Stress the importance of observing their tasks in their usual workplace. I’ve found this usually prevents problems.
Bad Idea: I Thought We Could All Meet Together
“Frank and Susan also do this work. So I thought they could join us and give you their opinions.”
In a few cases over the years, I’ve showed up to conduct a contextual inquiry and been surprised to find that the participant had invited several coworkers to participate in what’s suddenly become an impromptu focus group.
Why This Was a Bad Idea
Usually these participants think they’re doing you a favor. You wanted to interview them about their work, so they’ve decided to invite several additional people you can interview. The problem is that this turns into a group discussion, which doesn’t let you observe tasks or have individual discussions.
How This Worked Out
These impromptu group sessions usually don’t work out well. Sometimes I’ve been able to convince the participant to do the planned individual sessions. In other cases, I’ve been forced into accepting these group discussions.
Lessons Learned
When you send your introductory email messages to participants, explain that you want to observe them performing their tasks individually. Emphasize the need for individual sessions with participants.
In general, if you find that a participant has scheduled an impromptu group session, thank the extra people for their interest in participating, but refuse to conduct a group session. If you have time, you could schedule those extra people for their own, individual sessions.
Nevertheless, I have found that it sometimes does make sense to include more than one person. Perhaps there are two people who frequently interact during a task or tasks pass from one person to another. So you should be flexible when you discover situations in which it makes sense to include more than one person.
Bad Idea: We Don’t Need a Report
“We don’t need a user-research report. No one ever reads long reports anyway.”
Analyzing user-research results and creating reports is often the most time-consuming part of a user-research project. Over the years, I’ve experienced various trends toward minimizing the depth of analysis, the time it takes to create a deliverable, and the length of research reports.
Why This Was a Bad Idea
Yes, it’s true that most people don’t have the time or desire to read a very long, detailed user-research report. But it is necessary to have some kind of deliverable that at least summarizes your findings and recommendations. Even if the entire project team has observed the research, someone must gather and document the findings, ensuring that everyone has a shared understanding of the findings and recommendations.
How This Worked Out
Minimizing or completely eliminating research deliverables is an extreme move. The time or cost savings is rarely worth it. After spending a lot of time and money conducting research, it’s wasteful to rush through or skip the analysis and documentation of your findings.
Lessons Learned
Provide shorter, higher-level deliverables when that’s what the project team or client wants. Provide longer, more detailed reports and presentations when time allows. If a product team doesn’t want to wait for a detailed report, provide your high-level, key takeaways shortly after the conclusion of your research. This will tide them over until you can deliver your more detailed report.
Overcoming Such Misunderstandings
Most of these bad ideas came from people who didn’t understand user research or appreciate its value. Either they may have misunderstood what was important or they were seeking ways to make user research less expensive or time consuming. Happily, most of these misunderstandings took place long ago. Over time, as user research has become better established and more well understood, I’ve been encountering fewer such bad ideas. But I’m sure all of you who are reading this column have experienced your share of bad ideas over the years. Please feel free to share some of the worst ideas you’ve ever heard in the comments.
This was a good article and should be widely read. Give us more of these. In our Agile world, we would call any of these missteps an Anti-Pattern, and we would hope to recognize them in their varied form. There are probably many more for you to tell us about. Please continue…
This is a great article, and the points you cover are well worth mentioning in hopes that others can learn from your experience. I agree with all of them, except one: the bit about not reading from a script. Just as you experienced this requirement in grad school and subsequently found that you could do better with notes or a bulleted list, everyone has to start somewhere. In my view, this requires reading the script. Then, once the moderator’s comfort level increases over time, the script can be abandoned for a more natural approach that still covers the essentials. However, I have seen seasoned professionals bomb out without the support of the moderator’s script, so a more relaxed style does not work for all.
I guess it depends on what the bad idea is and how bad it is. Maybe you can improve it somewhat and still get value. I talk about this more in a previous UXmatters article, “Avoiding Half-Assed User Research.”
Jim has spent most of the 21st Century researching and designing intuitive and satisfying user experiences. As a UX consultant, he has worked on Web sites, mobile apps, intranets, Web applications, software, and business applications for financial, pharmaceutical, medical, entertainment, retail, technology, and government clients. He has a Masters of Science degree in Human-Computer Interaction from DePaul University. Read More
