A Complete Guide to Usability Testing

May 8, 2023

I headed an education company for ten years. When my line of yoga and mindfulness products launched in 2015, I recognized the need to test the products with actual customers, but struggled to conduct user research in a meaningful or consistent way.

The products were downloadable, printable educational materials such as illustrated yoga cards, activity sheets, and teacher’s manuals. Although that format was popular when we had begun R&D, it had become less ideal as technology evolved and customer expectations shifted. Our users had to print and, in some cases, even cut out the printed materials. Sometimes they laminated them, too. It was obvious these extra steps were creating barriers to conversion and retention.

My desire to learn more about technology led me to shift the focus of my career to software as a service (SaaS), user research, and UX design. When I first learned about usability testing, a light bulb went off in my head regarding my experience creating my product line. But hindsight can be costly.

Champion Advertisement

“The cost of fixing an error after development is 100x that of fixing it before development.”—The ROI of User Experience, Dr. Susan Weinschenk

In this article, I’ll take a deep dive into usability testing as a method of user research. I’ll explore what usability testing is, what questions usability testing typically answers, how to conduct usability tests, the ideal sample size, and the limitations and benefits of usability testing. If conducting usability testing suits your use case, this article gives you the facts, so you can reason with your stakeholders and convince them that funding testing is a worthwhile investment.

Remember, user-centered design requires us to consider the needs of potential and actual users and offer them user-friendly solutions to their problems. In the case of my yoga and mindfulness products, users could accomplish their goals, but not easily. Just a few video observations of users early during the design process could have helped me reimagine the product line.

What Is Usability Testing?

Usability testing is a research method whose aim is assessing how easily participants can complete tasks, typically using a software application. You can conduct usability testing at multiple points during the design process. You can do concept testing with a low-fidelity prototype, an interactive prototype, or a completed product or feature.

The three key elements of usability testing are the facilitator, or user researcher, participants, and tasks. As the participants complete the tasks, the researcher observes them—often remotely—and, in some use cases, listens for their feedback. Observing actual users or people who match the target audience influences the iterative design process by revealing where people are having problems using an application effectively. Researchers can collaborate with UX design teams, providing additional context to the customer journey and enabling them to create an updated customer-journey map (CJM).

Observing usability tests provides researchers with data that enables them to monitor the totality of the customer journey, looking for barriers, inconveniences, and general usability problems or trends.

An Example of Usability Testing in a Clinical Setting

I recently spoke to a physician informaticist to learn more about usability testing in the field of medical informatics. He shared information about an A/B test their usability lab conducted on a cancer nurse’s workflow when calling a patient.

First, the researchers learned about the current state, which was infamous for its poor design. After speaking with a patient, nurses were recording the information from the call in an unstructured note in the Electronic Health Record (EHR). However, they were also keeping track of the patient in two different shared Excel spreadsheets, using Microsoft Access for reporting purposes, and using a shared Outlook calendar to track the patient’s appointments. Based on what they observed, the informaticist suggested changes to the note template, making the valuable information that the nurses had collected available in the EHR and visible to other care providers. They could also use this data in reports on the clinic’s cancer patients.

Once the EHR’s programmers had implemented the workflow changes, the usability lab again observed the nurses in real time. The new workflow resulted in significant time savings for the nurses, reducing the average time to document a patient call from six minutes to two minutes and twenty-one seconds. Since each nurse serves twelve patients per day, this equates to a savings of about 42 minutes per nurse each day. This works out to approximately four weeks of time saved per nurse each year. The usability lab also interviewed the nurses after the redesign of the workflow to get their direct feedback about the changes, then took the nurses’ perceptions of the changes and measurable quantitative outcomes to senior leadership. The hospital leadership were, thus, able to justify their investment because of the time the nurses saved.

As this example illustrates, usability testing can typically get you answers to the following questions:

What steps do users take on their user or customer journey?
Where do they get stuck or abandon a task?
How can you improve the usability of specific features or the overall product experience?

By answering such questions through usability testing, you can learn what users understand about a particular screen or the language it uses, as well as why they did what they did.

When I think back to my experience designing my product line, I am confident that observing customers throughout their journey—from the time of purchase, through downloading, printing, and preparing their content—would have been enlightening. Metrics such as the time they spent on specific tasks would have demonstrated other opportunities to save them time and effort. The key to the successful evolution of our products through an iterative design process requires proper user research, and for my use case, usability testing, which could have been as instrumental in their improvement as the workflow testing was for the cancer nurses.

The goal of usability testing is to tailor a product to address users’ needs. For the clinical usability testing that I described earlier, the nurses needed a better workflow to save them time.

Moderated and Unmoderated Usability Testing

Moderated usability testing includes real-time interviews with participants as they complete the tasks that the facilitator assigns to them. In contrast, for unmoderated usability testing, there is no facilitator present. Participants typically click a link that takes them to a product or feature they can use to complete tasks. Unmoderated testing often employs question prompts so researchers can understand why users do what they do.

Use Cases

A prime use case for usability testing is the design of software user interfaces (UIs). Usability testing can identify opportunities to improve the user experience, as well as the software’s functionality. The outcome of the hospital usability testing was that the nurses were able to take better notes, more efficiently, which in addition to being helpful to them, also improved the overall flow of data across the entire organization, leading to improved patient outcomes. Now the patient’s other physicians can see the patient notes from the cancer center, enabling their entire healthcare network to deliver improved care. The usability lab is eager to present these findings to additional senior stakeholders.

Close-Ended and Open-Ended Usability Tests

Close-ended usability tests define specific parameters for test participants. For example, the researchers might ask the users of an application to test only its new features. In contrast, an open-ended usability task could ask the same group of users to explore the whole application, inclusive of the new feature.

Sample Size

Sample size refers to the number of participants you must recruit for a usability study. As we saw in the case of the usability tests that the hospital usability lab conducted, a sample size of just two was impactful. According to the Nielson Norman Group, a pool of five users can reveal 85% of the problems they might face when using a product. Thus, usability testing is both time efficient and cost effective.

How to Conduct a Usability Study

The process of conducting a usability study comprises eight steps, as follows:

Plan the usability study. As for all user-research methods, it is important to identify your research questions and design hypothesis prior to beginning the study. In the case of unmoderated usability testing, it is also important to prepare materials such as instructions or instructional videos for users.
Recruit users or participants. It can be easier to recruit participants for usability testing if you are testing a product that already exists and has users, because you can reach out to the subset of users that utilizes the feature, workflow, or product that you are testing. Sometimes it is necessary to recruit from a participant pool that matches particular specifications. Always vet participants to make sure they match your requirements to ensure that you’ll be able to understand your target users’ problems and create a solution that addresses them.
Conduct the test. Whether you’re conducting moderated, unmoderated, open-ended, or close-ended usability tests, you should always carefully record your observations of participants’ behaviors. Try using complementary technologies such as Hotjar to gain a deeper understanding of users’ behaviors, as well as their sentiments.
Interview participants. Once you’ve completed the test sessions, conduct user interviews to ask any final questions that would help you learn about participants’ perceptions, frustrations, and sentiments.
Analyze the findings. When analyzing your data, patterns of usability issues often arise. Make a list of priority fixes to address, starting with actual bugs.
Recommend design changes. Review your design recommendations with the appropriate stakeholders. Their greenlighting your suggested changes depends on their understanding the research outcomes.
Test the design changes. Once your team has implemented your design changes, you can either conduct another round of usability tests or do A/B testing to test different versions of a solution.
Report your findings to stakeholders. Report on the design improvements you were able to make after conducting usability testing, including the time users saved and what tasks were easier to complete.

Benefits of Usability Testing

As a user-research method, usability testing provides the following benefits:

A small sample size can yield helpful data, saving you time and money!
You can conduct remote research with many participants concurrently.
Analyzing the data from usability testing can lead to design iterations of a product, feature, or workflow that improve the user experience—again, by such measures as time saved or the ease of completing a task.
Researchers can parse their data to look for positive and negative feedback, general comments, and requests for new features. The team can then determine which problems to prioritize fixing or add to the backlog. Typically, the solutions that you prioritize should begin with bug fixes.
Researchers can learn which features users seldom use and consider whether to retire them.

Limitations of Usability Testing

Of course, no user-research method meets all needs. Let’s consider a few limitations of usability testing, as follows:

You should combine usability testing with other research methods such as user interviews—for example, to understand user sentiment. Usability testing does not yield data about users’ motivations or sentiment on its own.
Usability testing requires a working prototype or an existing product.
For many use cases, participants must have their own device such as a smartphone, which might exclude certain demographics from the research-participant pool.

Recruiting High-Quality, User-Research Participants

The recruiting of high-quality, user-research participants is paramount. Whether your research objectives lead you to choose usability testing or another research method, the outcome of your research project hinges on the quality of the research participants you are able to recruit. Prioritize the following three factors when sourcing participants for your research:

Relevance—Is the participant an actual or likely user of your product who fits your target persona?
Articulation—Is the participant able to deliberate thoughtfully and communicate his or her insights effectively?
Authenticity—Has the participant exhibited an interest in using the product and helping to improve the user experience, exhibiting motivation beyond whatever incentive you’re offering for his or her time?

You can streamline your participant-recruitment process for user research by using a participant recruitment and management platform. If you cannot easily access a niche audience or user base, a participant recruitment and management platform can save you time, energy, and resources.

Conveying the Importance of User Research to Stakeholders

“Businesses that invest close to 10% of their total budget on [User Experience] (UX) are likely to see tremendous [return on investment] (ROI). User research improves user experience, and it must be done right.”—Amazon AWS

I’ve analyzed many articles, academic papers, and blog posts to get a sense of the impact that user research can have on the user experience and its return on investment (ROI). Key insights include the following:

“The impact of UX is crystal clear: the more satisfied your users are, the more likely they are to do whatever it is you are encouraging.”—Abby Covert, author, How to Make Sense of Any Mess
“The world is not made for the user, which will cost businesses trillions of dollars.”—Raffaela Rein, Founder, The UX School by Career Foundry
“The cost of fixing an error after development is 100x that of fixing it before development.”—The ROI of User Experience, by Dr. Susan Weinschenk

In Summary

Usability testing has proven to be a powerful research method. It provides a wealth of data on the challenges customers might face when using a product, as well as insights into the customer journey. Plus, when you pair usability testing with prompts, user interviews, or use complementary technologies, testing can show you where and why users become frustrated. By placing users at the center of an iterative-design process, we can solve their problems, meet or exceed their expectations, and ideally, create loyal customers.

In Usability Testing