How to Determine the Right Number of Participants for Usability Studies

Studies for Problem Discovery

Two important issues for problem-discovery studies are that it can be hard both to define a problem and rank the importance of discovered problems. According to D.A. Caulton, problems are often a factor of the interaction between a user and the product, not necessarily a static feature of a user interface. Not only might a specific problem exist for just some participants, a specific problem might exist for just a single participant on one day, but not on another day. Therefore, it can be very difficult to agree on what actually constitutes a problem. Plus, the ranking of problems is highly subjective.

When setting up a usability study, you need to consider the probable mean percentage and what minimal level of problem discovery is necessary—in other words, the average percentage of problems you hope to find, as well as the minimum percentage. As Table 1 shows, going from 5 to 10 participants greatly increases the expected level of problem discovery, but going from 15 to 20 participants has far less impact. These numbers are very important to understand. We can have fewer participants in a study if undiscovered problems would have a small impact on users, but we must have more participants if the stakes are high—for example, in life- or enterprise-critical situations. We can also have fewer participants if there will be opportunities to find important problems during a later round of testing.

Table 1—Numbers of problems found relative to number of participants

Number of Participants	Minimum % Found	Mean % Found
5	55	85.55
10	82	94.686
15	90	97.050
20	95	98.4
30	97	99.0
40	98	99.6
50	98	100

From Faulkner, 2003

Another important consideration is the complexity of the study itself. Nielsen is often criticized for citing research from simple studies with well-defined tasks. Clearly, the more complicated the tasks, the more complex the study should be, and the more participants may be necessary. In such a case, it is also necessary to consider training issues. If the entire target user population will receive exactly the same training, this effectively reduces the study complexity, so you can use fewer participants.

Studies to evaluate a prototype of a novel user-interface design often concern the discovery of severe show-stopper problems. Testing usually reveals such severe errors relatively quickly, so these tests often require fewer participants.

So, contrary to popular thinking, there is no one-size-fits-all solution for problem-discovery studies. Context and complexity have a big impact on the appropriate number of participants necessary for such a study. This reality is probably a factor in the diversity of widely cited advice. For example, both Virzi and Perfetti and Landesman found that the appropriate number of participants for many studies ranges between three and twenty. Nielsen’s statement that five participants are enough was clearly based on simple studies, and he has more recently stated that the number of participants must increase with the complexity and criticality of a study. Faulkner argued that ten participants would probably find a minimum of 82% of problems. Turner believes seven participants may be ideal, even for complex studies.

In summary, research suggests that from three to twenty participants can provide valid results, and a good baseline is between five and ten participants. In general, there should be more participants for more complex, highly critical projects; while fewer participants are necessary when testing more novel designs.

Comparative Studies

Usability professionals often run A/B tests, which involve testing two designs against each other to determine which is better. Often, new design B is tested against the currently implemented design A. In other words, an A/B test is a hypothesis test—that is, a team hypothesizes that interface B will work better than interface A, so runs a test to determine whether that is true.

Comparative studies generally use usability metrics such as task-completion rate and time on task. These metrics are highly objective, so usability professionals often want to present statistically significant results from such studies. These results can be compelling, but they have some common pitfalls.

To reject the null hypothesis, we must be reasonably sure that different groups of participants and the target audience are equally skillful—that is, that design B did not have a better result because group B was significantly more skillful. This can be difficult to determine, so is a reason for caution when quoting such results.

It’s also important to remember that, even though a finding may be statistically significant, it might not be meaningful. For example, if you tested a new user interface with 100 participants and found they were able to complete a task 2% faster, the result would be statistically significant, but not necessarily meaningful enough to act upon.

One approach to comparative studies is to make them open ended, incrementing the number of participants in a group until one of the following occurs:

• The resulting data becomes significant.
• It becomes apparent that there will be no significant finding, despite testing with additional participants.
• You cannot continue the study because of time or budget constraints.

While open-ended comparative studies may be practical in academia, this is not typically a useful approach in the commercial world, where we need to set time and budget constraints for studies. Another approach is to test with very large participant groups, which is likely to produce statistically significant results—assuming that is possible. However, such tests are often too expensive and time consuming for commercial contexts.

These problems have led many usability professionals to guesstimate the appropriate participant-group size for comparative studies. The Common Industry Format (CIF) for Usability Test Reports advises a minimum of eight participants. This fits with advice from Nielsen and Landauer that we’re unlikely to get statistically significant findings from groups of less than eight participants, but quite likely to get them with groups of 25 participants. Spyridakis and Fisher found that groups of ten to twelve often provide statistically significant results.

Punctuated Studies

Another approach that Ritch advocates is to perform usability studies in stages. For example, during the first stage, you might test with five participants, and the primary intent of the testing would be to catch any show-stoppers. You’d schedule additional stages of testing, each with five participants, with the intent of generating statistically significant results. If, at any stage, you catch show-stoppers or produce statistically significant results, you can terminate the study early and allocate your time and financial resources elsewhere. This iterative approach, though uncommon, is often optimal and fits well with the philosophy of user-centered design and agile approaches. However, it can be harder to predict the time and money your study would require, which can make things difficult from a management angle.

Summary

There is no one-size-fits-all solution for determining the optimal number of participants for a usability study. Rather, we should think more in terms of ranges like those shown in Figure 1. Problem-discovery studies, which are subjective in nature, typically require between three and twenty participants, with five to ten being a good baseline. Generally, the number of participants should increase with study complexity and product criticality, but decrease with design novelty.

From Macefield, 2009

For comparative studies—which are typically more objective than problem-discovery studies because of their heavy reliance on metrics—group sizes of between eight and 25 participants typically provide valid results, with ten to twelve being a good baseline. Generally, group size should increase if you want statistically significant results; punctuated studies can be an efficient way of achieving this goal.

Whatever the number of participants you use for a particular study, you should always understand the assumptions, limitations, and risks that are associated with your decision. 

References

Caulton, D.A. “Relaxing the Homogeneity Assumption in Usability Testing.” Behaviour & Information Technology, Vol. 20, 2001.

National Institute of Standards and Technology. “Project: Usability Standards.” National Institute of Standards and Technology, 2015. Retrieved October 1, 2015.

Faulkner, L. “Beyond the Five-User Assumption: Benefits of Increased Sample Sizes in Usability Testing.” Behavior Research Methods, Instruments, and Computers, Vol. 35, No. 3, 2003.

Landauer, Thomas K. “Research Methods in Human-Computer Interaction.” In Handbook of Human-Computer Interaction, ed. Martin G. Helander. Amsterdam: North Holland, 1988.

Macefield, Ritch. “How to Specify the Participant Group Size for Usability Studies: A Practitioner’s Guide.” Journal of Usability Studies, Vol. 5, No. 1, 2009. Retrieved October 1, 2015.

Nielsen, Jakob. “Why You Only Need to Test with 5 Users.” Nielsen Norman Group, March 19, 2000. Retrieved October 1, 2015.

Nielsen, Jakob, and Thomas K. Landauer. “A Mathematical Model of the Finding of Usability Problems.” Proceedings of ACM INTERCHI, 1993.

Perfetti, Christine, and Lori Landesman. “Eight Is Not Enough.” UIE, June 18, 2001. Retrieved October 1, 2015.

Spyridakis, J. H., and J. R. Fisher. “Usability Testing in Technical Communication: The Application of True Experimental Designs.” Technical Communication, Part 4, 1992.

Turner, Carl W., James R. Lewis, and Jakob Nielsen. “Determining Usability Test Sample Size.” In International Encyclopedia of Ergonomics and Human Factors, 2nd Ed., Vol. 3, ed. W. Karwowski. Boca Raton, FL: CRC Press, 2006.

Virzi, R. A. “Refining the Test Phase of Usability Evaluation: How Many Subjects Is Enough?” Human Factors, Vol. 34, 1992.