An Overview of Expert Heuristic Evaluations

Conducting Expert Heuristic Evaluations

When experts conduct heuristic evaluations—particularly outside academic contexts—they do not usually attempt to map user interface problems directly to specific heuristics because the mapping of problems to heuristics is typically many-to-many, convoluted, and multidimensional. Mappings of specific problems to heuristics would be very hard to structure in any report—particularly when presenting findings to audiences who are not specialists in usability engineering—and explanations of the mappings would likely be verbose and complex. Instead, experts conducting such evaluations typically just state what heuristics they’re using, then keep them in mind when identifying problems and recommending solutions.

It is common for experts performing heuristic evaluations to do the following three key things in their reports:

• Uniquely identify each issue and any associated recommendations. This lets you easily refer to them in any report or discussion.
• Prioritize, or rank, each issue according to its severity, and codify the ranking in some way. Myself and many others use this traffic-light taxonomy:
  
  • example of a best practice (Green)
  • minor problem (Yellow)
  • serious problem (Orange)
  • critical problem (Red)
• Relate each issue and its recommended solutions to screenshots of the user interface. This enables people who read your report to see the problems to which you’re referring without their needing to have access to the system you’ve evaluated.

Other than these common practices, the approaches that usability experts use in performing expert heuristic evaluations tend to vary, but often include the following:

• screen-by-screen or page-by-page analysis—Taking this approach makes it easier to structure your report.
• using a goal-task-action model—Such a model is consistent with that for a cognitive walkthrough, which is a different method of evaluation.
• following prescribed user journeys—These may be the same as those for a parallel usability study for a system.

How Many Experts Do You Need?

Nielsen [8, 9] and Landauer [9] have argued strongly that heuristic evaluations should be performed by three experts. They based their recommendation on extensive research into how many problems individual experts would be likely to find in a user interface when conducting this type of evaluation. They found that three experts identified about 60% of the problems and that, as the number of experts increased, the number of additional problems that they found tailed off exponentially. For example, it took ten experts to find 85% of the problems. This led them to the conclusion that three experts offer the optimal cost-benefit ratio for heuristic evaluations.

There has been much discussion about Nielsen and Landauer’s research and their associated recommendation since its publication. In the last decade, many usability engineers have questioned its validity—in part, because the cost of using three experts is often similar to or even greater than that of conducting a usability study with a decent sample size. Of course, this goes against the very idea of a discount method of usability evaluation! So, while the use of multiple experts remains common, many organizations now use just one or two experts—particularly if they’re conducting an expert heuristic evaluation in conjunction with a usability study.

Reporting the Results

Some common ways of presenting expert heuristic evaluations are in printed reports and PowerPoint presentations. A less common method is to produce a video presentation, using a tool such as TechSmith’s Camtasia or Morae, which lets you synchronously capture the screen output, audio of the expert talking, mouse movements, and optionally, a video of the expert talking. I like using this method because I can often produce a video more quickly than a formal report or PowerPoint deck and, at the same time, provide richer feedback.

Whichever format you use for reporting your findings, it’s important to do four things:

• State clearly what heuristics you have used. This helps to establish the credibility of your evaluation because the heuristics come from authorities in usability engineering.
• Provide the CV, or resumé, of each expert doing an evaluation. It’s critical to establish and maximize the experts’ credibility to ensure that your client does not discount the work as “just another person’s opinion.”
• Provide an executive summary. You can’t expect executives to go through your whole evaluation! They just need to know your key findings and what they need to do next to fix any issues that need fixing—which usually means allocating more time and resources to a project.
• Arrange some time to discuss your evaluation with your client. Preferably this discussion should occur face to face. Just as for usability studies, reports on the findings from expert heuristic evaluations often contain a lot of bad news. Plus, you may be criticizing someone’s pride and joy. Although, in an ideal world, people would take professional criticism as constructive criticism, it’s not uncommon for people—especially the team who designed the user interface that you’ve evaluated—to take it personally. So it’s important to meet with them in person to soften the blow, clarify your feedback, and let them know the good news, too—that everything is fixable, and you can help them to fix it!

Using Axure for Expert Heuristic Evaluations

Since I’m an Axure expert, I’ve recently started using this tool when doing expert heuristic evaluations. In brief, for those who are familiar with Axure, I use its image-editing tool to slice, crop, and manipulate screen shots. I use its customizable page and widget annotation features, in combination with a custom widget library that I’ve developed, to cross-reference and prioritize issues on each page. Then, I use its customizable specifications generator, along with a custom Microsoft Word template that I’ve developed, to automatically output a very slick Microsoft Word document. Please contact me if you’re interested in learning how I do this in greater detail. 

Endnotes

[1] Travis, David. “How to Carry Out a Usability Expert Review.” Udemy, undated. Retrieved April 25, 2014.

[2] Nielsen, Jakob, and Rolf Molich. “Heuristic Evaluation of User Interfaces”. In CHI ’90 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York: ACM Press, 1990.

[3] Nielsen, Jakob. “10 Usability Heuristics for User Interface Design.” Nielsen Norman Group, January 1, 1995. Retrieved April 25, 2014

[4] Nielsen, Jakob. “Top 10 Mistakes in Web Design.” Nielsen Norman Group, January 1, 2011. Retrieved April 25, 2014.

[5] Lund, Arnold. “Expert Ratings of Usability Maxims.” Ergonomics in Design, Volume 3, Issue 5, 1997.

[6] Tognazzini, Bruce. “First Principles of Interaction Design (Revised & Expanded).” Ask Tog, March 5, 2014. Retrieved April 25, 2014.

[7] Shneiderman, Ben. “The Eight Golden Rules of Interface Design.” University of Maryland, undated. Retrieved April 25, 2014.

[8] Nielsen, Jakob. “Finding Usability Problems Through Heuristic Evaluation.” In CHI ’92 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York: ACM Press, 1992.

[9] Nielsen, Jakob, and Thomas Landauer. “A Mathematical Model of the Finding of Usability Problems.” In Proceedings of the ACM/IFIP INTERCHI '93 Conference. New York: ACM Press, 1993.