Still Waiting for Their Big Break
Several research methods from the early days of usability—in the 1980s and 1990s—never really caught on with UX professionals:
- GOMS analysis
- cognitive walkthrough
- pluralistic walkthrough
Perhaps their rigidity, impracticality, or the lack of user involvement doomed these methods to a mostly academic existence in textbooks and courses on human-computer interaction.
GOMS Analysis
GOMS (Goals, Operators, Methods, and Selection) analysis and similar variants such as KLM-GOMS (Keystroke-Level Model GOMS) could be the poster child for techniques that, while they have been widely discussed in academia, never caught on with UX professionals. Like many others who have a degree in human-computer interaction, I first learned about and last used GOMS in graduate school.
GOMS analysis is a method for determining how long it would take a user to perform a task with a user interface by breaking the task down into each user interaction and the time it would take to perform it. The UXPA’s Usability Body of Knowledge [1] describes GOMS as, “A family of predictive models of human performance that can be used to improve the efficiency of human-machine interaction by identifying and eliminating unnecessary user actions…. Empirically derived values for basic operators like keystrokes, button presses, double-clicks, and pointer movement time are used to estimate task times…. To obtain the predicted time for a task, you add the times for individual operators.” This complicated description alone might tell us why UX professionals never adopted this method as part of their practice.
Why Didn’t GOMS Catch On?
There are several reasons why GOMS never really caught on:
- GOMS’s narrow focus is on efficiency and the time it takes to perform actions, which is a very limited view of user experience. Most UX and usability professionals are interested in the entire user experience rather than individual, minute interactions that allow them to predict efficiency.
- It’s not very practical to use GOMS to measure every one of the many interactions in a typical user interface.
- GOMS doesn’t involve users. In the early days of usability, when it wasn’t as easy to get access to users, some saw techniques like GOMS as good alternatives. As usability testing and user research became more widely accepted and practiced, UX professionals came to perceive activities that didn’t involve users as less valuable—and even frowned upon them.
- GOMS never gained much exposure outside human factors and human-computer interaction programs. Few students who learned GOMS found it practical to use this approach in the real world after graduating.
Cognitive Walkthrough
Over my 12-year career in user research, the only time I performed a cognitive walkthrough was for a human-computer interaction class in graduate school. One of several types of usability inspection methods from the early years of usability, the cognitive walkthrough provides a way to evaluate user interfaces without involving users.
In a cognitive walkthrough, one or more usability professionals walk through a user interface, attempting to perform selected tasks, while asking four questions from the user’s perspective:
- Will the user try to achieve the right effect?
- Will the user notice that the correct action is available?
- Will the user associate the correct action with the effect he wants to achieve?
- If the user performs the correct action, will he see that he is making progress toward a task’s solution?
Why Didn’t the Cognitive Walkthrough Catch On?
Doing cognitive walkthroughs failed to catch on for a couple of reasons:
- They don’t involve users.
- This is a rather complex, inflexible method in comparison to simpler inspection methods such as heuristic evaluation and expert review. If you’re going to use a technique that doesn’t involve users, why not use something that’s simpler and more flexible?
Pluralistic Walkthrough
The pluralistic walkthrough is an even more obscure usability inspection method than the cognitive walkthrough. You’ve probably never even heard of it unless you have a human-computer interaction background. I first read about this technique in grad school in Nielsen and Mack’s book, Usability Inspection Methods. I’ve never used this method; nor have I ever come across anyone else who has used it.
The UXPA’s Usability Body of Knowledge [2] describes the pluralistic walkthrough as, “A systematic group evaluation of a design in which usability practitioners serving as walkthrough administrators guide users through tasks simulated on hard-copy panels and facilitate feedback about those tasks while developers and other members of the product team address concerns or questions about the interface.”
Why Didn’t the Pluralistic Walkthrough Catch On?
UX professionals didn’t adopt this technique because:
- Other than hard-core usability professionals, few have even heard of this technique.
- Although this technique does involve users, the product team does not observe them in a realistic situation. Walking users through paper-based designs with a group of developers and other product team members in a conference room hardly presents a realistic approximation of how people would normally use a user interface.
- A pluralistic walkthrough involves much more work than a cognitive walkthrough or heuristic evaluation. If you’re going to go through the trouble of recruiting and scheduling all participants, why not just conduct usability testing or field studies?
Has-beens
These user research methods achieved some fame and were in wide use in the early days of user research:
- focus groups and surveys
- heuristic evaluation
However, other more useful and valid techniques have since overshadowed them. Yes, UX professionals still employ these methods from time to time, but their reputation and popularity have greatly diminished.
Focus Groups and Surveys
Although UX professionals still occasionally conduct focus groups and surveys, they widely regard them as less useful and valid techniques for user research. Some clients still insist on conducting focus groups, and there are times when surveys are useful for gathering supplemental information, but very few people depend on either approach as their primary research method.
How Did Focus Groups and Surveys Become Has-beens?
The popularity of focus groups and surveys has diminished because:
- Focus groups and surveys focus on what people say, not what they do. User research methods that observe user behavior in context are much more valuable. Focus groups and surveys take users out of their natural context and, at best, ask them to describe their behavior, which isn’t very accurate. These methods are best for gathering opinions and perceptions, which are helpful, but not enough to understand the user experience.
- Because of these limitations, UX professionals now frown upon focus groups and surveys as user research techniques—or at most, use them only to supplement other methods.
Heuristic Evaluation
“Hey, wait a minute,” you might be saying, “A lot of people still use heuristic evaluation!” Well, yes, they do something similar to heuristic evaluation, but very few use the traditional method of heuristic evaluation that Jakob Nielsen originally described in his book Usability Inspection Methods. These days, most people instead employ an evolution of the heuristic evaluation that is actually an expert review. (I’ll discuss the expert review later, as a technique that did catch on.)
Jakob Nielsen described his method of heuristic evaluation as: “Having a small set of evaluators examine the interface and judge its compliance with recognized usability principles (the ‘heuristics’).” [3] The key to this method is that it requires several usability professionals to do their own individual evaluations of a user interface, using the list of heuristics that Nielsen described in his “10 Usability Heuristics for User Interface Design,” then get together to compare and combine their findings.
How Did Heuristic Evaluation Become a Has-Been?
Why did the traditional method of heuristic evaluation fade away, getting replaced by what is essentially an expert review?
- The traditional heuristic evaluation requires from three to five usability professionals. In the real world, it’s an extremely rare luxury for a UX team to have so many usability professionals who can work on the same project. In reality, a single usability professional incorrectly conducts most so-called heuristic evaluations—making them, in actuality, expert reviews.
- Although Nielsen intended his heuristics to be general usability principles that could apply to many types of problems, some UX professionals began to feel that they were too abstract and didn’t cover everything that it was important to evaluate. So they either developed new heuristics, guidelines, or checklists—or simply began to rely on their own expertise.
- As a method that doesn’t involve users, some UX professionals frown upon heuristic evaluation and question its value.
- The effectiveness of heuristic evaluation has become somewhat discredited. Studies have shown that heuristic evaluations often miss the majority of usability problems—especially when they don’t follow Nielsen’s traditional method.
Need a Better Agent
User research methods in this category still find some work, but are in danger of becoming has-beens, so need stronger advocates for their use. In other words, they need a better agent. One example of such a user research method is the use of scenarios.
Scenarios
Few researchers or UX designers today take the time to create scenarios. It would be very controversial to call scenarios a has-been because they still have some strong proponents. But I think most would agree that scenarios need a better agent to encourage their wider use.
Scenarios are stories about how people would perform tasks in a planned system. Since they are based on user-research findings, they act as a bridge between research and design, describing what a design needs to provide.
Scenarios came into use as a user-centered replacement for use cases, which provide stark descriptions of user actions and system reactions. Use cases do not consider user characteristics or needs at all. Scenarios provide a way to use the information that we gain about users through research to explore possible design solutions.
However, over time, personas and other more visual depictions of user experiences such as workflow diagrams, storyboards, and customer journey maps have supplanted scenarios. Some people still write scenarios for their personas or have incorporated scenario-like elements into their personas, so scenarios haven’t completely disappeared. But they aren’t in as wide use as they once were.
Why Haven’t Scenarios Become More Ubiquitous?
There are several reasons that scenarios are no longer in wide use:
- Scenarios sit at an awkward juncture between user research and UX design. When different people are doing user research and design, scenarios can seem like the responsibility of the other guy, so tend to get neglected in favor of other techniques.
- For many UX designers, writing scenarios in a Word document doesn’t feel like design. Designers are often more comfortable beginning with something more visual—such as workflow diagrams, storyboards, screen-flow diagrams, sitemaps, or just sketches of design ideas.
- For UX designers who don’t have strong writing skills, it can be very time consuming to write scenarios for the many different tasks that a system involves. The time it takes to explore scenarios might not always feel productive, so it can be difficult to justify the effort it takes to clients and project team members.
- Prose-based descriptions of user experiences can be difficult for some UX designers and project team members to digest and leverage. Often, visual depictions of user experiences such as storyboards, flow diagrams, and customer journey maps are easier for clients and project team members to quickly understand.
