In this Ask UXmatters column—which is the second in a three-part series of columns focusing on usability—our experts discuss how to conduct usability testing with limited funding. To read Part I of this series, see “Usability Testing Versus Expert Reviews.” Next month’s column will cover what usability techniques you should use when time is tight and how to best conduct remote usability testing.
For answers to your questions about user experience matters, Ask UXmatters! To see our experts’ responses to your question in an upcoming edition of Ask UXmatters, please send your question to: [email protected].
Q: Given a shortage of funds, what types of usability testing should we perform? Can you recommend any low-cost usability approaches?—from a UXmatters reader.
Champion Advertisement
Continue Reading…
The following experts have contributed answers to this question:
Todd Follansbee—User Experience Architect and Lead Consultant at Web Marketing Resources
Mike Hughes—User Assistance Architect at IBM Internet Security Systems; UXmatters columnist
Whitney Quesenbery—Principal Consultant at Whitney Interactive Design; Past-President, Usability Professionals’ Association (UPA); Fellow, Society for Technical Communications (STC); and UXmatters columnist
Jim Ross—Principal of Design Research at Electronic Ink
Janet Six—Principal at Lone Star Interaction Design; UXmatters Managing Editor and columnist
Kyle Soucy—Founding Principal at Usable Interface
Josephine Wong—Principal Usability Consultant at Apogee Usability Asia
Before Doing Usability Testing
When you are short of funds, I encourage you to first make sure you catch all of the obvious design flaws like inconsistent labels and lack of feedback before doing any usability testing. I like to use Ben Shneiderman’s “Eight Golden Rules of Interface Design,” from Designing the User Interface, to catch the problems designers and developers easily miss. If you create a user interface that follows these guidelines, you’ll be well on the way to having a very easy-to-use interface. I say this, because at present, many software companies still do not follow good design principles. Many software companies know how to make something cute, but few know how to make something both usable and useful.
You can catch these types of errors with an expert review. Either hire a consultant or work with a member of your staff who is a certified usability professional. It is vital that the reviewer not be on either a project’s design or development team, because it is only human to see what we expect to see in a design, instead of what is actually in the design.
Whitney says, “First, see my response to the question on the use of expert reviews versus usability testing in Part I of this series, ‘Usability Testing Versus Expert Reviews’.”
I suggest you make a list of your product’s top three goals, its top three measures of success, and the top three personas it should please. You may actually have many customers—the evaluator, the buyer, and the user, which may or may not be the same person. Make sure you reach your key goals and measures of success and please those key people. You might be surprised by how many companies do not design success into their products. Prioritizing your goals can have a huge impact on your products’ usability. Because of time and financial constraints, you might not be able to create the perfect product, but you can still create a very, very good product that does what people need it to do most of the time.
Doing Cognitive Walkthroughs
“If you cannot afford to do a usability test, the next best thing is a cognitive walkthrough,” suggests Mike. “In that method, you write out test scenarios—just as you would with a usability test—but instead of bringing real users into a lab, you bring designers and stakeholders into a conference room, project the user interface, and collectively walk through the scenarios, discussing what actions you think users would take. This technique can be surprisingly effective, because it forces the team to look at the user interface through use-case-like scenarios, and the collective perceptions of the team catch many usability issues. It works best if the developers listen to how the other team members think the user interface would work. In other words, avoid running a cognitive walkthrough like a demo; run it more like a usability test.”
Enabling Non-Experts to Evaluate Usability
Todd felt the lack of tool that would let both experts and non-experts evaluate usability, so he’s designed and built his own, called The Dudley Tool. He named it after an MIT engineer “who had a great ability to make complex tasks appear simpler.” Todd describes The Dudley Tool, as follows:
“We have just made available—after many months in beta—a low-cost ($35.00) usability compliance tool, which we designed for amateurs and professionals to use in improving Web-site success. Users review Web sites and select either Pass, Fail, or Not applicable for each of over 70 guidelines. We describe each guideline in layman’s terms. Each guideline is weighted for impact on the bottom line and ease of compliance. Thus, the tool can also help you devise a strategic plan to improve a site.
“The tool describes the benefits of compliance and, where appropriate, offers examples and references for further reading. Of course, it cannot make specific recommendations about a site, but can surely help improve compliance across a broad range of guidelines. The tool looks at and grades persuasion, usability, navigation, graphic design, and technical issues. Our intention is that it be a first, but significant step that small business sites can take to immediately understand what needs to improve on their sites. It will not make someone a usability expert and results are completely dependent upon how one complies with the guidelines, so a company may need some expert advice along the way. This tool is not and will never be a substitute for an expert review or direct usability testing.”
The Basics of Usability Testing on Any Budget
It’s best if a usability professional is not a member of your design or development team. If you do nothing more, be sure to quietly observe a person trying to accomplish the most important tasks using your product. Of course, it is best to test with your target or representative users. Do what your budget allows. Certainly, do not skip usability testing altogether, instead letting your customers do the testing with your first software release. Repairing the damage done by a bad first version could be much more expensive than doing usability testing in the first place.
Making Usability Testing Fit Your Budget
“Stop thinking about usability testing as something that’s hard or expensive,” declares Whitney. “Like anything else, it’s on a scale. At the low end is Steve Krug’s Hey You test. Stop someone in the hall and say Hey you. Would you take a few minutes to look at something for me? Then, ask them to try using it. Not just look at it. Not opine on it. But interact with it. To have an experience. At the high end is a carefully constructed quantitative project with 100s of users and a carefully controlled research environment. Most usability testing falls somewhere in between. Now that you are assuming you will do a little bit of testing, your next questions are What do I want to learn? and What is the least work I can do to learn that?”
“Given a shortage of funds, you can do informal usability testing—using low-fidelity prototypes like PowerPoint presentations, wireframes, or sketches— anywhere and anytime,” encourages Jo. “As long as you are armed with an understanding of what you want to find out; a task; the purpose of a screen, function, or journey; what you want to do with the results; how you want to communicate and to whom, then you are good to go. The focus is not on the method itself, but rather on what you want to find out and on the opportunity to iterate the product toward success.”
“For the cheapest usability testing, skip all the extras—the lab and the fancy equipment—grab a pen and a pad of paper and go to the participants’ locations—their homes or workplaces,” responds Jim. “At the most basic level, usability testing is just a usability professional observing a participant at a computer and taking notes with pen and paper. All the other technologies are just helpful extras.”
“Don’t use labs,” agrees Kyle. “You can gather the same great feedback by just using a quiet conference room.”
“We have conducted testing in conference rooms, coffee shops, and more formal setups with a one-way mirror,” says Jo. “We now have a definite leaning toward more natural settings, in a place that makes a user feel at ease and where the conversation flows easily.”
“Your results will be more realistic when skipping the lab and observing participants in their own locations, using their own computers,” explains Jim. “Plus, you may be able to give them less expensive incentives, because they are spending less time and money when they are not coming to you.
“It is nice to have some kind of recording to review and provide examples to show others. If you can afford TechSmith’s Morae recording software and a Webcam, you can record sessions on a laptop, but that is not essential. If you cannot afford Morae, TechSmith’s Camtasia is cheaper screen and audio recording software. A digital audio recorder is another relatively inexpensive option for capturing participants’ comments.”
“Don’t write reports,” suggests Kyle. “Conduct an in-depth debrief with your observers and go over the observations and implications from the testing, then put together a one-hour presentation of the key findings. Don't create highlight videos either. It takes a lot of time to go back over hours and hours of testing footage, and it’s unnecessary if you have the stakeholders observing the tests firsthand.”
Finding Participants for a Usability Study
“Do you need strict recruiting?” asks Whitney. “Well, it’s important for the people you test with to have some resemblance to the people who will actually use your product. But the most important thing is that you think about how they are different and account for that in your decisions. Let’s say you are working on a clinical management system for a large hospital group. Do your usability test participants have to work there? Or would health professionals with appropriate experience give you most of what you’re looking for?
“Do you need 12, 24, or more participants? Why not start with 5 or 6, learn from them, then try again? This way, you are improving the design with each group, and you can easily stop if you start to observe the same thing over and over.”
“Running a test with 4 to 6 users usually reveals a number of ways to improve a product,” agrees Jo. ”Save your budget to assess how you want to test a new group of 4 to 6 people. You may need to reassess what your overall testing objectives are. Depending on your recruitment profile, if you cannot afford to hire a recruitment company, you can use your own network of family, friends, and Facebook—the 3 Fs—or other people in your company whose have similar profiles.”
“You can save a lot of time and money on recruiting by using existing customer lists or, if you are testing a company’s internal software, recruiting employees,” adds Jim. “If you are doing usability testing with your client’s or your company’s employees, you can often find volunteers for either a minimal or no incentive. The prospect of helping to improve a frequently used and problematic user interface is often incentive enough.”
Doing Remote Usability Testing
“Now more than ever before, there are a lot of options available for conducting low-cost usability testing,” suggests Kyle. “Unmoderated remote usability testing tools such as Loop 11 and Open Hallway are a great option when you need quick feedback on a user interface, but you don’t have the budget or time to conduct in-person research. The tools from Optimal Workshop—Chalkmark, Treejack, and Optimalsort—are also great for quickly gathering low-cost feedback on your information architectures and interface designs.”
“Can you use remote moderated testing?” asks Whitney. “Sure. Unless your prototype is only on paper. That gets a little harder to do remotely. GoToMeeting is all you need. You don’t even need a phone line, because it now includes a voice channel. I’ve used remote testing to enable me to include busy professionals—for example healthcare professionals—around the country or around the world, when the limiting factor was the cost of getting me and the participants in the same room, at the same time.”
Next month, we’ll continue our discussion of remote usability testing in the third part of this series.
Gaffney, Gerry. “Design Implications.”Apogee, April 6, 2009. Retrieved November 4, 2009.
Rubin, Jeffrey, and Dana Chisnell. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. 2nd Edition. Indianapolis: Wiley, 2008.
Shneiderman, Ben, and Catherine Plaisant. Designing the User Interface: Strategies for Effective Human-Computer Interaction. 5th Edition. Reading, MA: Addison-Wesley, 2009.
Dr. Janet M. Six helps companies design easier-to-use products within their financial, time, and technical constraints. For her research in information visualization, Janet was awarded the University of Texas at Dallas Jonsson School of Engineering Computer Science Dissertation of the Year Award. She was also awarded the prestigious IEEE Dallas Section 2003 Outstanding Young Engineer Award. Her work has appeared in the Journal of Graph Algorithms and Applications and the Kluwer International Series in Engineering and Computer Science. The proceedings of conferences on Graph Drawing, Information Visualization, and Algorithm Engineering and Experiments have also included the results of her research. Read More
