Testing Products People Will Use in a Variety of Environments

“While, theoretically, it would be great if we could say, ‘Let’s test the product in all environments,’ it isn’t always practical or really necessary to do so,” replies Cory. “In fact, the main point of testing a prototype is often to identify sticky points that need fixing—and those sticky points are likely to be just as visible in a formal usability lab, an ad-hoc testing environment such as a conference room, or remote testing via a shared screen.

“However, if users will perform different activities that are specific to particular environments, this may dictate your testing different sets of tasks with different audiences or each participant getting a finite set of test scenarios. Even so, if the impact of specific kinds of distractions in different environments really is critical to understanding how people will use a product, you can either simulate those distractions or actually conduct testing in those specific environments. Of course, doing usability testing in different environments will likely result in higher testing costs, managing more complicated testing logistics, and increased labor hours.”

“Certainly, you should test the overall usability of a product. But you should also determine whether any environmental conditions would negatively impact the product’s usability,” advises Jim. “However, this doesn’t mean you would have to test in every conceivable environment. Instead, consider what environmental conditions could actually affect the use of the product.

For example, some environmental impacts on usability include bright or dark lighting conditions; a crowded, noisy environment; distractions; or shaky conditions inside a vehicle. Also, consider how often people would use the product in such conditions. Then, conduct usability testing in environments where one or more of those conditions is present. To evaluate a product’s usability without considering interference from environmental conditions, you also conduct usability testing in a more stable environment such as a home or office—if people will often use the product in those contexts.”

“In a perfect world, I would suggest performing research in as many of those different environments as possible,” recommends Ben. “But I have yet to work in a perfect world, so I suggest that you identify a handful of environments—maybe three to five—that are at different ends of the spectrum and perform research in those contexts. If the product works well in extreme environments, hopefully, it would also work well in the contexts between those extremes. However, I would continually test the product in as many different environments as possible.”

Simulating a Variety of Environments

“If various environments are likely to affect the usability of the product, you need to get out into those environments or, at least, simulate them in the lab, so you’ll know whether the product design will be effective in likely environments,” answers Carol. “For instance, if you need to know the impact of environmental noise and distractions, you can go to a public space where there are background noise and plenty of distractions.

A common location for such in-the-field testing is a coffee shop or shopping mall. However, if you can’t actually leave the building to test in a noisy environment, but work in an office building, try the lobby of your own building. Or, if you don’t have access to a noisy lobby, pipe noise into your testing facility. A case in point: In an upcoming study, we’ll be testing a set of instructions for both experienced and lay users, where their inability to follow the instructions could have potential health risks. To provide a distraction to see how well the instructions work for users in a noisy environment, we’ll have a TV playing near the participants.”

“When creating and testing a product, you’ll want to create a situation that is as close to realistic usage as possible,” responds Mark. “For example, if you were designing software for a fast-food chain, you’d want people to be able to use an app in a restaurant, on their phone, and on the go. It’s important to make sure you’re testing in a setting that’s as close to the real thing as possible.”

“Remote testing offers the benefit of allowing participants to use the product in their natural environment,” suggests Jordan. “That said, I’ve seen a lot of value in quick-and-dirty testing with colleagues and other people to whom you may have access. Some people call this coffee-shop usability testing, which is just one of many terms that refer to quick-and-dirty usability testing. While you should do testing with actual users in their own environment at some point, quick-and-dirty testing can reveal whether, in general, human beings are able to use—and find value in—a particular design solution.”

The Higher Cost of Testing in Multiple Environments

“While you’ve not specified what you mean by environments, I’m going to assume you mean physical environments rather than platforms such as mobile or desktop,” replies Peter. “The challenge you’re likely to face when you test in many environments is the higher cost of testing. Testing takes time and money, and the more environments in which you need to test, the more expensive it gets.

“Triage your environments: are they all equally important or can you prioritize one over another? Then, look at your options for testing. Do you need to create a high-fidelity prototype or can you go low fidelity all the way? I mean seriously low fidelity—for example, a walkthrough of a paper prototype might be sufficient. I’ve found that getting a solution in front of users—even a really basic solution—can help them to better understand what they want. In this regard, very basic prototypes have an edge over more sophisticated prototypes.

“Research can be expensive, but the general rule is that the further along in the design lifecycle you do testing, the more expensive testing and resolving the issues you discover during testing will be. So test early.

“Whatever approach you use, keep reflecting on what you’re doing. The more deeply you think about the problem you’re trying to solve—reflecting not just on the data, but on your research approach—the more value you’ll get from investing even limited resources in testing.”

A Two-Phased Approach

Once you’ve done your best to design a solid prototype—whether a complex, digital prototype or a simple paper prototype—it’s time to test it. First, make sure you catch all the low-hanging fruit by doing some quick usability testing, in any environment in which real users would actually use the final product. Many design flaws become very apparent as soon as real users get their hands on a product. Regardless of the environment in which particular users will actually use the product, you must fix these fundamental problems.

Subsequently, test your prototype in the most important environments. Which ones are those? To decide, use the 80/20 rule: “80% of our profits come from 20% of our users.” Do everything you can to make those users happy! Once you accomplish this, you might find that those varied environments are not quite so different as they seemed at first. For example, two environments may both be extremely noisy or have lots of distracting physical motion. So you might, in fact, be able to cover more of those environments than you originally thought.