Remote Usability Testing: Thinking Outside the Lab

April 21, 2014

In the world of user experience, usability testing may be one of the hardest things to sell, while simultaneously being one of the most critical elements of ensuring an application’s usability. Because usability testing is relatively expensive and time consuming, executives and development leads who have never observed a test session often dismiss testing as unnecessary and devalue the data that comes out of usability studies. In many sectors, the perception is that usability testing requires a dedicated lab with one-way glass, video cameras, and expensive microphones—all to observe users who a company has flown in at great expense and trouble to perform a couple of tasks and complain about their user experience.

However, with advances in technology over the last ten years, the ability to do usability testing with remote users has increased significantly, both reducing the investment in time and money and expanding the reach of researchers. Remote usability testing lets us get access to users from across the globe, ensuring a more representative set of results. This is vital in our hyper-connected reality. And we can now accomplish this without traveling or having a dedicated lab. Remote usability testing is a welcome addition to any researcher’s arsenal—particularly as we try to fit data collection into agile sprints and work within the constraints of an increasingly lean bottom line.

Champion Advertisement

Unmoderated Remote Usability Testing

Unmoderated remote testing is one of the most popular methods of testing Web applications. Many confuse such testing with the use of Web analytics tools such as Google Analytics and Clicky, but unmoderated remote testing focuses on obtaining data about the usability of individual workflows and experiences rather than capturing broad, overarching metrics about general usage.

In unmoderated remote tests, participants complete predetermined tasks using an application. Services like Loop11 or UserTesting let us collect data about how long it took participants to complete a task, where they made mistakes or referred to the Help system, whether they were able to complete the task, and how they navigated the application to complete the task. Researchers can analyze the results and provide specific feedback on the product design.

Pros

Unmoderated remote testing lets you recruit and test using large numbers of participants in a relatively short timeframe. Because remote testing does not rely on a moderator’s conducting the test sessions, participants can work through your test tasks simultaneously. Large amounts of data can be very powerful when you’re making a case for a significant and potentially costly change to existing software. If you’re working in a Lean UX environment, unmoderated testing gives usability a foot in the door, providing immediate value at a low cost. An unmoderated remote–testing service generally handles participant recruitment—although you can take this on yourself—and scheduling is unnecessary. In addition, you can easily test competitors’ products, creating an opportunity to present side-by-side results. You can easily fit unmoderated testing into an agile sprint schedule and provide timely feedback for the next sprint.

Cons

Unmoderated remote tests require a fully functional product or prototype, so you cannot use this approach when testing concepts through a cognitive walkthrough. Unmoderated testing often leaves the researcher to wonder why? With no means of obtaining participants’ feedback aside from comment fields or a questionnaire that you administer at the end of a test session, it can be difficult to accurately interpret the reason why a participant has made a mistake or misstep.

For example, an unmoderated test might show that several participants were unable to complete a shopping cart workflow, abandoning the task at the last step before committing the purchase. Was it difficult to see how to complete the purchase? Did participants think they had submitted their order, so were finished with the task? Did a cat sit on a participant’s keyboard and cancel the transaction? Did the process take too long, so participants got bored and quit? In unmoderated tests, these questions remain unanswered. A participant’s voice is silent for the developer and executive as well. Hearing and seeing someone’s frustration and confusion during a test is invaluable. Non-researchers’ hearing such visceral feedback can make the difference between fixing a problem and merely accepting it as it is.

From an enterprise perspective, unmoderated remote testing may be inaccessible to large companies because of legal considerations, the scale and complexity of their software, or the need to test with a limited and highly specialized user base. Many large B2B (Business-to-Business) companies have significant legal exposure when testing their software with users, requiring lengthy non-disclosure agreements and assurances of confidentiality. Remote testing services work on the assumption that the applications to be tested are Web based and consumer oriented. Neither of these assumptions is valid for most large B2B companies—many of which still maintain thick-client, complex software that is geared toward an elite user base such as security administrators or systems administrators.

Is Unmoderated Remote Testing for You?

Unmoderated remote testing is generally best for assessing targeted areas of a user interface that have well-known issues, A-B testing, simple competitive analyses, collecting data in large numbers, and initial problem discovery—assuming you follow up with moderated testing to uncover the why. Unmoderated testing is not recommended for solving complex or poorly understood issues or as a substitute for discovery work such as user interviews or cognitive walkthroughs.

Moderated Remote Usability Testing

A less well-known, but very useful testing technique, moderated remote usability testing lets participants complete predetermined tasks using an application, while interacting with a moderator who prompts them for feedback during the test, using a think-aloud protocol. You can collect data about how long it took participants to complete a task, where they made mistakes or referred to the Help, whether they were able to complete the task successfully, and how they navigated the software. You can also discover why issues arose, whether different participants encountered the same problems when using the software, what other problems participants experienced, and what features they desire that would improve the product. Numerous tools are available for moderated remote testing, but setup and hosting of the software that you are testing is generally on your own network.

Pros

Moderated remote testing lets you recruit and test using participants from all over the world, without the expense of travel. Difficult to schedule users become easier to reach because they can complete the test session from their own office, eliminating the travel time to and from your lab. Because a moderator conducts the tests, you can collect significant amounts of qualitative and contextual data—often more than you’ve originally anticipated. And you can invite as many observers as your bandwidth permits, without making your participants feel uncomfortable or nervous. You can often fit moderated testing into an agile schedule—but not for every sprint.

Cons

Moderated remote testing can present significant setup challenges, especially in regard to performance. For example, if you are using a tool like WebEx to let participants control an application on your desktop, there can be a delay between their selecting an option and seeing a result. This can lead to false errors that are not representative of how a user would actually interact with the software.

The think-aloud protocol can be difficult for some participants who are perhaps more reticent by nature. The moderator must be skilled in remote testing to ensure good results. The inability to see facial expressions and body language, as well as the potential for interruptions, can make remote testing less ideal than in-person testing in a lab, where you can more closely control the environment. For example, participants could be texting during their test session, their cube mate might be having a loud discussion with coworkers, or a fire alarm could even go off.

From an enterprise perspective, moderated remote testing may also present legal issues. Because participants could potentially take screenshots, most legal departments require lengthy non-disclosure agreements and assurances of confidentiality.

Is Moderated Remote Testing for You?

Moderated remote testing is generally best for studying complex or poorly understood issues, when it is important to understand why a participant is doing something. It is also good for interviews when you’re gathering requirements and cognitive walkthroughs when you’re exploring an idea or concept with potential users before creating a fully functional prototype.

What About Mobile?

While remote usability testing on mobile devices is still in its infancy, there are some promising testing tools currently on the market. Unfortunately, most of them are still in the realm of unmoderated testing or require you to preload an application on a mobile device for participants to use, making these tools inappropriate for remote testing. How to provide your application on a mobile device, collect a participant’s environmental information, and ensure the security of both the information that you transmit to the device, as well as the information that you collect from the device are all still open issues.

Today, usability testing has come a long way from being confined to a lab. As technology continues to progress and our users’ expectations of a flawless, contextual user experience grow, we must continue to explore new ways of creatively collecting data on how our applications delight or disappoint our users.

In Remote UX Research | Usability Testing

6 Comments

Russell Wilson

April 22, 2014 2:39 PM

Great article. I would really like the author to post a follow-up piece that expands on remote usability testing on mobile.

Jonathan Hicken

April 24, 2014 4:56 PM

Thank you, Rebecca, for your article!

In response to some of your cons of remote unmoderated tests—many of your points are avoidable.

‘Ascertaining the why when users confront a problem’—A skilled researcher would be able to design a test to capture these insights. You can ask focused questions during and after the session to elicit these juicy responses. For example, clearly defining success criteria in the script will prevent false reporting of task success.

Non-disclosure agreements—Some companies will administer and collect NDAs for you, including UserTesting.

Narrow demographics—Again, UserTesting will help you recruit your exact target market.

Note from the editor: This comment was contributed by an employee of UserTesting.

Scott Nelles

May 17, 2014 9:01 PM

Nice article. Very spot on. I used to work at UserZoom, so I’m pretty familiar with Unmoderated Remote Usability Testing (URUT). Legal at big companies can be a pain for sure and more difficult with URUT. A work around for that can be removing your branding from the prototype. That helps sometimes because then no one will know the prototype is from you. Really concise and great article. For someone getting their feet wet, this is way better than reading a whole (and boring) book on the subject.

Rebecca Baker in reply to comment from Scott Nelles:

June 3, 2014 9:34 AM

Thanks, Scott! Excellent point about removing the logo and any identifying marks.

Rebecca Baker in reply to comment from Jonathan Hicken:

June 3, 2014 9:40 AM

Thanks, Jonathan!

Good points all. I definitely agree that a carefully crafted script is key in any kind of testing to avoid misinterpretation of the results. And certainly you can have a post-test interview to follow up on anticipated clarifications. One of the advantages of many unmoderated services today is the ability to quickly change the test and reissue it to try to get more insights in a timely fashion. However, there remains that slice of why that requires more of a real-time interaction with the user. Thanks for you comments!

Adam

May 17, 2019 3:57 AM

Thanks for the article! It complemented the previous one nicely. I found the section about picking the right users particularly useful. A colleague of mine recently wrote a similar piece that compared the benefits of moderated and unmoderated remote usability tests:

“Moderated and Unmoderated Remote Usability Testing: What Is It, How to Run It, and What Are the Benefits?”

Rebecca Baker

Senior Director at UX Active Network

Plano, Texas, USA

A usability disciple, quixotic scientist, excellent listener, and author of mundane fantasies, Rebecca is a professional writer and speaker with more than 20 publications and 30 speaking engagements on topics ranging from information encapsulation to remote usability testing. Since acquiring a PhD in Information Sciences, she has specialized in enterprise-level user research, usability, and remote testing techniques and has personally conducted over 100 usability tests, interviews, and walkthroughs. With 20 years of enterprise software experience, she currently leads the central User Experience team at CA Technologies and dreams of feeding electric sheep in the metaverse. Read More

Remote Usability Testing: Thinking Outside the Lab

Unmoderated Remote Usability Testing

Pros

Cons

Is Unmoderated Remote Testing for You?

Moderated Remote Usability Testing

Pros

Cons

Is Moderated Remote Testing for You?

What About Mobile?

6 Comments

Join the Discussion

Rebecca Baker

Other Articles by Rebecca Baker

Other Articles on Remote UX Research

New on UXmatters