With the current COVID-19 pandemic, all UX researchers have had to rely exclusively on conducting remote UX research. In Part 1 of this column, I discussed the many advantages of remote UX research. Although remote UX research does have many advantages, it also has some disadvantages. Now, in Part 2, I’ll consider the disadvantages of remote UX research and how to overcome or mitigate them.
The Disadvantages of Remote UX Research
What are the disadvantages of remote UX research and how can we minimize them as much as possible to get the best results?
You Cannot See Much of the Participants’ Context
The biggest disadvantage of remote UX research is also the greatest advantage of in-person UX research—being able to observe participants performing their tasks in the context of their natural environment. If you’re interested in understanding a particular group of people, their behavior, their typical tasks and processes, the tools and artifacts they use, the people with whom they interact, and the environment in which they’re performing their tasks, there’s no better way of understanding all of this than to visit participants in person so you can observe and interview them. When you’re not in the same room with participants, your view is limited to what you can see through their Webcam and what they share on their computer, phone, or tablet.
Champion Advertisement
Continue Reading…
Overcoming This Disadvantage
Although it’s very difficult to observe tasks remotely that normally take place offline, you can conduct remote contextual inquiries to understand participants’ tasks and processes that take place primarily or fully on computers, phones, or tablets. For example, if you want to learn about how consumers research and shop for cars online, you could easily do that by conducting remote contextual inquiries. Participants can share their screen with you and demonstrate their research processes, using various sites and apps, while thinking aloud.
During remote sessions, you can see a little bit of participants’ environment, and you can ask them to move the Webcam to give you a better view of their environment. That still doesn’t allow you to see very much, but you can see more of their actual environment than you could see if you brought them into a research facility.
For the parts of a process that take place off screen—such as using physical documents, artifacts, and tools—ask participants to talk about those items and perhaps hold them up to the Webcam. Obviously, this isn’t as good as seeing those parts of the process in person, but if they make up only a minimal part of the process, remote contextual inquiries can be surprisingly effective.
You’ll Miss Some Facial Expressions and Body Language
You can see the facial expressions and body language of participants much better when you’re in the same room with them than you can when seeing only their face through a Webcam. Plus, you’ll tend to notice more when maintaining eye contact during an in-person session. You’ll usually pay much less attention to participants’ Webcam video during remote sessions and focus much more on the screen they’re sharing.
Overcoming This Disadvantage
Although facial expressions and body language provide some insight into what participants are thinking and feeling, I think their importance is a little overblown. During remote research, I find that I focus more on what the participant is doing and saying. So I feel that I’m better able to pick up on hesitations and the tone of the participant’s voice than I would during in-person research. In fact, you can often see participants’ faces more clearly during remote research, because they face toward the Webcam. During in-person research, you must often sit beside a participant as you both look at their interactions on the screen. This lets you see only part of one side of a participant’s face.
You Have Less Control Over the Technology Participants Use
For remote UX research, participants connect to an online session from their own device rather than your having them use the same computer, phone, or tablet that they would in a lab. As a result, they may be using large or small monitors, a PC or a Mac, Chrome or Safari, iOS or Android, or an older or newer smartphone. All of that variety is great if you want to get a realistic view of the technology they use, but you’ll have less control than you might want over how the participants experience a prototype during usability testing. Some participants may have very fast Internet connections, while others have very slow connections that cause their audio and video to cut out during their session. While you might have asked all participants to have a Webcam, some of them might have older cameras with very poor video quality.
Overcoming This Disadvantage
If it’s important for all participants to use a particular type of computer, phone, or tablet, include those specifications in your recruiting criteria. If you want to have more control over the display of a prototype, open the prototype on your computer, share your screen, and give the participant control over your mouse so they can click through the prototype. Most major online-meeting tools provide such a remote-control feature. Unfortunately, none of these tools currently has the ability to give someone remote control of your mobile device. This works only on computers.
Include a question in the screener about the speed of a participant’s Internet connection, and eliminate those whose speeds are too slow to participate effectively. If the quality of their Webcam is important, include a question about the model or age of their Webcam. However, if seeing participants’ faces isn’t essential, you can consider recruiting people with slower Internet speeds or older Webcams. If their connection is too slow and causes problems, they can always turn off their Webcam to save bandwidth.
You Have Less Control Over Interruptions
Whenever your participants are in their natural environment—whether the session is remote or in person—you’ll see the same types of interruptions they would typically experience. Especially with so many people working from home during the COVID-19 pandemic, family members, pets, phone calls, texts, or deliveries might interrupt participants. While this can give you a realistic sense of the interruptions they would typically encounter, it can also be distracting and disrupt the session.
Overcoming This Disadvantage
When recruiting participants, ask them whether they can participate in their session from a quiet location, without interruptions. Emphasize that they need to be able to devote their full time and attention to the session. However, during their session, be understanding and cut them some slack, if necessary. Occasional interruptions are bound to happen, for both you and the participants. As long as participants can quickly end interruptions and continue with their session, it shouldn’t be a big deal. On the other hand, if participants must leave their session for a long period of time to deal with an interruption, that is a problem.
It’s Harder to Keep Prototypes Secure
In a usability lab, you can have complete control over access to your prototypes. Whether they’re installed on a computer or accessible through a link, the participants have no way to access them after their session.
However, in remote sessions, you have much less control over the security of your prototypes. If you send participants a link to a prototype, they could note its URL or check their browser history to return to the prototype after their session. While you could password protect the prototype, you must tell participants the password to enter, which they could note and use later. During their session, participants might be taking screenshots or recording their screen. Yes, all of this sounds highly paranoid, but some companies are very sensitive about the security of their designs.
Overcoming This Disadvantage
There are quite a few things you can do to help keep your prototypes secure. Let’s look at some of them.
Get Participants to Sign a Nondisclosure Agreement
If you’re especially concerned about security, have the participants sign a nondisclosure agreement, once you’ve recruited them, stating that they will not talk about or share what they see during their session. Include language about not taking screenshots or recording their session. During your introduction to the session, you can also politely remind them: “As you read in the NDA, please don’t share or talk with anyone about anything you see during your session today.” I usually include that text right after I’ve told them: “Everything we talk about will remain anonymous, and none of your personal information will be included in the report.” This gives participants a sense that we’re both respecting each other’s privacy.
Ask Participants to Share Their Entire Screen
When participants share their screen during the online meeting, ask them to share their entire screen, not just their Web browser. This makes it easier to see whether they’re trying to take screenshots of the prototype or record their screen. If you notice that they are taking screenshots or using recording software, politely ask them to stop.
Use an Unbranded, Generic Prototype
If doing so fits your research goals, you could also make your study anonymous—not telling participants what company is conducting the study. Create an unbranded, generic-looking prototype so participants won’t know what company the designs are for.
Let Participants Use the Prototype on Your Screen
The safest way to get a prototype to participants during a remote session is to display it on your computer, share your screen, and give the participant remote control of your mouse. Maximize the browser screen to hide the address bar, so participants can’t see the URL. I use a Chrome browser extension called Open-as-Popup that maximizes the browser window and removes the address bar.
Share a Mobile Prototype from Your Desktop
Unfortunately, as I mentioned earlier, the remote-control feature in most online-meeting software works only with computers, not mobile devices. You can share your phone or tablet’s screen during an online meeting, but you can’t give a participant remote control of your device.
Instead, share a mobile prototype on your desktop computer and give the participant remote control. The disadvantage of this approach is that participants must interact with the mobile prototype using a mouse, which doesn’t support the same interactions as tapping or swiping a screen with a finger. Most participants also find scrolling confusing at first because they’re scrolling with a mouse on a computer, which is very different from how scrolling works on mobile devices. You can mitigate this difficulty by first allowing participants to practice scrolling on a practice screen that’s not part of the prototype you’re testing. That’s better than having them fumble around and practice scrolling on the first screen of your prototype.
Use Prototyping Tools That Let You Share Prototypes on the Web
If you want to allow participants to use a prototype on their own computer, phone, or tablet, the easiest way to give them access to the prototype is to send or read them a URL. Most prototyping tools such as InVision let you share prototypes via links and provide the option of password protecting them. For the greatest security, avoid prototyping tools that share prototypes as files that participants must download to their devices. If you’re concerned about security, you probably don’t want participants to have your prototype on their device permanently.
Make It Difficult to Access the Prototype After the Session
To make it harder for participants to go back to the prototype after their session, ask them to open their Web browser in Private or Incognito mode, so it won’t save the prototype’s URL in the browser history. Then read a shortened URL to them to type into their browser’s address bar rather than sending them a link that they could save to use later. If your prototype has a password, read it to participants so they can enter it. At the end of their session, ask them to close their browser window while you’re watching, to ensure that they can no longer access the prototype.
Change the Password
Change the prototype’s password at the end of each day of sessions—or for maximum security, change the password after each session. Link shortening services such as Short.io let you create and password protect a customized URL for each participant. You can also set an expiration date and time for each URL, so participants’ prototype link expires a few minutes after the end of their session. Once you’ve completed all of the sessions, deactivate the prototype by taking it offline.
You Can’t Use Paper Prototypes
When conducting remote UX research, you can’t use paper prototypes, as you could for in-person sessions. You can use only prototypes or designs that you can share electronically. Paper prototypes have two major advantages: you can create them quickly and easily and their informal, unfinished look encourages participants to be critical of the designs.
Overcoming This Disadvantage
These days, many prototyping tools let you quickly create low-fidelity prototypes and easily add simple interactions. So you can create an electronic, low-fidelity prototype in about the same time it would take to create a paper prototype. In fact, ask yourself when was the last time you conducted usability testing with a paper prototype anyway? For most of us, it’s not very common these days.
However, if you really like creating paper prototypes, you can still draw paper-prototype screens and scan or take photos of them to get them into an electronic form, then use a prototyping tool to connect the screen images with hotspot links. Participants can then click through your hybrid, paper-electronic prototype.
Some Participants Have Difficulty Using Online-Meeting Software
In UX research, we want to observe problems that participants experience when using a design solution, but we really don’twant to watch them struggle to use the online-meeting software. Occasionally, participants have difficulty connecting to their meeting, getting their Webcam and microphone turned on and working, sharing their screen, and taking control of your screen. Such difficulties can take up valuable session time. Participants with lower technical skills are more likely to encounter these problems, but you still might not want to eliminate them from your research.
Overcoming This Disadvantage
Use online-meeting software that’s easy to use and familiar to most people. The usability of online-meeting software has greatly improved over the last ten years. Zoom, for instance, is fairly easy to use and has exploded in popularity during the COVID-19 pandemic. With so many people using it in their personal lives, there’s a good possibility that your participants would already be familiar with it. If you’re doing remote research with employees of a particular company, use the same online-meeting software that company uses, so they’re already familiar with it.
Ask participants to install the online-meeting software, test it out, and get familiar with the way it works in the days before their session. Have your recruiting company schedule each participant to join their online meeting 15 minutes before the start of their session. Someone from the recruiting company can also join sessions early to help participants connect and ensure that they can share their camera and microphone.
Observers Might Miss In-Person Collaboration
Even though it’s easier for clients and project-team members to observe your research sessions remotely, some people miss the discussions and debriefs they would have if everyone were together in an observation room. Some people feel they get a better dynamic and have better discussions with everyone together in the same room than they do when observing remotely. They enjoy doing group analysis and problem solving, using Post-it notes and writing on a whiteboard.
Also, when everyone gets together in an observation room, they tend to focus more attention on all of the sessions. When everyone observes remotely, it’s easier to skip some of the sessions or get distracted by multitasking.
Overcoming This Disadvantage
You can try to replicate the observation-room dynamic by using collaborative, whiteboarding tools such as Miro or Mural. This helps keep observers more actively engaged and paying attention to the sessions. Schedule debrief time between the sessions for observers to discuss your findings and collaborate on analysis. Tools such as Miro and Mural also let observers draw, create diagrams, and type up their insights on virtual Post-it notes that they can move around the screen and group into categories. For example, observers could type up their notes on Post-it notes during the research sessions, then sort their notes into an affinity diagram during the debrief.
The Advantages Outweigh the Disadvantages
Although remote UX research does have some disadvantages, the advantages of remote research greatly outweigh them. By following the advice I’ve provided in this column, you can overcome most of these disadvantages. That’s not to say that we shouldn’t go back to conducting in-person sessions when it’s finally safe to do so. It really will be nice to get to a time when it’s again safe enough to decide whether we want to conduct sessions remotely or in person.
Fortunately, one side benefit of COVID-19’s forcing us to conduct all UX research remotely is that we’ve had to use our creativity and ingenuity to overcome all of these disadvantages. This has led to improved remote UX research methods and technologies that we can continue to use long after the pandemic is over.
Jim has spent most of the 21st Century researching and designing intuitive and satisfying user experiences. As a UX consultant, he has worked on Web sites, mobile apps, intranets, Web applications, software, and business applications for financial, pharmaceutical, medical, entertainment, retail, technology, and government clients. He has a Masters of Science degree in Human-Computer Interaction from DePaul University. Read More
