This is a sample chapter from Steve Portigal’s new book Doorbells, Danger, and Dead Batteries: User Research War Stories. 2016 Rosenfeld Media.
Chapter 11: The Myth of Objectivity
In our culture in general, we place a high premium on the notion of objectivity. We hold high the values of fairness and neutrality. Journalists frequently face the criticism of bias. But the endeavors of law, news, science, and user research are led by humans. Unlike Vulcans, humans are not wholly led by logic. The field of behavioral economics, increasingly totemic for business people of all stripes, seeks to understand the ways in which people’s behaviors and decisions are influenced by irrational factors.
The field of science has long understood this about people, establishing the practice of blind experiments in the 1700s. In blind experiments, the subjects who received different conditions didn’t know what those conditions were. Starting in 1907 and fully established by the 1950s, the double-blind experiment goes even further. In a double-blind experiment, neither the subjects nor the experimenters know what the conditions are.
Champion Advertisement
Continue Reading…
Consider the scientist who is trying to determine the efficacy of a new drug: some subjects receive the new drug; others receive a placebo. In a blind experiment, the subjects don’t know what they’ve received. In a double-blind experiment, the subjects don’t know, but neither does the scientist. It makes intuitive sense that the subject would report different effects based on their assumptions about what they were given, but it goes farther—knowledge of the drug they received—or did not receive—could also manifest in physical signs, such as pulse, blood pressure, or reaction times. But the double-blind goes farther—to ensure that the subject doesn’t have this knowledge, the scientist herself doesn’t even know. This protocol doesn’t presume that the scientist would ever reveal details to the subject, rather it eliminates the possibility of the scientist unknowingly and unconsciously signaling crucial information through body language, tone of voice, or choice of words. It also ensures that the scientist won’t be influenced to ignore symptoms or patient complaints or emphasize one type of reading over another.
That’s how powerful bias is and how extensively the practice of science is structured to minimize it. In user research, we bring in our own subjectivity toward the outcomes, such as the expectations and aspirations of our clients and stakeholders, our emerging hypotheses, etc. And that’s just only one flavor of bias, experimental and otherwise, that we must be cautious of in user research.
In this chapter, we’re going to look more deeply at a specific challenge to our objectivity—that as humans we are not only flawed, irrational, emotional, and judging beings, but we are also individuals who are the result of our accumulated life experiences. As Trekkers will know, it’s not that the Vulcans didn’t have emotions, it was that as a society they chose to actively control them in favor of logic. Like the Vulcans, there is no denying who each of us is and the paths we’ve gone down. And like the Vulcans, there is no easy path to work through the challenges to our own values and beliefs.
You’ll read how researchers grappled with challenges to their own values, experiences, and circumstances. Gregory Cabrera spent time in an Afghan village, Raffaella Roviglioni met a farmer who had his own ideas about who she was, Marta Guy went into a chaotic household and reacted differently than her colleagues, and Priya Sohoni visited a maternity ward while pregnant.
While these researchers did not achieve an unrealistic level of objectivity, we don’t fault them. They relate here how they each encountered, acknowledged, and addressed the ways that the world outside differed from the way they might have imagined it would be. Of course, this is the inevitable consequence of stepping outside, and indeed it’s why we do this work, to find out the ways the world differs from our own assumptions. The musician Neil Young talked about how he experienced this in his own career: “’Heart of Gold’ put me in the middle of the road. Traveling there soon became a bore, so I headed for the ditch. A rougher ride but I saw more interesting people there.”
These stories surface a crucial question for user researchers: When you stride forward past the borderlands of your own world view, what will you do with what you find? What will you do when you see or hear or experience something that challenges your own personal values? This is well beyond hearing that someone doesn’t believe your product’s new feature is compelling. These challenges to your personal values are, well, very personal. A great researcher doesn’t let them bounce off, they process them and if they are very lucky and very good, they can also put that back into the work itself.
Managing Biases in Research
Dak Kopec, PhD
Research is about the process of discovery and gaining further insights about a given situation or phenomenon. To gain the most from research findings, one must first understand some of the fundamental concepts that underpin all research. One important concept is that all research can be invalidated by biases. Biases in research can occur during the formation of research goals, development of tools used to gather responses, during the gathering of information, and when communicating the results.
All research begins with an idea that a researcher would like to explore. This should come from an inquisitive desire to learn more, but sometimes our personal passions or beliefs overshadow the research. This commonly happens in one of two ways.
Confirmation bias occurs when a researcher promotes his or her beliefs in the research. This occurs when the researcher is too close to the subject matter or has an invested interest in the outcome.
Culture bias occurs when the researcher’s cultural lens is projected onto the research. This happens when the researcher stereotypes a group, or imposes his or her own beliefs and values onto another group.
Once a researcher has been able to set aside his or her own beliefs and values, and can fully understand and empathize with the subject population, the researcher can take the next step. A fundamental part of social science research is the development of a tool to gather information. How this tool is developed will determine if biases might arise. Some of the biases related to the research tool include:
Question-order bias, which is when the ordering of questions in the tool predetermines or influences the responses in subsequent questions. The idea behind this bias is that the researcher can set a tone with his or her questions, and can thus influence the subjects’ responses.
Leading questions and wording bias is exactly what it states. It occurs when a researcher uses vocal tone, helps respondents finish their sentences, or uses leading questions to promote a desired outcome.
Once a research tool has been developed, it needs to be administered to a sample pool of subjects. Just as the researcher can bias the results, so can the subjects. These biases can be minimized if you know what to look for and how to manage them. Subject-driven biases can come from:
Acquiescence bias occurs when the subject says what he or she thinks the researcher wants to hear to “get it over.”
Social-desirability bias happens when the subject answers the way he or she thinks will bring about greater acceptance from the researcher.
Habituation happens when subjects provide the same answers to questions that are worded in similar ways. This happens because respondents are unaware of subtle differences.
Sponsor bias happens when the subjects have an idea of the purpose, or the sponsor, of the research and wish to promote a particular agenda.
The halo effect happens when the researcher or respondent reacts to a physical trait or aspect that influences the responses or interpretation of the responses. The halo effect usually happens when the subject wants to please the researcher, but can also result from a fear evoked by a physical trait—that is, a man wearing a tie.
The final stage of research involves the interpretation of data and characterization of that data. There are two ways in which research findings can be biased:
Overgeneralization is the application of research findings beyond the actual research. An example of overgeneralization occurs when the research findings apply to Granny Smith apples, but the researcher speaks of the findings in relation to golden delicious apples. Just because they are both apples doesn’t mean the research is applicable.
Misrepresentation pertains to the depiction of data, or failing to disclose limitations. Misrepresentation of data can occur when the researchers choose to use percentiles without the use of whole numbers. Seventy percent of 10 is only seven. However, seventy percent is more compelling than seven. Misrepresentation can also come from not identifying limitations, which might include the tool only being offered to select subjects, only during select times, or only using non-probability sampling techniques—cherry-picking the subjects.
The majority of biases in research are unintentional and come from a lack of experience. Biases can present with quantitative and qualitative approaches and can affect the internal and external validity of the research data. When biases occur, the reliability comes into question. However, biases can be avoided.
One way to avoid bias is to ask a third party to review the research goals and tools. If the researcher informs the third-party reviewer of his or her personal beliefs and values, the reviewer will be better equipped to know what to look for. After careful construction of the research tool, the researcher will need to pay careful attention to the subjects and their responses. The researcher must be prepared to eliminate data that he or she suspects was biased by the subject. To avoid biases in the presentation of results, the key is to disclose. If it seems like additional information should be disclosed, then it should. Good research is hard to accomplish, but careful consideration and attention can make the research solid and provide good information.
Gregory Cabrera: Culture Shock
One of the first places I visited in Afghanistan was a security checkpoint along a major route in northern Kandahar. The security was contracted to a private group of Afghans, mainly from the south and east, to provide route security and protect military and civilian supply routes. Their job was to protect the route against insurgents who wanted to disrupt the convoy and see oil tankers burn.
A few days before insurgents did exactly this. They stopped a convoy carrying military supplies by using an improvised explosive device (IED), hitting the first truck and killing the driver. Then they attacked the last truck and shot a rocket-propelled grenade, which effectively exploded and hit the side of a fuel truck. Civilians fled, the insurgents attacked the checkpoint, and it was utter chaos. These security guards returned fire and called the local police for reinforcements. All that was left at the end were a few burned trucks, dead bodies, and some burned firearms.
Upon arrival, I could see where these men were being shot at, how they fought back, and where they stored their weapons. They worked on this mountain and lived here, too. There were approximately 15 to 20 men living in this bunker. All they carried were machine guns, assault rifles, ammunition, and blankets. Of course, they also had food, chai, cooking supplies, and utensils. As I inspected their site and positions, they told me about the event and shared their war trophies, burned AK-47s captured from insurgents. It was unusual to observe so many men living in such a tight area together, away from their villages and homes. This was security, Afghan style, and it felt like a group of armed nomads living under the radar. They were living and working together in a confined space in the middle of what felt like nowhere in particular. I would later find out that these men often worked for two to three months at a time before going home for a short period.
When we all sat down for chai, I noticed some of the people who were working at this checkpoint did not look old enough to be here. I thought to myself, Shouldn’t these kids be riding their bikes or playing in the village? The individual who was serving chai and placing candies out for our consumption did not have facial hair and had henna-painted fingers and toenails. I looked over at my interpreter and asked him on the side what these kids were doing here hanging out with security guards. My interpreter, looking down, smiled, and turned to me saying, “They have fun with them at night.”
The sergeant whom I worked with was sitting across from me. When he heard this, his face turned blank. I could tell this made him uneasy. I always wondered what the expression on my face looked like. As the young boy finished serving everyone chai, he moved near an older male who was resting comfortably on a pillow on his side. That’s weird, I thought to myself. I had just arrived in country, at this field site, surrounded by strange men who did strange things. I grabbed my cup of chai and drank it down.
Despite the weirdness of the situation, I carried on. I asked lots of questions, took lots of notes, and attempted to be as respectful of their culture as possible, even though it bothered me and made me uncomfortable. Who was I to judge? I wondered to myself, what business did we as a nation have in this country? How can its people allow human exploitation to exist like this? I learned later on that Kandahar was a different place than most of Afghanistan. It retained practices unlike the rest of the country. Although this specific instance of culture shock made me uneasy to say the least, I learned to see past it. This was an unconventional war in a strange, neglected land, and I was not there to change their culture, only study it.
Raffaella Roviglioni: Learning to Deal with Expectations
I’m currently a freelance UX designer based in Rome, Italy, but I used to be an agronomist. I like to see my professional shift not as a mutation but rather as an evolution. I understood that my passion was working with people, and now user research and UX work is fulfilling that need. Despite the different context and purpose that drove me as an agronomist, I had to interview people quite often, and I didn’t have any formal training in it. I guess I was attracted to this kind of activity because I’m an outgoing person and consider myself to be a good listener.
Back in the days when I was a research fellow at the University of Viterbo, I was involved in a pretty interesting project: investigating the old fruit tree varieties in my region. Part of the job—for me the most exciting part!—was interviewing old farmers who were between 80 and 90 years old; they were both the guardians of those old plants and the living repository of the related knowledge.
The job required me to travel to their houses and farms to perform the interviews. Given the distance and the remote location of the rural areas, the best way of getting there was by car. As a research fellow, though, I wasn’t allowed to drive the department car, since I wasn’t considered to be an actual employee, according to Italian law.
Figure 11-3—Work boots
Photo by Raffaella Roviglioni
A lab assistant—also a good friend of mine—agreed to come with me with on the field trips. He was basically acting as my driver, but helped out with taking pictures and collecting plant samples in the field.
It was during a first visit to one farmer’s house that the unexpected happened.
We arrived, got out of the car, and went over to the farmer who was waiting for us at the front door. He greeted my assistant first, and then looked at me and said to him, “So this must be your wife!” Even after an embarrassed explanation from our side, he clearly could not believe I was the one in charge of the research—with a college degree!—whereas my friend was only my assistant.
I have to confess that at first I had to rationalize a bit not to feel offended by his reaction. But after all, I told myself, he was over 80 years old, and even my grandfather would have had the same reaction in a similar situation. But the awkwardness continued because, given the context, this farmer wouldn’t expect me to conduct the interview either!
So what I did instead was direct us all—the farmer, his wife, my assistant, and me—to have coffee together, inside the house. We started chatting while drinking our coffee, as any pair of couples would do. Slowly I moved the conversation to the questions on the plants we were interested in. This allowed us to establish a more acceptable situation where the farmer felt comfortable enough to start sharing that information.
What I learned from this experience was that, in order to ensure my interview was successful, I needed to be able to deal with the expectations of others, embracing them and trying to let go of my emotions—as in this instance, avoiding feeling offended. I also realized that as an interviewer I needed to adapt to the interviewee’s setup—in this case, transforming what I view as an interview to a visit to the house—and act accordingly.
It wasn’t my first interview, nor the last, but it taught me a lot!
Marta Guy: On Confronting Judgment
I was fresh out of the Peace Corps. With an educational background in photography and videography and a unique set of personal contacts, somehow I landed a contracting gig at a design strategy firm. It turned out this was the perfect place for me. It made sense that all roads led here, although I wasn’t able to articulate why at that time. My experiences on that first project would solidify both my approach to ethnographic-style research and my interest in innovation in the business sector. The techniques and tasks associated with international development shared some remarkable similarities with those we utilize in business innovation and design strategy today.
In our Peace Corps training, we were encouraged to “do nothing” for the first six months of our service—to just sit with the host-country nationals in their day-to-day activities, and observe and ask questions. Indeed, most of my Peace Corps experience was composed of these moments of quiet observation, learning about a culture so foreign to me as to deeply challenge my own beliefs. In both international development and strategic design, this anthropological approach to learning about the people we’re designing for is the foundation of an often ambiguous process to create new concepts that will be adopted by people like those whom we’ve studied and ideally help improve their lives.
Although I had mostly casual experience doing this as a Peace Corps volunteer, an incident occurred when I was on my first professional foray into ethnographic-style user research. Our team was learning about people’s experiences using medical devices in the home. At this point, we had spoken with a couple of dozen people, including medical practitioners in their professional settings and patients in their homes. We were on our final interview of the study. My role had been to photograph and video the interviews, take notes, and generally follow the lead of my teammates who were directing the sessions. But for this final interview, my colleagues asked if I’d like to conduct the conversation, and I took them up on the opportunity to lead my first formal in-context interview.
We drove to a relatively remote location in Connecticut to see a middle-class family of two parents and three boys. Two of the boys had an immune condition that required them to pump medication for one to two hours every two weeks. The parents had decided that rather than stigmatize or devitalize the process of the boys’ drug infusion, they would celebrate it by joining together as a family for a pizza party and movies on Friday night. This celebration was in full swing when we entered their home.
It was a lively atmosphere. It turned out this wasn’t just a family of five; they lived with a menagerie of animals in their small home: cats, dogs, birds, rabbits, reptiles, and guinea pigs, all in all between 20 and 30 inhabitants. We were introduced to the guinea pigs and shown the rabbits. Everyone was supremely generous and inviting. They gave us a little tour and encouraged us to get comfortable, offering us food and drink several times. Cats snuggled up beside us, intermittently disrupting our video equipment or the conversation, while birds squawked in the background. Comfortable and confident among one another, this family moved freely and raucously around me and my two colleagues, all squished onto too few pieces of furniture for eight humans.
The parents graciously answered our questions about their children’s health and medical needs. Meanwhile, the boys played video games and watched cartoons energetically, occasionally peppering the conversation with commentary or boisterous requests for attention—“Watch this! Watch!” The parents showed us how they hooked up the medication pumps, from prepping their sons’ skin to inserting the needles. The father proudly brought out two large toolboxes full of medical supplies that they took along whenever they got in the car. He had come up with the idea of creating toolkits for the supplies they needed to be mobile. The interview continued successfully, if a bit disordered, given all the different activities happening. Not at any moment were they embarrassed or ashamed of the boys’ condition or the things they had to do to treat it. To them, this was just their life.
When we finally said goodbye, and the door shut behind us, I think all three of us researchers breathed a sigh of relief. Truthfully, it had all been quite chaotic, though we had done our best to take it in stride. But our last interview was complete, and we got into the car, heading toward New York to fly home the next day.
Driving along, one of my teammates offhandedly said, “Well, I don’t think we learned anything useful from that. That scene was a complete mess! What a waste of time.” This thoughtless comment infuriated me. Sure, the situation was intense, chaotic, and a less tidy environment than we might have desired. They had more animal friends than a small farmer might, and the lifestyle this family lived was obviously busy and disorganized. Certainly, they had some health problems, probably some difficulty making ends meet, and a shortage of square footage for all of the living things in their home. But they also clearly loved one another and were just doing the best they could to live full, healthy, and enjoyable lives. I might have been totally green and unfamiliar with conducting research for new business innovation, but I knew it wasn’t our place to judge, whether we approved of their lifestyle or not.
I was so angry. Never one to hold back, I told this teammate exactly what I thought— that these people had generously invited us into their home so that we might learn about how they lived, how they experienced their medical conditions, and how they interacted with these essential medical devices. Whether we found their lifestyle appealing or disgusting, it was valid. Their experiences were real, and we were there to learn about them. It was unfair and totally inappropriate to judge them, and it missed the entire point of what we were there to do. I said all this, I’m sure, not nearly as eloquently as I say it now, and likely with less respect than my colleague deserved, as he had more experience and knowledge on the subject than I did. He actually took it relatively well, all things considered, and we remain friends today despite the words exchanged on that trip.
I’ve found this to be one of the formative moments of my career, a moment when I expressed with passion and understanding just exactly what our purpose was there. And I’ve found similar sentiments coming to my lips again and again—with increasing grace and respect, of course—as I’ve had to remind most often clients but sometimes colleagues why we do this work. For an hour or two, we go into the home of a stranger, with a respect and appreciation for the validity of each individual’s experience. We must practice empathy, reserving judgment, allowing ourselves to stand in the other’s shoes, understand how he lives, why she does what she does, what they want to achieve and what makes that hard for them. So that in the end we might create better solutions that help them do it and make theirs and other people’s lives better and healthier. Sometimes, we just have to remind ourselves.
Priya Sohoni: Taking Empathy to a Whole New Level
I’ve never been too comfortable with hospital environments—the smells, sounds, sense of urgency—it makes me nervous. Yet, as an ethnographer should, I’ve attempted to conquer my queasiness and conduct research in medical facilities several times.
In October 2010, I was conducting research in a hospital in the San Francisco Bay area. I was almost eight months pregnant with my first child. I was given a choice between spending a day in the ICU, emergency, or the maternity department. I picked maternity—I was excited to be among so many about-to-pop mothers and so many who had just delivered. I thought to myself that for the first time I wasn’t feeling so queasy. I could hear babies in nurseries, and we shadowed some nurses as they took the babies for their first immunizations, observed visitors greeting happy families with flowers, balloons, gifts … it seemed so odd that this was a part of a hospital environment.
Figure 11-4—Kaiser maternity department
Photo by Priya Sohoni
On one of the shadowing sessions, I sat in on a nurse shift change. The nurses went around the table sharing information about the newborns and their mothers and taking careful notes of the patients’ needs and requests. On one of the nurse’s share-outs, she turned to the nursing manager and said, “Baby girl in room 203, born vaginally at 8:02 a.m., had trouble breathing, survived for 53 seconds, and then died. Should I register her as a live birth or a still birth?” I felt as if someone had stabbed me in my stomach. So much pain that I clenched my tummy, sat down on the floor, and burst into tears. I was expecting a baby girl, too, in just over a month. Why was the nurse so unemotional about a baby’s death? The nursing manager noticed me sitting in the corner, brought me a glass of water, and apologized that I had to sit through that. She suggested I take some rest in the nurses’ break room. But I wiped my tears away and stuck around.
In a few more minutes, the shift change was over, and the nurses dispersed. The nurse from 203 then walked over to another room to check in on another mother and her baby. I continued shadowing her. She entered the room with a big smile on her face, congratulated the parents, and commented on what a beautiful baby they had. She changed the baby, swaddled her, gave the mother her meds, and assured her that she could call for help whenever she felt like it. It then struck me that the nurse was concerned about her patients. Deeply concerned. She, too, had felt the pain that the family in room 203 had gone through. But she had made a commitment to hundreds of other patients, a commitment to take care of them and make them feel better. She could not have done that if she had carried the sorrow with her, out of room 203.
As ethnographers, we are trained to empathize with our respondents. To speak their language, to make them comfortable, to be one of them. I had just witnessed a remarkable new level of empathy that the nurse had. Where I had failed, she carried out each one of her roles with respect and propriety.
I went home that day with a new appreciation for the nursing profession.
Takeaways
Know yourself. Awareness of who you are and what you believe in can make these encounters less precarious. You can simply note that this person or situation is different from you. Indeed, that can be valuable data!
Feeling judgment isn’t the same as acting on it. Fieldwork is a series of choices. You may feel disturbed, surprised, offended—or thrilled, overjoyed, delighted—but you don’t have to expressly state your feelings nor—to the extent possible—default to having them guide your actions. Learn to identify that feeling when it starts creeping up on you.
Work in teams. Interviews work best with a team of two. Those two may react similarly or differently to any particular situation; in either case, it’s valuable to have each other to debrief with. By the same token, having different teams conducting interviews also means you have access to different perspectives among the different sessions.
When debriefing, talk about what you feel, not just what you see or infer. Of course, it’s good for us as people to process our feelings as a way to take care of ourselves, but it’s also good for the research. What you feel influences what you see and hear, so talk about it to unpack your insights as part of your own unique perspective.
When I was in graduate school, I spent a month observing middle-school math teachers in Texas. I was documenting the choices they made with regard to technology use and how this affected their students. I remember so clearly that the teachers were incredibly sensitive to my presence there. Teachers are so routinely scrutinized, observed, and, frankly, criticized that they seemed amazed that at the end of my observations I wouldn’t have a laundry list of “improvements” for them. Yet to me, those teachers all seemed like wizards with the rare ability to get a room full of 14-year olds to engage with something meaningful. I learned a lot about the hidden aspects of teaching culture, as well as how my previous experiences with teachers and schools had biased my original perceptions.
That experience has stuck with me, and reading these stories in the article makes me miss it! I wish there was more room for ethnography in my current position. Thank you for a lovely article. I may have to pick-up the book, too.
In the past 15 years, Steve has interviewed families eating breakfast, rock musicians, home-automation enthusiasts, credit-default swap traders, and radiologists. His work has informed the development of medical information systems, music gear, wine packaging, corporate intranets, videoconferencing systems, and iPod accessories. Steve has lectured at Stanford University, Institute of Design, California College of Art, and UC Berkeley, and is an accomplished presenter who speaks about culture, innovation, and design at companies such as eBay, Adobe, Nokia, Hewlett-Packard, and Dolby Laboratories. He is the author of the Rosenfeld Media books Interviewing Users: How to Uncover Compelling Insights and Doorbells, Danger, and Dead Batteries: User Research War Stories and writes regularly on topics from interaction design to pop culture for Interactions, Core77, Ambidextrous, and Johnny Holland. He has a graduate degree in human-computer interaction from the University of Guelph and is an avid photographer who has a Museum of Foreign Groceries in his home. Read More
