How to Ask About User Satisfaction in a Survey

A 2006 replication of the study explained it like this:

“Winning any medal in Olympic competition is an amazing accomplishment. Folk beliefs suggest a linear decrease in positive emotions, with the gold medalists experiencing the most, followed by the silver medalists, and then bronze. However, in reality, [in Judo] the silver medalist loses the gold medal match, and the bronze medalists win their last match, capturing the bronze and avoiding going home without a medal…. Bronze medalists from the 1992 Barcelona Games appeared happier than silver medalists at the end of the match and on the podium, and … silver medalists’ comments were much more characterized by counterfactual thinking about ‘what might have been’ and were associated with greater regret and less enjoyable emotion.”—From

D. Matsumoto and B. Willingham’s “The Thrill of Victory and the Agony of Defeat: Spontaneous Expressions of Medal Winners of the 35 2004 Athens Olympic Games,” from the Journal of Personality and Social Psychology, September 2006.

Richard L. Oliver delves into the varieties of factual and cognitive comparisons that can contribute to satisfaction or dissatisfaction, which I have summarized drastically in Table 1. He points out that, if you don’t process the experience at all, the result is indifference—not dissatisfaction.

Table 1—Comparisons that contribute to satisfaction

An experience compared to…	A person’s resulting thoughts…
Nothing	Indifference
Expectations	Better / worse / different
Needs	Met / not met / a mixture of the two
Excellence / the ideal product	Good / poor quality / good enough
Fairness	Privileged / equal / picked on / disadvantaged
Events that might have been	Vindication / regret

Memorable Experiences Are Also Complex

If the emotions in satisfaction are complicated, what about the experience that gives rise to them?

Let me tell you a story about a rather simple experience. Today, I’ve used a whole bunch of Web sites: I bought some stuff. I browsed some stuff. I checked the news. But the experience that stands out as memorable: signing up for a Web site to which I’m going to log my routes and training for a half-marathon.

Was the experience satisfying? Well, yes and no. Looking at it through the lens of the different comparisons that affect satisfaction from Table 1, where did I start?

• I expected it to be straightforward. Registering for a Web site should be simple these days.
• I needed it to be reassuring. This half-marathon is double the distance I’ve ever attempted.
• Ideally, I wanted it to be inspiring—to suggest ways of conquering this training challenge in ways that I hadn’t yet thought of.
• I feared that it would be too US-centric and wouldn’t understand that I need to be treated a bit differently because I’m a UK-based person.
• If I hadn’t signed up for this site, I’d have had to go hunting for another one. Plus, over a few months, I’d already invested effort getting familiar with this site as an unregistered guest, so looking elsewhere seemed like a lot of work.

And where did I finish? With mixed results. It was mostly easy, but part-way through, there was a bit where I ended up paying a very modest fee to remove an ad. I had thought I was paying to use the site ad free, but then there appeared to be a fee to remove a particular ad rather than remove all advertising. Grumpily, I thought about spending time to find a different option. Later, I realized that I needed to sign in, so the site would know that I’d paid. Just a few bumps along the road, but they tangled me up and undermined the initial experience. Looking back, I had

• multiple goals—signing up, intentions about long-term use, meeting a personal challenge
• multiple levels of effectiveness—completing the signup, paying the fee, my initial use of the service, my long-term use
• a variety of types of resources—the actual payment, my time, my emotional reaction to the mix-up
• multiple contexts of use—while it was just me at my desk, there are so many different contexts of who I am as a person: beginning runner, UK person, UX consultant, supporter of a particular charity

So, on a scale of 1 to 10, how satisfied am I? You answer for me! Your guess will be at least as accurate as mine. The point here: memorable experiences tend to be complex ones.

The Opportunity for Satisfaction in a Survey: Focus

With all this bad news about complexities, is there any point trying to ask about satisfaction in a survey? My view: yes, definitely. The key is to pick one small aspect of satisfaction, and ask about that. Briefly. And provide space for a free-form comment.

Find Out About Users’ Contexts, a Bit at a Time

When I was first collecting stories about how UX experts use surveys in their work, Suzanne Boyd of Anthro-Tech told me about asking one simple question: “Why did you come to this Web site today?”

They’ve achieved great results from asking this question, and that makes a lot of sense to me. It’s an easy question, and it explores one of the most important aspects of user experience: the user’s goals.

Of course, it’s tempting to go on to ask about all the other aspects of a visitor’s experience at the same time. Many of the worst surveys that I see do that—apparently taking the view that, if they’ve managed to grab my attention “to answer this short survey,” I’ll be prepared to answer screen after screen of questions about all sorts of details.

Don’t fall into that trap. Instead, keep your respondents happy by asking just a bit at a time—and not every single time they visit your site either. You’ll gradually build up the details that you need.

Ask About Their Recent, Vivid Experiences

People are much better at reporting how they feel right now and what they’re doing right now than on their experiences from days or weeks ago. Try to answer these two questions:

• “Did you print anything in the last hour?”
• “How many pages did you print last month?”

Most of us find it much easier to answer the first question and are much more likely to give an accurate answer. But an easier question than either of these might be:

Did you buy a printer this week?

For most of us, printing is a rather frequent and somewhat trivial task, whereas buying a printer is a rather unusual, more important, and therefore, more vivid task. The more trivial the experience, the less memorable—so the less accuracy you’ll get in people’s answers about it. Try to ask about recent, vivid experiences.

Ask a Sample, Not Everyone

If you ask every customer to provide feedback, every time they visit your site, that’s a feedback form, not a survey. You know that, your respondents know that, and this contributes to declining response rates and poorer quality data—for both your feedback forms and your surveys.

Ask fewer people, and make it clear that you’re deliberately asking only a specific few. They’ll feel more special and be more likely to respond.

For Any Survey: Interview First

Often, stakeholders consider that it’s quicker to put together a quick survey than to go out to find users and meet them face to face. Or even worse, they might say, “Let’s do a survey, then do some follow-up interviews.”

Survey methodologists do agree that a successful survey should always include an interview stage, during which you explore the topics of the survey in detailed interviews with typical respondents. But the interview stage should always happen early in the survey process. Interviewing first saves so much time because:

• It helps you avoid problems like asking questions that users don’t understand or don’t have answers for.
• It gives you preliminary data from the interviews that may be enough to move your project forward without the expense of an actual survey

Summary

Satisfaction is a complex emotion, and memorable experiences are also complex.

Don’t try the patience of your survey respondents with long questionnaires that probe every aspect of those complexities. Instead, you should:

• Find out about users’ contexts—a bit at a time.
• Ask about their recent, vivid experiences.
• Ask a sample, not everyone.
• And as with any survey: interview first. 

Acknowledgement—Thanks to Francis Rowland for the sketches.

Resource

On SlideShare: “How to Ask About Satisfaction on a Survey”