Doing Research with People Who Are Not Users: Consultation

We’ve chosen to focus on consultation with SMEs because many product teams correctly place a lot of value on having people working on their team—whether occasionally or permanently—who have first-hand knowledge and experience of being users, as illustrated in Figure 1. Some people—especially within the disability rights movement—argue that a team cannot be truly user centered unless it includes people who belong to the exact target audience. The slogan “Nothing about us without us” captures this idea. Demetrius Madrigal and Bryan McClain explore some of the arguments for and against having SMEs on a team in their UXmatters column “Subject-Matter Experts: Putting Users at the Center of the Design Process.”

Image credit: “Users and Experts,” by Naintara Land

We’ve also chosen to consider user advocates because of the fascinating work they do around raising awareness of and creating understanding of the needs of the groups they represent. For example, if you wanted to understand the impact of a new piece of legislation on people in the UK who are deaf or have hearing loss, it would be a good idea to ask Action on Hearing Loss for their views.

Clearly, there are many other variations on these themes. The Senior Manager we mentioned earlier undoubtedly has valuable subject-matter expertise. SMEs may have been user advocates in the past or continue to combine user-advocacy activities with their current role.

And there’s a variation of this that often applies to us personally, in our work as UX consultants: negotiating the overlaps between our role as consultants—whether as internal employees or outside experts—and consultation, by which we mean providing our knowledge and expertise. Plus, sometimes, to complicate matters further, we happen to actually belong to the target audience for the product or service, so are also users. We deal with this variation by making sure that we frequently ask ourselves, “Which role am I playing right now?”

User Research Aims to Discover What Users Do

Our overall aim for user research is to understand what users do and why they do it. It helps to inform us about user needs, then supports the design process by providing feedback about whether the product or service is meeting those needs.

So, when we do user research, we mostly choose methods that gather behavioral data to find out what users do. We conduct contextual interviews with users to understand how their environment and goals influence what they do. We do card sorting to establish how they group topics or where they would look for them. And, of course, we do lots of usability testing, watching users try to use a product or service.

Consultation Aims to Discover What People Know

Our overall aim for consultation is to discover what people know about a product or service, a business, or organizational needs. We might conduct a workshop with stakeholders to identify key priorities and areas of interest and concern. We may have one-on-one meetings with SMEs to learn about the history of the product or service—focusing on successes and failures so we can try to avoid repeating old mistakes. Or we might have a public meeting or call for interested parties to comment on proposed changes.

How a Consultation Differs from User Research

In the UK, many government services use consultations to call for comments on proposed changes. The UK Government Digital Service also mandates user research as part of the process for creating government services. In 2015, the UK Ministry of Justice described the results of a consultation on proposed changes to court fees. They asked questions about the details of some proposed changes and learned about respondents’ knowledge of the system, as well as their views on how the changes might work.

Alongside that, user researchers at the Ministry of Justice described how they conducted user research to improve their service that helps people with their court fees. They did usability testing of form prototypes with typical users. They also looked for behavioral data to establish whether the proposed new forms really did make it easier for users to get through the necessary processes.

Consultation Can Help You Improve the User Experience

If you followed the links to investigate the Ministry of Justice consultation, you discovered that the reports are full of very technical language and detailed questions. You may be surprised that we think consultation can improve the user experience of your product or service, but we do, and that’s what we will focus on for the rest of this column.

SMEs Help You Understand the Domain Much More Quickly

SMEs can help you understand the history, terminology, and aims of your product or service, bringing you up to speed much more quickly. The clue is in their name—subject-matter experts know about, well, the subject. They often have many years of experience in a domain and have previously seen teams try out all sorts of good and bad ideas. Consulting an SME appropriately can enable you to understand important issues such as legislation and standards—and avoid rookie errors in the scoping and design of your product or service. Of course, sometimes what was a bad idea some years ago might actually be a good idea today. But it’s helpful to at least have some knowledge about the past to avoid repeating an old mistake in exactly the same way, with the same result.

The knowledge and context that you can gain through consultation with SMEs can help you make better use of your time during user research. Instead of hearing about the domain for the first time from your users, you’ll just be validating what you already know and extending that knowledge.

For example, when Caroline was working on a system for farmers in the UK, she found it tremendously helpful to have discussions with an SME who explained many of the technical terms the farmers used in that specific context. For example, the term feature did not mean something the system could do, but instead referred to a permanent structure on the land—such as a water tower. The SME also explained some sensitivities about an earlier version of the system that helped her to prepare for negative reactions to the proposed new system from some users during her subsequent user research.

Referring back to our Ministry of Justice example, a new user researcher working in the area of fees might find it helpful to know that the courts use the term fee remission to mean a reduction in the court fee for a person on a low income.

Advocate Groups Help You Quickly Discover the Problems of Hard-to-Reach Groups

Another important advantage of consultation is that user-advocate groups have a more comprehensive understanding of niche or specialist use cases that might not come up during typical user research, but would have a significant influence on the overall viability of the product or service you are designing. An example: the UK suicide-prevention charity, Samaritans launched an app called Samaritans Radar, which they described as follows:

“A free Web application that monitors your friends’ Tweets, alerting you if it spots anyone who may be struggling to cope. The app gives users a second chance to see potentially worrying Tweets, which might have otherwise been missed.”

Although the original press release for the app mentioned consultation with academic experts, there was an outcry about the potential for those with less than wholesome intentions—such as stalkers, trolls or bullies—to exploit the app. The Samaritans recalled the app nine days after its launch. While we expect that concerns over privacy might well have emerged eventually through user research, perhaps the particular threat that stalkers and bullies posed could have been identified more quickly through consultation with other organizations that support vulnerable people.

Looking again at the Ministry of Justice example, one of the questions that came up during the consultation was specifically about the impact of changes to fees on “court users who have protected characteristics.” In the UK, the Equality Act 2010 uses the term protected characteristics to refer collectively to nine areas in which discrimination is illegal: disability, gender reassignment; marriage and civil partnership; pregnancy and maternity; race; religion or belief; sex, and sexual orientation.

Using the sort of participant recruitment that is typical of user research, it might have been difficult to find someone on a low income who needs help with their court fees and is also willing to disclose their sexual orientation in court. Reaching out to some LBGT+ advocacy groups might be a good way to discover specific concerns of people in those circumstances and would help the team to decide how best to involve them in user research.

Consultation Can Be Misleading

The term remission neatly illustrates one example of how consultation can be misleading. Once again, the clue is in the name: SMEs are experts and, therefore, are immersed in the language of their area of expertise. The Ministry of Justice user researchers found that people on low incomes did not realize what remission meant and were confused about the difference between legal aid—money from the government to pay for a lawyer—and fee remission—help with the fees they had to pay to the court.

We’ve found that people joining our teams as SMEs become full-fledged team members really quickly, which is wonderful. But it also means they rapidly become familiar with our jargon and the concepts for the new service. For example, if you were a member of the team that came up with the idea of filleted accounts, you’d probably struggle to understand how odd this term sounds to someone hearing it for the first time.

SMEs and Advocates May Focus on Edge Cases, Forgetting the Typical Case

We have learned to be wary about generalizing from the specific examples that SMEs and advocacy groups describe. If you ask anyone about their experiences over the last year or so, the unusual and exceptional events tend to be easier to recall than the mundane and typical. The people you consult—no matter how broad their background—are not immune to this effect. They can become quite focused on niche scenarios within their domain.

For example, Caroline recalls discussions with SMEs that occurred when she was working on UK tax returns. These SMEs had robust knowledge of some rather unusual topics—such as authors and artists averaging—that affect only a very small number of UK taxpayers. She does not recall ever hearing about childminders from an SME, even though there are far more childminders in the UK than authors or artists. Plus, childminders tend to be somewhat unfamiliar with the specialist language of taxes, so find completing their tax returns quite a struggle. But from an SME point of view, the taxation of childminders is quite straightforward.

SME Experience Can Become Dated or Overly Focused on Your Project

Similarly, based on their experience, SMEs may be able to tell you in great detail and depth how to do things. But the places they’ve worked or their contexts of use naturally limit and constrain their perspective rather than encompassing all variations and scenarios. Therefore, relying solely on SMEs for your understanding of user needs may introduce bias into your designs. It’s also possible that bias may occur just because you consulted or chose SMEs to work on a project specifically because their experience and consequent viewpoints might be more favorable to or aligned with the agenda of other stakeholders on the project.

For example, Tara recalls a teacher who acted as the SME on a project focusing on how to provide curriculum based–learning resources online. The teacher explained that that lesson preparation included creating both schemes of work and lesson plans. The team, including the SME, then embarked on a month-long discovery effort, carrying out contextual user research in over 18 schools across the country. During discovery, the team collected artifacts, including lesson plans—ranging from illegible scribbles in a pocket diary, at one end of the spectrum; to Word documents that detailed the learning objectives, starter, main and plenary portions of the lesson, as well as how to adapt the lesson for less-able and higher-ability students. This provided a very tangible demonstration of the range and diversity of planning artifacts. But, while the SME was able to point the way, there was still a need for user research to enable the team to understand in detail how to do things.

SMEs and Advocates Who Are Also Users May Not Be Typical

The final pitfall we’ve encountered is confusing the user with the typical user. We’ve seen projects on which user research was conducted solely with SMEs who were also users. Maybe, when teams recruit users internally, SMEs are more likely to sign up. Maybe the project team worked hard to bring a user onto the team to act as an SME, then couldn’t get the budget to bring in any other users for research. The risk is again the problem of expertise. When working solely with SMEs, you’re unlikely to capture the needs of the full spectrum of users. What about new recruits who have never seen the product before? What about the users who use the service only very occasionally, while the SME uses it every day? Maybe the SME has excellent digital skills, while some users may have low digital skills. User research with a full range of users would fill in those gaps and help you to create a more robust design, ensuring that everyone can use your product or service.

Similarly, we’ve seen challenges arise when a particular advocacy group captured stakeholders’ attention to the exclusion of other people who do not share their point of view. For example, one service became fixated on the needs of people with deep concerns about privacy and data sharing—even though the majority of users did not care about this. (Maybe the users ought to have been more worried about these issues, but the fact was that they weren’t.)

Consultation Saves Time and Can Deliver Better Value from User Research

You need to do both user research and consultation to resolve the tension between designing for the lowest common denominator, in terms of skill and knowledge, and supporting niche scenarios and edge cases. Consultation can save you time and money by flagging issues before you’ve committed too much effort to designing a solution that may just not be viable.