Benefits and Limitations of Doing Usability Testing with Your Colleagues

July 9, 2018

It is generally accepted that testing your product with your company’s employees comes with a certain set of risks. Employees, by default, have a higher level of company and product knowledge than your actual users. They harbor their own personal opinions about what direction the business and its products should take. There is generally a much higher chance of conscious and unconscious bias creeping into the data when testing with your colleagues.

These are certainly all valid points. However, there are some instances when testing with your colleagues can actually be of benefit—not just to your product, but also to your UX team and even the broader business. I’ve experienced these benefits myself.

Champion Advertisement

During the early part of 2018, my UX team at Emerald Publishing completed a round of usability tests with staff from our head office. The team completed twelve, scenario-based test sessions with colleagues from various departments across the business. This was the first round of testing for one of the company’s most important projects and proved to be a great success for a number of reasons.

Demonstrating Our Process and Achieving Buy-in

Emerald is in the early stages of its agile transformation and is slowly adopting User Experience as a bona fide function within the business. Over the past year or so, there have been a lot of changes in the ways we work. On this particular project, Product and User Experience are leading our effort to embed an agile approach within our development process.

This provided a great opportunity for my team to demonstrate the kinds of processes we employ for research and testing; show how much time testing takes; and exhibit the level of detail and quality of our outputs and the importance of this work in truly understanding users, identifying their painpoints, and addressing them by iterating on designs or creating new product features.

This demonstration of our UX processes has helped give our colleagues a clearer understanding of what we do and how we work. Testing has also convinced our colleagues of the value User Experience can bring throughout the product-development lifecycle.

A company-wide understanding and appreciation of the value that User Experience delivers—usability testing in particular—is highly important as our business embeds User Experience and agile in the company culture.

Refining Our Ways of Working

Despite its being in its infancy, my UX team is already involved in a number of different projects across the business. The team currently comprises three UX researchers and two UX designers. Our increasing involvement shows that the company recognizes the value and need for User Experience. However, before this project, we hadn’t yet had a chance to do any usability testing as a team. As it turned out, many of us were a little rusty when it came to organizing, moderating, and analyzing usability studies.

This was the perfect opportunity for us to shake off some of that rust, get our heads together, and plan and execute a usability-testing project within a relatively forgiving environment.

We came up with a test plan—including the scenario we were going to test against—recruited participants, scheduled the tests, set up the test environment, conducted the test sessions, analyzed the test artifacts, created the appropriate outputs for the client, and helped them to assimilate our findings into their product backlog.

Levels of expertise vary quite widely across the members of my team. Since we had not previously worked together, this usability study provided a good opportunity to find out how we worked together as a team and how we coped with the stress of an intensive two weeks of testing.

In the end, we were able to create an end-to-end, skeleton process for future usability testing, while shaking off the cobwebs and forming better working practices as a result.

What Did We Learn?

So what did we learn from this experience? We learned that we can achieve a lot in two weeks. With a small team, we managed to conduct twelve moderated, face-to-face usability tests and analyze and present our findings.

On the other hand, we learned that we may not currently have the resources necessary to scale this kind of testing for some of the larger projects in which we’ll be involved. For example, if we need to conduct in excess of 30 moderated usability tests every two weeks, completing this work to a satisfactory level is likely to require more human hours.

We need to decide on the appropriate methods for testing with real users. Would online, unmoderated testing be sufficient? Will we require more live, moderated testing? Or will a mix of both approaches be appropriate? How many test sessions would we need to complete versus the value each individual test would deliver?

We learned about our individual team members’ strengths and weaknesses, which will help us to better deploy the resources we have, where we can do cross-skill collaboration with one another, and where more training would be beneficial.

We also learned that it would be beneficial to have a UX lab set up in the office to properly curate the environment in which we test. This is something plenty of companies do, and it’s something I believe we’ll push for as a team rather than dealing with the rigmarole of booking random meeting spaces around the office and carting our equipment around to different locations.

A dedicated space would allow us to set up our equipment correctly, save plenty of time, and give us the space in which to analyze our findings appropriately.

Limitations

Obviously, there are limitations to testing with colleagues. Any UX team should be well aware of these limitations. I mentioned a few of these at the beginning of this article. Real users always provide the most accurate, trustworthy results in any research or testing scenario, helping you to identify key product issues and ensure that development addresses primary user needs.

When doing usability testing with internal colleagues, you must always be aware of the following factors:

bias—both conscious and unconscious
personal preferences and opinions
colleagues’ level of experience with your products
the time they’ve spent at your company

What’s Next?

These two weeks of usability testing have provided a fantastic primer for the team and allowed us to refine the next phase of our testing plan—during which we’ll conduct testing with real users. The lessons we’ve learned from our colleagues should stand us in good stead. They’ll help us to run the best possible testing program we can with real users and, thus, gain the most relevant insights for product development.

In Usability Testing

Benefits and Limitations of Doing Usability Testing with Your Colleagues

Demonstrating Our Process and Achieving Buy-in

Refining Our Ways of Working

What Did We Learn?

Limitations

What’s Next?

No Comments

Join the Discussion

Jamie Allen

Other Articles by Jamie Allen

Other Articles on Usability Testing

New on UXmatters