Visual Data Collection: Making Sense of Tally Sheets

Participant Annotations for Discussion-Guide Questions

Gathering participant annotations onto a tally sheet is most useful for capturing responses to the key design questions a study needs to answer. These might be open-ended questions or closed-ended questions. Figure 3 demonstrates their use in gathering responses to closed-ended questions, with lines connecting participant annotations to each of the responses. As you collect more data in your tally sheets, you’ll continue adding more participants to the responses. As Figure 3 shows, Participants #1 and #2 did not understand the purpose of the site they were viewing, while Participant #3 appears to have understood the site’s purpose.

Participant Annotations on Design Elements

As Figure 3 shows, you can link participant annotations to specific elements within a design. Rather than writing down what a participant is describing, use the visuals within the discussion guide to replace those words. For example, instead of jotting down “Participant #3 is referring to the Find a race button when she says, ‘It looks like a running site because I can find races,’” the researcher simply draws a connector line to the Find a race button on the screenshot in the discussion guide, adding the participant’s quotation next to it. This approach makes notetaking substantially easier and interpreting your annotations becomes very obvious later on. The researcher won’t have to wonder, “What are they referring to in the design?!”

Participant Annotations to Capture Emerging Trends

The surprising insights that come from participant or customer feedback are what make UX research so interesting and valuable to product teams. Perhaps, during the data-collection process, you might discover that you had overlooked asking a very important question in the early sessions that then arose in a later research session.

The tally sheet is an ideal place to gather such new, after-the-fact questions, ensuring that you’ll capture everything you need to support your research goals. Even though you were unable to gather specific data from the earlier participants on such emerging trends, you can go back to your transcripts or recordings later on and revisit evidence that can help you to answer these new questions. At a minimum, you’ll have captured the emerging trend somewhere rather than just keeping it your head, risking that you might forget it later on.

For example, perhaps a number of participants—presumably people who are interested in running—commented on how important receiving advice from experts has been in their lives as runners, but your discussion guide didn’t facilitate your capturing such insights. You can still capture this feedback in a structured, albeit ad-hoc fashion by noting the emerging trend you’ve identified and using participant annotations with it, as shown in Figure 4.

In the lower-left corner of Figure 4, the researcher captured the finding that participants desired expert advice from experienced runners in the form of a question, with two possible responses—yes and no. Lines connect specific participant annotations to each of these responses, depending on whether participants expressed interest in reading expert advice on Middlepacker.com during a research session.

Because there was no plan in place for the researcher to ask participants whether they were interested in expert advice from other runners, the researcher would likely have to go back and mine the data from the earlier sessions to check for the presence of this finding.

Managing a Cluttered Tally Sheet

As you might have noticed in Figure 1, tally sheets tend to become quite cluttered. Of course, if you’re running a study with only a handful of participants, you’re less likely to run into space problems when adding more data. However, as your sample size increases to perhaps to ten or more participants, the tally sheet can become overcrowded as you add more data, making the job of analyzing the data later on much more difficult. Therefore, working with the tally sheet becomes unwieldy.

Using Connectors More Efficiently

One way to manage the clutter is to use connecting lines more efficiently. For example, when logging the same finding for multiple participants, rather than repeating that finding multiple times, which would take up a lot of space, connect multiple participant annotations to just one instance of the finding. If you need to elaborate further, just add a branch to a related finding, as shown in Figure 5.

As Figure 5 shows, participants found the foot ad on the site unappealing. Connecting lines demonstrate specific participants’ relationship to this finding. Plus, the researcher added quotations from a couple of participants that add dimensions to the finding—beyond the foot ad’s just being tacky.

Switching to Landscape Orientation

Another way to create more space for annotations is to lay out your discussion guide in a landscape orientation. Also, if you end up with a larger sample size, consider adding data to the back of a tally sheet, where you should have plenty of space. If you decide to reserve the back of each tally sheet for additional data, make sure your printing preferences are set to print on one side rather than two sides of each sheet. If all else fails, you can put your data into a spreadsheet.

Analyzing Tally Sheets

Once you’ve collected all of the data from your annotated discussion guides onto tally sheets, you can start analyzing the data for useful findings and insights.

All of your up-front work on completing the tally sheets minimizes the effort it takes to analyze the data. There is usually little reason to go back and revisit an individual participant’s discussion guide to locate a finding once you’ve consolidated everything onto the tally sheets—unless you’ve inadvertently omitted logging some data.

In a future edition of Discovery, I’ll describe how to analyze the data you’ve collected onto your tally sheets, so stay tuned!