Revisiting the Pile-Sort Method of User Research

Our Study’s Setup and Procedure

To understand what activities a user task flow would comprise and what objects people associate with each activity in the task flow, we prepared 57 information cards, shown in Figure 2. Each card contained information belonging to one of the following three categories:

• user activities—25 green cards
• artifacts—26 blue cards
• attitudes—6 red cards

Our study included 12 participants of mixed gender, age, and ethnicity. Each study session involved one participant, a moderator who ran the study, and a note taker who captured participants’ comments on a laptop computer. Each session took approximately 30 minutes.

We divided each session into two steps. During the first step, we invited a participant to arrange the green information cards for user activities into a timeline, from the activity’s starting point to its end point. To avoid any potential priming effect, we randomized the pile of green information cards before presenting them to each participant. Study participants sorted the cards into a timeline, leaving out any of the cards for activities that were not relevant to them.

We employed an extended pile-sort method during each session’s second step. The timeline in which participants had placed the green activity cards during the first step served as the seed for the second step, during which participants assigned artifacts and attitudes to related activities. Participants placed blue artifact cards and red attitude cards around green activity cards. Thus, the results from first step predetermined the number of piles into which participants could sort the rest of information cards.

Figure 3 shows an example outcome from the first and second steps of a session. During the first step, the participant had laid out cards representing concurrent activities in a timeline, in parallel to each other, as shown on the left in Figure 3. During the second step, the participant represented an association between artifacts or attitudes and activities by placing red and blue cards next to green cards, as shown on the right.

Based on the associations that participants represented between artifacts or attitudes and activities, we performed weight assignment and factor analysis in later steps.

Aggregating User Task Flows

To aggregate the pile-sort results from individual participants, we first assigned a weight to each green card from step 1, with the first instance having a value of 1 and each subsequent instance increasing the weight by 1. The weight scores from different study participants varied greatly as a reflection of individual differences between them. Some participants organized events for friends and, thus, created long task flows, while others had many fewer tasks to perform in preparing to attend an event. Simply aggregating the raw weight scores would have biased the results toward those of participants with longer task flows. To overcome this issue during data preparation, we performed the first round of normalization, dividing each weight score by the largest weight from an individual, giving a score to relative weight values, ranging from 0 to 1

Aggregating the relative weight values from all participants gave us a clear picture of the position of each task in the overall workflow. However, as we observed later during analysis, participants selected some tasks more often than others. So, simply aggregating the relative weight scores from our first round of normalization would again have given a biased outcome—in this case, biased toward those tasks that participants selected more often. We resolved this concern through a second step of normalization, dividing each score in the aggregated list by the frequency of a task’s selection by participants.

Estimating the Contexts of Use Using MDS Analysis

To get a glimpse into the contexts of use for each task stage, we performed a multidimensional scaling (MDS) analysis on the data from the second step in our study, during which participants associated cards representing activities, artifacts, and attitudes with tasks.

Similar to Weller’s 1983 study, we calculated a proximity matrix by counting the number of times participants associated the two blue artifact cards with one green activity card, then aggregated the results from individual participants to produce an association map. The resulting MDS graph, shown in Figure 4, clusters the activities, artifacts, and attitudes our participants associated with each task into four major groups that we labeled Event Discovery, Learn Event Details, Coordination, and Go to Event. By investigating the contents of each group, we were able to understand the context of use at each stage of a task. This understanding laid the groundwork for more comprehensive studies on the stages of a task that the product team chose to focus on.

Conclusion and Future Improvements to Our Study Design

In this article, we have summarized our experience using the pile-sort method to study user task flows and described how we extended this method to accommodate particular project requirements when working in an agile environment. We hope that our approaches to research and analysis will benefit other UX professionals.

As we reflect on the research design for this particular study, we would also like to share our concerns about the study design—especially those regarding its robustness. In the second step of our study, during which we asked participants to associate information objects with tasks, we, in essence, set up a seeded version of the pile-sort method. The number of seeds—which participants determined during the first step of our study—has the potential to dilute participants’ responses. In other words, because we asked participants to group information into many piles, it was only by aggregating the responses from a larger number of participants that we could gain a reasonable understanding of the context of a particular task stage.

Finally, even though our study was quantitative in nature, the interpretation of its results in the multidimensional scaling (MDS) graph was subjective. We found a team brainstorming session beneficial to drawing conclusions from the data.