Forge Conference 2015

The Hype and Reality of Big Data

Presenter: Judd Antin, Head of Insights, Airbnb

There are some people who inspire deep confidence with every word they speak. This is the kind of presence that Judd Antin brought to his talk on the common failings of UX research methods. It was a perfect fit for the subject matter. Judd communicated the comforting thought that even a person of obvious competence struggles in getting reliable insights, describing them accurately, and using them in the right way.

The Main Points

• There’s really no comparing the benefits of having an embedded research team versus a centralized team. When a team is embedded, researchers become experts on the products and their domain. Design inspiration and science should not be disconnected from one another.
• At Airbnb, the researchers’ mission is to “belong anywhere” and be involved at every stage. They’re skilled in all research methods, so they can use them as the need arises.
• Judd is very concerned about the rigor of research and is critical of both qualitative and quantitative methods. Both can result in specious findings or be bent to reflect personal biases. Researchers should be committed to high integrity in their methods or get out of the profession. It’s important to acknowledge that our methods are sometimes flawed and that we, as researchers, are also flawed.
• Confirmation bias is a common pitfall. This occurs when researchers believe they will see a certain outcome, so select findings that demonstrate the truth of their belief while ignoring those that would disprove it. Of course, they don’t do this out of malice or believe that they’re biasing the findings. It’s just how people think. 
• Anytime we say, “Let’s just do a quick study to validate our design,” we’re setting ourselves up for trouble. Judd calls this bulldozing. In this situation, the researcher is under pressure to confirm what designers believe to be true. For example, suppose you were doing usability testing on a completed product and found that the entire basis for its design was fundamentally wrong. Would you have the courage to say so?
• We also tend to develop bias toward things with which we’re already familiar. In qualitative testing, we sometimes latch onto a single thing that only one person said just one time, then use that statement as a lens for all subsequent findings. We may go so far as to embody that one thing in a persona and shape the entire design around a completely idiosyncratic remark. 
• Know your limits and acknowledge that you’re subject to the same biases as every other human being. Remember, objectivity is a myth.
• Know the limits of your methods. Each one has its strengths and weaknesses. Qualitative methods offer rich insights, but they don’t scale well, and you have to be careful about generalizing their findings. Surveys generalize very well, but it’s extremely difficult to write questions that everyone will interpret in the same way. Big-data metrics can describe a phenomenon with great precision, but they don’t tell you why it occurs.
• Adopt a multi-method approach. Select testing methods that may possibly cancel one another out. If you’re running a usability study, for example, write up your findings, then run a survey to see whether they hold up.
• All models are wrong, but some are useful. You won’t always have the data you need or the capacity to run multiple studies to serve as checks and balances on one another. Go with the best data you have, but make sure that everyone knows its limitations. You don’t want to launch with a false sense of confidence.

My Analysis

In my mind, this presentation was a clear standout. I was very gratified that Judd was saying the things that need to be said about the limits of research, researchers, and data. He was committed to creating a safe space in which to talk about these uncomfortable truths, so we could have an open dialogue about how to make things better. Judd’s presentation often felt confessional. He told the audience again and again, “It’s okay. I’ve been there, too.” My big takeaway from this talk was that the single most important quality of research is not sample size or repeatability or metrics, but honesty.

The Art of Messing with People

Presenter: Zander Brimijoin, Creative Director, Red Paper Heart

The cross-disciplinary ethic underlying Forge paid off in a big way with Zander Brimijoin’s presentation on his company’s interactive art installations. His work grew out of his dissatisfaction with the “immersive experiences” he used to design, which focused almost exclusively on what was happening on screen.

Zander found that people weren’t getting what he intended out of these experiences, so moved on to design work that focuses on the affective, physical, and kinetic elements of experiences. For example, Red Paper Heart’s Pool Party covered the surface of a swimming pool with Ping-Pong balls, then projected animations onto them. Some observers would plunge in, disrupting the patterns, becoming a component of the art themselves. The studio’s tagline is “Art & Code,” and their work delivers on its promise through aesthetics and technology.

The Main Points

• Red Paper Heart formed around the goal of creating child-like wonder. Zander’s vision was to create a sense that everything around us is new again.
• Zander provides only a bare minimum of instructions for any installation—or none at all. He likes to meld into the background and watch what people do when no one is there to direct them, observing how they decide what they’re supposed to do.
• There are risks to this approach. For example, one exhibit depended on the audience smiling.  But what if they didn’t smile? Could the project still be successful? How hard can we expect people to work to get the payoff?
• Is the creation of such installations UX design? Certainly not in the conventional sense of creating wireframes and affinity maps. But the installations do have users, are experiences, and are designed. We could get a fresh perspective on UX design through the lens of interactive art.
• Zander likes to call his work “the art of messing with people.” It’s about creating a feeling of engagement, immersion, wonder, and fun. It’s like playing a game at an art-gallery party.
• One of his challenges is how to draw people into an installation. It’s not reasonable to hand people a manual and expect them to follow it. Nevertheless, the introduction to an installation needs to be interesting enough that people feel like trying it. So Zander often lures his audience in with a recognizable device that people already know how to use—such as a tennis racket.
• Once people start to use the familiar object in expected ways, he likes to mess with their expectations. So a player’s swinging the tennis racket might control a projection display of fireworks. Zander wants to reinvent experiences through technology.
• The installations often use sensors that are embedded in objects. When using physical actions to control digital experiences, the accuracy of measurements is important. If a designer doesn’t take adequate care, there may be a gap between what the system is measuring and what users think it’s measuring. 
• Input technologies are always lossy. There are lag times and inaccuracies in measurements. Signals degrade at various points along the pathway from the brain to the digital input.
• In some cases, users may feel disconnected from the output. In other cases, they may discover ways in which to exploit the system’s inaccuracies. These problems have long existed with experimental video-game peripherals. For example, players could cheat Nintendo’s PowerPad by shifting their weight from foot to foot rather than actually running. It’s debatable whether people are actually cheating or just trying to do the best they can.
• Many installations use minimal cues to communicate what people should do. For example, designers have found that, if you draw a circle on the ground, people will intuit that they should stand inside it. People also learn through collaboration and by watching others, so installations often have spaces in which groups can work together, as well as spaces for onlookers.
• Technology does not always provide a great experience. Our personal devices have had the effect of isolating us, separating us from other people. Zander asserts that many people are starved for real-life experiences and advocates for building technology that helps us create such experiences.

My Analysis

Zander’s presentation powerfully illustrated that we cannot limit the definition of UX design to simple screenware. UX design encompasses all artifacts for human use. Zander vividly demonstrated the applicability of his designs for art installations to UX design work, with obvious analogs to user research, human factors, ergonomics, and service design.

Social Media and the Artist

Presenter: King Britt, Music Producer

This was the session that stood out more than any other. King Britt spoke extemporaneously, without any slides, outline, or pretense. He simply sat at the center of the stage, speaking softly over background music. He related the personal story of how his eclectic musical tastes developed over the course of his upbringing and have influenced his career as a DJ and music producer. King’s informal talk gradually built up to his central discussion about how he’s used social media to help build his career.

The Main Points

• King reflected fondly on his experience with MySpace, which allowed him to network with other artists, promote his music, and book gigs. He saw artists like Flying Lotus and The Gaslamp Killer build their brands on MySpace, empowering them to fuel their own career trajectories.
• Napster was a means of broadening your exposure to music and discovering the songs to which people with similar musical tastes were listening. It wasn’t just about sharing MP3s, it was about sharing culture.
• Critics’ blogs broke the spirit of community that had developed among musicians on the early social Web. People who didn’t understand the art wrote scathing reviews, changing the tone of the dialogue, shaming up-and-comers, and driving them out of the business. It changed the music, too, because musicians were trying to please the critics.
• King sees Facebook as less his style, but it’s his most productive promotional network. He times his posts in three daily bursts to hit peak usage in the US, Europe, and Japan. Facebook charges to boost exposure, and King pays the fee, but believes it’s exploitative.
• As an artist, King believes that Twitter is all about connecting with other artists. He feels a close personal connection with both his own fans and the people he admires, and he uses Twitter all the time. But this exposure comes with a downside, because it demystifies artists.
• King cautioned that the regular use of social media can be like falling into a rabbit hole. He finds himself spending much more time with it than he wants. He’s taken Instagram off of his phone more than once to keep himself from overusing it, and he occasionally goes offline entirely for a few weeks as an “Internet detox.” 
• King has banned Wi-Fi from his studio because people are there to work, and social media is just a distraction.

My Analysis

This presentation expressed what we all feel: while social media enriches our lives, it also consumes the time we need to put that enrichment to good use. King’s argument was not against social media, but that we need to adopt enough personal discipline to make the best use of these powerful social-media resources. He also acknowledges that doing this isn’t particularly easy.

Closing Keynote: Design Everything

Presenter: Cap Watkins, Vice President of Design, Buzzfeed

The conference organizers couldn’t have picked a more memorable presenter to send everyone home. Cap Watkins is an electrifying speaker, a shameless ham, and every ounce the hipster that you’d expect Buzzfeed’s head of design to be. He told anecdotes of his experiences working at Zoosk, FarmSpring, Amazon, and Etsy to explain how his view of design and his management ethic have evolved. His presentation centered on how he effected real change in the way Buzzfeed operates.

The Main Points

• We’re prone to thinking that we’re the designers, and everyone should just defer to our expertise. But when we do that, we’re not really being honest about our own capabilities.
• The truth is that design is hard and highly error prone, so we can’t put ourselves at the center of the universe. As Cap put it, “If Design made all the decisions, the world would suck and everyone would be depressed.”
• When Cap was at Etsy, the designers wrote and deployed production-ready code. But when he joined Buzzfeed, none of the designers were coding, and most didn’t know how to code. Everyone thought that was the natural way of things, but Cap’s experience had demonstrated that there are better ways to work. 
• When designers don’t participate in the coding, there are many small, but important things that just never get done. For example, there might be a screen element that looks just terrible because the padding is off by 5 pixels. But, if you entered a problem that minor as a bug, it would be given a very low priority, so no one would ever bother to fix it. However, when designers are empowered to code, they can quickly go in and fix problems themselves. Then things don’t look terrible anymore.
• Both designers and developers were anxious about making this change. They were being asked to adjust the ways in which they had been accustomed to working. They were suspicious about Cap’s motivations. He was challenging their professional identities.
• Change is stressful. When driving change, managers need to acknowledge people’s concerns and empathize with them. Make the people who are feeling the stress into your partners, so they’ll be excited about what you’re trying to achieve.
• Cap’s thesis was that we can see the work environment itself as a user experience and approach solving problems through design. “Design everything.” Whatever the problem, define an ideal state of the world and ask what success looks like.
• It wouldn’t have been enough for Cap to clap his hands and say “Change now.” If he was going to give people new and unfamiliar responsibilities, he had to help make the change possible.
• Cap got the designers a subscription to Team Treehouse, and they all learned HTML and CSS.
• They started collaboratively adding design notes about their projects in Basecamp, which served as a record of what they were doing and why. Everyone contributed to this dialogue, so developers had a hand in the process of design, too.
• Designers and developers collaborated on creating an atomic, immutable CSS framework called SOLID.
• Cap lied to new hires, telling them that people at Buzzfeed share their work with one another every day—even though they didn’t. Creating that expectation encouraged new people to act in that way, so over time, this became the reality.
• Cap helped people to figure out how to handle disputes. Often, this meant learning how to let things go. When two people couldn’t see eye to eye, he‘d ask both of them to rate how much they cared about an issue on a scale of 0 to 10. This turned the discussion into why something matters to them.
• Of course, things will go wrong. Get good at saying whatever you need to say, then letting it go and moving on.
• At the beginning of any major culture shift, people are going to say, “I can’t do that.” It’s the manager’s job to say, “Yes, you can. You’re awesome and capable.” Until you get where you need to be, the best thing is to pretend you’re already there.

My Analysis

Forge wrapped up on a theme that recalled the opening presentation. Both Cap Watkins and Michael Lebowitz were dismissive of the conventional strictures that job titles imply. Both value the capabilities of the individuals who inhabit roles. Both called for working environments that are more flexible, egalitarian, and collaborative. Cap’s talk provided specifics about how he’s gotten people to transition to a better way of working and feel invested in what they’re doing.

Final Thoughts

Forge is an easy choice for UX professionals in the Northeast. The conference delivers on its promise to find great voices who can give you new ways to think about your work. The speakers were all well chosen, the sessions ran like clockwork, and the setting was stunning. Just as Philadelphia’s UX community has grown, I expect this conference to continue to grow in the years ahead.