Conference Review: O’Reilly Design Conference 2017

Proceedings

This year, O’Reilly provided comprehensive information about the O’Reilly Design Conference 2017 via a responsive Web site—including daily schedules for the tutorials, keynotes, sessions, and events; as well as detailed information about individual tutorials, sessions, speakers, and events.

O’Reilly has preserved an extraordinarily complete level of historical information about the conference on its Web site, including speaker slides for many sessions—as well as videos of the keynotes—that the organizers added to the site after the conference. They’ve also provided links to video highlights on the conference site, some video clips on YouTube, and a collection of conference photos on flickr.

Plus, on the home page of the O’Reilly Design Conference 2017 site, you’ll find a Video Compilation link to a collection of videos of all tutorials and conference sessions on Safari. To view these videos, you must have a Safari membership, but they offer a ten-day free trial that requires no credit card.

Venue

The Westin St. Francis Hotel, on Union Square in San Francisco, which is shown in Figure 2, was a huge improvement over the 2016 venue: the Fort Mason Center for Arts & Culture. This luxury hotel provided an aesthetically pleasing environment that was much more conducive to both learning and socializing, as well as a much better hotel and dining experience. The original two twelve-story wings of this Beaux-Arts hotel with 250 rooms opened in 1904; the modern, 32-story, 600-room Pacific Tower, in 1971.

Image source: Westin St. Francis Hotel

Beautiful, eye-catching O’Reilly Design Conference signage in the Landmark Lobby guided me up a flight of stairs with more colorful signage, shown in Figure 3, to the Registration desk on the Mezzanine, shown in Figure 4.

Image source: O’Reilly Media, Inc.

Image source: O’Reilly Media, Inc.

The main deficiency of this venue was its poor navigability. Since the hotel was built in two stages and the two parts aren’t optimally integrated, it was somewhat difficult to get from sessions in the old part of the hotel to those in the new part. Plus, even though most of the sessions took place in rooms in the original hotel, those rooms were spread out across a couple of floors and laid out in a confusing manner, as shown in Figure 5. The Grand Ballroom, where each day began, was hidden behind the Italian room, which was packed with sponsor booths that completely blocked the view of its entrance. While the conference organizers did provide good signage, indicating the way to the various rooms, there weren’t always good lines of sight that would let one see any distance ahead. I wish I’d had this map of the venue during the conference. It provides a better representation of the venue than the map on the back of the single-sheet schedules attendees received daily.

Image source: Westin St. Francis Hotel

On Monday, the tutorials were held in the neighboring California East and California West, just up another flight from the Mezzanine, and in Tower Salon A. On Tuesday, while most of the session rooms—California East, California West, and the Olympic room—were arrayed around the Grand Ballroom where the keynotes took place, shown in Figure 6, others were in the distant Tower Salon A. On Wednesday, sessions again took place in the Grand Ballroom, California East, California West, and Tower Salon A, as well as in the Georgian room, which was near the Grand Ballroom. Even though the Grand Ballroom where the keynotes took place and California East and California West were in fairly close proximity to one another, the latter two rooms were up a flight of stairs.

Image source: O’Reilly Media, Inc.

In most cases, the sizes of the rooms in which particular sessions occurred were well suited to the crowd size. When there was space, but not enough chairs, the hotel staff was great about bringing in more chairs to accommodate everyone. However, another issue was that there were very few places where people could sit down in the common spaces or the Sponsor Pavilion—in the Colonial and Italian rooms—where refreshments were usually served.

I commuted into San Francisco daily for the conference, but would have loved to stay at the Westin St. Francis—an exceptionally fine conference hotel. A key asset of the Westin St. Francis as a conference venue is its central, downtown location on Union Square, providing easy access to restaurants, nightclubs, shops, and public transportation—including the Powell Street cable car—that can get you anywhere in the city.

Hospitality and Events

On Monday, during morning and afternoon breaks from the tutorials, refreshments were served in the Mezzanine Foyer. During the main conference, on Tuesday and Wednesday, the day started at 7:30am, with coffee service in the Sponsor Pavilion, shown in Figure 7. At morning and afternoon breaks, refreshments were served in the Sponsor Pavilion.

Each day, attendees gathered for a complimentary lunch. Dining rooms were set up in Elizabethan AB and Elizabethan CD on the second floor, adjacent to California East and California West sessions. Alternatively, attendees could take glass-walled elevators up to the 32nd floor of the Tower, enjoying expansive views of downtown San Francisco on their way to the main hotel dining room, Alexandra’s, shown in Figure 8, where floor-to-ceiling windows afforded stunning views of the skyline, bay, and bridges. I cannot comment on the quality of the food because, unlike most conferences, O’Reilly did not provide lunch for the media.

Image source: O’Reilly Media, Inc.

Image source: O’Reilly Media, Inc.

O’Reilly organized a great variety of events during the conference, as follows:

• Monday, March 20:
  
  • Architecture Walking Tour of San Francisco—Information architect Jorge Arango guided this architectural tour of downtown San Francisco, describing how to apply wayfinding and context for architecture to designing information environments for digital products and services. He also discussed the elements that compose the city and the various forces that, over time, have shaped it. I was very sorry to miss it because of a scheduling conflict that afternoon.
  • Opening Reception—At the end of the day, this reception took place in the Sponsor Pavilion where hors d’oeuvres and drinks were served. Figure 9 and the hero image at the beginning of this review show the crowd that gathered for the opening reception.
• Tuesday, March 21:
  
  • Design Dash 5K Fun Run and Walk—This 5K run took place at a bright-and-early 6:30am, beginning at the St. Francis, going over to Market St. and taking it down to The Embarcadero, then taking a left to proceed to Washington St. The return leg back to the hotel covered Washington, Kearny, and Post Streets.
  • Tuesday Speed Networking—Before the opening keynotes, attendees could gather for rapid-fire, 60-second conversations with fellow attendees.
  • Tuesday O’Reilly Author Book Signings—During the morning and afternoon breaks, attendees lined up to meet O’Reilly authors—in the morning, Kristin Skinner, Peter Morville, and Jorge Arango; in the afternoon, Jim Kalbach, Cathy Pearl, Kathryn McElroy, and Richard Banfield—the first 25 getting a free, signed copy of a book. Figure 10 shows Cathy Pearl’s book signing.
  • Tuesday Lunch and Birds-of-a-Feather (BOF) Sessions—Attendees with common interests shared lunch and ideas, discussing topics that included Design and Emerging Tech—VR, AR, AI and Machine Learning, and Robots—Hardware and the IoT, Voice Design, Data and Design, Enterprise UX, Chatbots, Diversity in Design, Design for Social Good, Design Startups, Design Education, Prototyping, User Research, Design and Business, Managing and Leading Designers, Women in Design, Design in the Public Sector, and Mentorship and Apprenticeship.
  • Design Portfolio Review—A limited number of attendees could sign up for a 20-minute portfolio review by experts from “some of the world’s most design-centric companies.”
  • Sponsor Pavilion Reception—Attendees gathered for drinks, socializing, and a chance to check out the sponsors’ wares.
  • Design Dine-A-Round—O’Reilly made reservations at nine of San Francisco’s many great restaurants, where conference goers could meet up with one another for dinner. Nothing for vegans though.
• Wednesday, March 22:
  
  • Wednesday Speed Networking—Some attendees came together before the opening keynotes for brief, 60-second conversations.
  • Wednesday Lunch and Birds-of-a-Feather (BOF) Sessions—Again, attendees with common interests gathered at lunch for round-table discussions, covering similar topics as on Tuesday. Figure 11 shows a BOF discussion.
  • Wednesday O’Reilly Author Book Signings—Another round of book signings took place during the afternoon break on Wednesday. Attendees lined up to meet O’Reilly authors Tim Frick, Elizabeth Goodman, and Martin Charlier, in the hope of getting a free, signed copy of a book.

Image source: O’Reilly Media, Inc.

Image source: O’Reilly Media, Inc.

Image source: O’Reilly Media, Inc.

For those of us who weren’t fortunate enough to get a free O’Reilly book, there was still some choice swag for conference-goers, including the booklet What Is the Internet of Things?, a print version of O’Reilly’s 2017 Design Salary and Tools Survey, some cool socks. Figure 12 shows the O’Reilly booth in the Sponsor Pavilion.

Image source: O’Reilly Media, Inc.

Community

Creating a unified sense of community at such a large conference is challenging. There was a large contingent of San Francisco designers present who already belong to a somewhat cliquish community that may have felt closed to outsiders—designers from across the US and the world beyond. The many social events that O’Reilly organized probably helped attendees to meet new people—especially the speed networking sessions and BOF lunches.

This conference again attracted quite a diverse community of designers, specializing in many different design disciplines. Speakers represented a broad spectrum of levels of professional competency—from the expertise of thought leaders, including UX leadership and some O’Reilly authors, to specialists at earlier stages in their career.

Tutorials

On Monday, March 20, six half-day tutorials—three in the morning and three in the afternoon—provided deep dives into quite diverse topics.

Morning

• “Mapping Experiences: From Insight to Action”—Jim Kalbach, Head of Customer Success at MURAL
• “Org Design for Design Orgs”—Peter Merholz, VP of Design at Snagajob, and Kristin Skinner, Head of Design Management at Capital One
• “VUI Design: How to Wireframe for Voice”—Ann Thymé-Gobbel, Voice UI/UX Design Lead, D+M Group

Afternoon

• “Designing Intelligence and Trust with Chatbots”—Austin Beer, Experience Designer at Huge
• “Explore, Persuade, Destroy: Storyboarding for Product Development”—Abi Jones, Interaction Designer at Google
• “Make Fast, Learn Fast: Introduction to Prototyping with Arduino”—Kathryn McElroy, UK Design Lead at IBM Watson, and Kelly Lohr, UX Designer at IBM

I decided to cover Peter Merholz and Kristin Skinner’s tutorial “Org Design for Design Orgs,” while Krispian Emert chose to review Ann Thymé-Gobbel’s tutorial “Voice User Interface Design: How to Wireframe for Voice.”

Org Design for Design Orgs

Reviewer: Pabini Gabriel-Petit

Presenters: Peter Merholz and Kristin Skinner

Peter Merholz, shown in Figure 13, is VP of Design at Snagajob. A cofounder of Adaptive Path, he was Head of their Design Practice. More recently, was VP, Global Design, at Groupon and Senior Director of Design at Jawbone. Kristin Skinner, in Figure 13, is Head of Design Management at Capital One and was formerly Managing Director at Adaptive Path. She is also the programmer and host of the LX Conference—formerly MX. Peter and Kristin are the coauthors of the O’Reilly book Org Design for Design Orgs: Building and Managing In-House Design Teams.

Image source: O’Reilly Media, Inc.

Image source: O’Reilly Media, Inc.

After Peter and Kristin had introduced themselves, Peter spoke about the origin of their book: “I realized, upon leaving Adaptive Path and starting to be a design executive in house, how little there was out there about shaping design orgs—very few resources to turn to. So having my own experiences, I started to reflect on those—and, with Kristin, realized there was a book and workshops to be taught on this subject.”

This workshop was, as Peter described it, “low on activities and high on discourse.” “The nature of talking about Design organizations is one of leadership, and it tends to work better when we talk about it, hear your stories, hear your questions, and share with one another. … Let’s have a conversation,” suggested Peter.

Kristin presented the agenda for the workshop, as follows:

1. Your Product and Design Organization—“We want to talk about your Design organizations. What type of organization do you work for? How big is your team? What is your role?”
2. The Expanded Role of Design—We’ll go through the current state of design. Where are we in the industry right now. What are the trends that are happening? What are the different maturity models that we’re seeing. Peter and I have done a bunch of research around this. We’d love to hear your perspectives as well.
3. Organizational Models and Design Output—We’ll go through the organizational designs.
4. The 5 Stages of Design Organizations—We’ll go through a lot of detail about the rationale—why we’re recommending that you have certain roles come on at certain times.
5. The Untapped Opportunity for Design—If we have time…

Your Product and Design Organization

The workshop kicked off with a five-to-ten-minute activity. Kristin asked us to describe our own Design org by drawing an org chart, listing the roles, and noting anything else that defines our organization—such as measures of success or a mission statement. “What we really want to see is, on one sheet, what your Design organization looks like to you. Then, we’ll talk about why things are structured that way.”

Peter advised, “Don’t over think it. Just quickly sketch it out. Focus on your Design team’s org chart. Then, if you have time, you can sketch in what the company looks like around it. How it relates to the broader organization. List out the roles within your Design team—what you call them.” At the conclusion of this exercise, several workshop participants described their Design organizations.

The Expanded Role of Design

“We’re putting forth a set of themes and a thesis around design.” stated Peter. Communicating how important it was for all of the workshop participants to have a shared understanding of design, Peter told us, “We’re going to talk about the expanded role of design because it’s a way for us to define what we mean by design. Companies are investing in design. There are two primary reasons companies are making this investment:

1. Design leading to innovation leading to the ability to generate new business value. To generate new revenue streams—new business value—innovation became the new black, and design kind of followed along on that wave, associated with innovation. Tim Brown, the CEO of IDEO, talked about the power of design for companies to realize this new value. While important, that would not be sufficient. Most [designers] are not actually doing this type of work. so that wouldn’t be a sufficient explanation for why companies are investing so much in design.
2. The more active driver is that every company is now a tech company, because software is now essential for business. You need design in order to build good software. So we’re following on that wave as well.

“So companies are investing way more in design than they had before. And, as part of this wave of every company becoming a tech company, everything is also becoming a service,” as shown in Figure 15. “The implication is that all design is service design. It’s important for design to orchestrate the whole experience.”

Image source: O’Reilly Media, Inc.

“What’s changing is that design is seen less as a phase in a process than it is as an activity that can inform an entire process. So design can help you think better from a strategic and management standpoint.”

As Figure 16 shows, “the double-diamond diagram is a way of thinking about design process—actually, product-development process. Historically, Design was seen primarily as an execution function. Now, Design is being brought in to help figure out: what should we be doing to build the right thing, as well as then how do we build it—building the thing right. So we’re starting to see Design be more involved in strategy, planning, definition, and thus, being involved earlier in the process.”

Image source: O’Reilly Media, Inc.

“Design is not just about execution anymore. Design has the opportunity to embrace a more expanded role where it helps identify and generate new business value, manage the complexity of software, cohere multichannel service experiences, and orchestrate and coordinate all those different efforts. Design isn’t just a phase in a process, but a tool that can enhance all practices. Design can help define new offerings—do the strategy and definition work—as well as the execution, and in doing so, inform the planning of that execution. So this is the larger mandate for Design now.”

“As companies are making these big investments in Design, the concern that we have is: they’re not setting Design up to realize this full success because they’re trying to bring it into a structure that is not optimized for great design.” As Design leaders, “how do we create a space for a Design organization to thrive within these other contexts, so that investment does realize its full potential?

“There’s a lot of stuff that happens behind the scenes that affects the ability for design to succeed that often doesn’t get addressed. Design success is predicated on the ability of Design leadership to figure out how to work the organization—to figure out Design operations: Are you staffed appropriately? Are your teams structured well? Are they communicating well? Are they coordinating internally well, and are they coordinating with other teams well? All this stuff matters way more than people recognize.”

Kristin told us, “One of the things that helped to inform the book was our research. We went out and talked to 25 other Design leaders to figure out how they were setting up their teams, what the challenges were, what was working well and what wasn’t. So that helped inform our perspective as well.”

When Peter launched a discussion of the ideas he’d presented during this segment of the workshop, a participant asked: “How do you hire designers to do this kind of work—as opposed to what we’ve traditionally looked for?” Peter replied, “Going into organizations and inheriting design teams where, when I was brought in, I was told, ‘We want to be a design-led organization,’ and then I looked at the team and they were really good executors. This is not the team. There’s a conversation about UX maturity models. Those are usually about the outside organization’s readiness to invest in design. What I saw, though, is it’s less about the outside organization. I’d step into these companies, and they’re like ‘What do you want? We’ll give it to you.’ And it’s more about the Design team getting in its own way because it’s so focused on craft and execution and not embracing the breadth and the strategy. Think about your Design org from the standpoint of teams, as opposed to individuals, to be able to address the breadth that Design is taking on.”

Organizational Models and Design Output

Next, Peter talked about the pros and cons of various “models for how teams organize.”

• centralized internal services—“When companies first embrace design, they often do it by having a centralized internal services model for design.” They recognize that design is core to their business, so they build an agency in house. These teams tend to be project-based and organized by function, as shown in Figure 17. “This is design as a phase…, as opposed to design pervading an entire organization. Design is not strategic in this mode. There’s no room for innovation, creativity, or strategic thinking.” This model “doesn’t work because it disempowers us.”
• decentralized and embedded—Some companies create program-based teams, each of which gets a designer, as shown in Figure 18. This is similar to the Spotify squad model. “These teams work best when they’re given a lot of autonomy. All of those teams are working in isolation.”
• centralized partnership—“Think in terms of design teams,” creating a hybrid, federated model, which allows both load balancing and specialization, as shown in Figures 19 and 20. “Every team of designers is committed to partnering with a set of product teams around contiguous or coherent parts of the user experience. Try to get the best of both worlds—the things that worked well in that centralized model, but also what makes decentralized work.”

Image source: O’Reilly Media, Inc.

Image source: O’Reilly Media, Inc.

Image source: O’Reilly Media, Inc.

Image source: O’Reilly Media, Inc.

Peter suggested focusing design teams on different customer types, as shown in Figure 21, then scaling while maintaining the same focus. He said, “There is a sentiment out there around organizing by customers and their journeys that is not just happening in design, but I think we’re starting to see happen more broadly. And I think it’s happening because of everything becoming a service. At the heart of any service is the relationship between the company and a customer. When you embrace what it means as an organization to be delivering a service, it means that you’re realizing that customer relationship is more important than anything else you’re doing.”

Image source: O’Reilly Media, Inc.

Peter led a discussion about embracing the central partnership model by designing in teams and organizing by customer journey. In response to a question from a design leader whose team was too small and, thus, spread too thin, Peter said, “You need to make the case for hiring more designers. You’re going to have to operate under that fully centralized model if you’re not going to be given enough resources. Focus on the most important issues that your company is facing and get [your team] to work the right way. Then show [the results they can get] when they get to work the right way.”

Kristin added, “Funding models will always get in the way, especially in a large, distributed organization. It’s a real challenge to figure out who’s actually funding design. So, when you’re trying to get everything centralized, that’s an essential component that needs to come along with it. That’s why having design positioned correctly and having autonomy to figure out how you’re actually going to spend that budget is super important. It’s essential for this to work well—or for it to work at all, frankly.”

After the morning break, Kristin took up the presentation. “Once you have all these design teams, you need one Design organization to tie it all together. That’s one of the core tenets of the centralized partnership. Organize by customer journey. What does that require? All of this,” said Kristin, gesturing toward the slide shown in Figure 22. “It requires a lot of different skills. These are the core skills when we’re talking about the current state of design.”

Image source: O’Reilly Media, Inc.

“But more importantly, once you have all those skills, you need to figure out what’s the right mix. When do you bring them in? What are the right ratios? What are the teams actually working on? You really need to think about all the levels of scale the Design organization can address. There needs to be one autonomous, well-positioned, influential person thinking about the big picture. [You need] a team of folks who are focusing on the strategy. You need a team of designers who are focusing at the structure level—and then, the surface. You need folks who are focusing on these four core areas.

Kristin led a brief activity in which participants assessed their team’s levels of scale and what percentage of their Design organization is focused on each level. Peter interjected, “It’s meant to be where you’re able to put your effort and how much weight you can give each of those. How capable is your Design org across all these levels of scale?” Kristin proposed these levels: “About 5% of the focus on the big picture, 15% on strategy, and the rest split between structure and surface.” Peter said, “Structure is super important to get right. In my experience, most teams have underinvested in the structural aspects of design. Keeping your structure and surface at parity is important.”

In the discussion that followed, participants considered the capabilities of their teams to deliver across the entire customer journey. Are they able to do this?

“You might have multiple teams,” observed Peter. “The structure of each design team should be four to seven people. If you get much more than that, your design team is probably just too big. Once you get to eight, split into two teams of four. If you think about designer to developer ratios, anything from 1:4 to 1:8 to 1:20. IBM is a 1:8 company, so let’s use that. Seven designers can, thus, theoretically keep fifty-six engineers busy. If do you need multiple teams, the team still stays the same. You’re just creating multiple teams to work across these larger efforts, and they might need to figure out ways to coordinate if you need to make sure this experience is coherent.

“The argument isn’t that you can solve all problems with four to seven designers, it’s that four to seven is a good number for a design team that’s working together toward some [goal]. If it gets much bigger than that, the communication and coordination just within the design team starts getting out of hand. If you’ve got one leader and twelve designers that are all trying to work together, that just becomes unwieldy.

“One of the bigger implications of this model of the one Design team is that Marketing and Product Design are the same team, which is not common. The recommendation is: Marketing and Product Design should be on the same team.” Kristin interjected, “It’s all customer facing.” Peter continued, “It’s all delivering against a customer journey—particularly in a connected services world. Now, it’s all the same. It’s all screen-based experiences that are butting up against each other if you’re having two different Design orgs in the same company trying to work together.”

Kristin asked, “Where does Design live? Who’s the head of Design? This is a really big indicator about the areas of focus, the skillsets, and the work that you’re doing, depending on who’s in that role and what their experience is with design. In some design-led organizations, there are VPs of Product that have design backgrounds, and there are some that have engineering backgrounds. A Design team needs to be in charge of its own destiny, and this requires focused leadership with autonomy and executive access. Executive access is really important. It might be okay to roll up through the VP of Product. But you’ve got to have autonomy, so you’re not wrangling for budgets, you’re not wrangling over who’s responsible for delivering on the objectives and the strategy that’s being set at the highest level. Also, it’s super important that you’re really close to the CEO of the organization. If your Design organization is embedded lower than two rungs from the CEO, it’s probably going to be an exponential degree of challenge for you to have that big-picture view—because that work is being done elsewhere, and you’re probably inheriting it.”

“Customer Experience is funny because it’s not a well-defined field right now,” said Peter. “Ultimately, I think we could all find ourselves working in a Customer Experience organization. We’re not there yet.

In answer to a question about who a Creative Director should report to, Peter said, “The Creative Director should probably report to [the head of Design]. The problem is: if you have multiple design leads, then their boss is the head of design, and that person often doesn’t know about how to make design go best. By having that single head of design, you know where the buck stops when it comes to any design decision.”

The 5 Stages of Design Organizations

Peter introduced this segment of the workshop saying, “We’re going to walk through the evolution of design organizations. For each stage, we’re going to talk about roles that we’re introducing. You don’t need every role at every stage. There are certain roles you don’t until you get to a certain size.”—and outlined its five stages, as follows:

• Stage 1: The Initial Pair—A Head of Design, whose role is creative, managerial, operational, and hands on—and a Product Designer, who focuses on structure and surface. “You should never have just one designer working by themselves.”
• Stage 2: A Full Team—This full team of seven now includes several Product Designers. Kristin made this important point: “The expectation is that the Product Designers in Stage 1 and 2 are doing research.” Peter continued, “So user research is happening earlier. You just don’t have a dedicated researcher.
  Two new roles join the team at this stage:
  
  • Content Strategist—The focus of this specialist is voice, tone, content structure, and copywriting.
  • Communication Designer—This designer is responsible for branding, graphic design, and information design. “Because Marketing and Product Design are all on the same team, you’re going to need a Communication Designer—somebody who’s familiar with offline design, if you’re doing print and packaging—marketing design.”
• Stage 3: From Design Team to Design Org—At this point, the Design organization splits into two teams and the Head of Design may struggle with scale. One of the teams still reports directly to the Head of Design. “The Head of Design needs to start leveraging [himself or herself] and expecting others to do the work,” said Peter. “Design teams need strong, singular leadership. This pivot point of around eight to ten people on a design team is often where you do end up bringing in new design leadership.”
  Two additional roles join the team:
  
  • Team Lead—One of the two teams has a Team Lead “who does creative direction for the team. [The Team Lead] might be a people manager, but not necessarily.”
  • UX Researcher—At this stage, you can have a dedicated researcher who covers generative and evaluative research. “It is possible to build really good products with no user research. It is impossible to deliver a really great service experience with no user research. User research is necessary to articulate … an end-to-end experience. You want somebody to manage that process. This person should be doing everything they can to continue to level up the research capabilities of the whole team.”
• Stage 4: Coordination to Manage Complexity—Now, the Design organization comprises three teams, as shown in Figure 23. “Going from 3 to 4, you introduce some roles to manage this greater complexity and scale of what the design team is now overseeing.”
  Three more roles join the team:
  
  • Service Designer—This dedicated Service Designer focuses on strategy and structure. At this stage, you need “someone to help coordinate the experience across all these teams. Someone who’s working on journey maps, experience maps, service blueprints—doing that superstructural work to make sure that all these efforts are cohering.”
  • Program Manager—Kristin described this role, which focuses on design operations and organizational effectiveness. “The way we have Design Program Management set up at Capital One is based on the model that Peter was describing earlier—having a Team Lead partnered with a Program Manager. We have one Design Program Manager to a team of 20 designers. A very senior person who is really running the practice of design for that line of business or that part of the organization. They’re focused on making sure that we have a clear definition about how design work gets done—we have a good understanding of the roles on the team and where there are gaps. We’re helping with things like culture and professional development. But primarily, we’re really making sure that we understand the velocity, the forecasting we need to be paying attention to for the work that’s coming. Also, the health of the practice.”
  • Design Manager—One or more Design Managers join the Design organization to lead individual design teams. This person is a people manager, but still practices design. “At this size, you need more management within the team,” said Peter. “This might be a Team Lead role—not necessarily.”
• Stage 5: Distributed Leadership—“Stage 5 is basically Stage 4 doubled. We had three teams. Now, we have six,” as shown in Figure 27. Once you’ve gotten so big, it’s hard to expect one leader to do everything well. “Find people who excel at different aspects of leadership and have them work as a unit.” Thus, there are a few new roles:
  
  • Design Director—“We have Design Directors instead of Heads of Design. Someone to be the bridge between the Head of Design and a complement of teams.” They oversee a variety of teams and experiences. “They are working to make sure that all these efforts are integrating across these teams.”
  • Creative Director—“At this point, if not sooner, the Head of Design—even if they’re doing creative leadership—has got such a large organization they probably can’t deliver enough of it. So, a dedicated, very senior Creative Director—who may or may not have a team and, if they do have a team, it’s a couple of folks—can focus solely on creative leadership and establishing quality standards.” (This was influenced by Doug Bowman, who was Creative Director at Twitter. He was the one responsible for the quality of Design at Twitter, establishing guidelines, quality standards, brand standards.)
  • Director of Research—“We’ve got five researchers, so now we need someone who can run that team. This is a small team within the Design team that sees a lot of what’s going on in the organization.
  • Creative Technologist—This role is responsible for prototyping new experiences. “These folks are primarily technical, but are not expected to do production work. This role is about helping design teams explore technology. They might be building tools that designers can take advantage of.”

Image source: O’Reilly Media, Inc.

Before describing Stage 5, Kristin took what Peter called The Operations Detour, asking whether our teams suffer from the issues shown in Figure 24. “Think about what your team is trying to address. You’re in growth mode. Are these symptoms happening? What are the practices you could put in place to try to address them?

“I’m a big proponent of having small, dedicated teams tackle bigger problems. Think about bringing in a small team of people who are really dedicated to solving these types of issues.”

Image source: O’Reilly Media, Inc.

Figures 25 and 26 list some operations issues. Referring to those issues in Figure 25, Kristin said, “If you are only doing only this—if you’re just thinking about scheduling and budgets and tools and procedures—all of these things are table stakes.”

Image source: O’Reilly Media, Inc.

Then, referring to those issues in Figure 26, she continued, “But when you think about these activities, this is where the Head of Design can get really bogged down, so it’s best to think about it from a team perspective, if you can. Think: Is a dedicated service organization part of my team that could really help me with these things the way we've structured things? What are the right partners, the right skillsets, to help you get these things done?”

Image source: O’Reilly Media, Inc.

There was a brief discussion of operations issues, during which Peter remarked, “What’s interesting to me about this is the number of design teams that don’t have someone in this role. But if you don’t have someone in this [Program Management] role, someone has to do all that work. You cannot run an organization without paying attention to the operations. But if this becomes the whole job, that means this person isn't doing their creative or managerial leadership either. Because once you get to a certain scale, operations end up taking over. Operations become the highest priority because you've got to keep them all going. As the organization scales, this just grows with it. Program Management is a role that you should have on your team if your team is larger than ten people. This is explicitly not about delivery. This is about getting the team working really well.”

Image source: O’Reilly Media, Inc.

In Conclusion

I found this workshop really valuable. Peter and Kristin are both highly skilled speakers and put together a beautiful slide deck that was just packed with useful information. The discussions they led were really stimulating as well. It was fascinating to hear about how other UX leaders are handling the challenges of our profession.

VUI Design: How to Wireframe for Voice

Reviewer: Krispian Emert

Presenter: Ann Thymé-Gobbel

In her workshop on voice user-interface (VUI) design, Ann Thymé-Gobbel posed the question: since a wireframe is visual, how do you wireframe for voice? The answer: write and record sample dialogs—an audio wireframe. With VUIs, specific changes in the wording and delivery of a voice prompt can massively affect how users interpret and respond to it.

Ann, who is shown in Figure #, outlined the concepts, tools, best practices, and challenges of designing voice-based experiences. Her workshop covered how to write sample dialogs, build audio wireframes, test prototypes for voice user interfaces, and do a short hands-on Wizard of Oz test—all while exploring different tools for documenting VUI design. (This is still a shockingly sparse area.)

She broke her workshop into the following parts:

• Introduction
• Basic Concepts and Terminology
• How VUI Is Different from GUI
• Steps to Wireframing a VUI Dialog Design
• Workshop Exercises

Introduction

Ann Thymé-Gobbel’s background is in linguistics and cognitive science. At the beginning of the workshop, she polled the audience. Most of the people in the room identified as graphic user interface (GUI) or user interface (UI) designers and had some kind of Alexa or Google Home device.

Ann began her workshop by describing some of the drawbacks of VUIs. She said she wishes she could talk to her Apple Watch. She told a story about once asking a voice-only interactive assistant: “What’s the current Giants score?” The assistant replied, “San Francisco Giants or New York Giants?” Ann said, “It didn’t know anything about me or the context of my question. What might have been a better way to reply? In what other ways might you ask the question to get the response you wanted? Consider the following questions: What is the premise? What do I know about the world? What do I know about this person? How might I make this conversation flow?

Basic Concepts and Terminology

Ann covered some basics of speech recognition:

• Performance is amazing in comparison to just a few years ago.
• There is much more awareness of and access to devices that speak. They are everywhere—for example, in cars, toys, call centers, and medical devices.
• It takes two to have a conversation. Therefore, VUI design assumes a human-to-computer, spoken conversation.

We learned that social expectations guide spoken language. Spoken language is uniquely and innately human. Language comprises infinite rule-based combinations of a finite set of audio units. Humans subconsciously follow rules of conversation and discourse. Ann said, “There is an expectation of things that speak. Research shows that we expect things that speak to act human. We make judgments and evaluate speakers based on their voice. VUI design requires conscious effort to account for this and rely on these expectations. For example, a banking app shouldn’t have cartoon voice.”

Ann went on to tell us about the inexactness of language: “How many ways can you say yes—correct, sure, you bet. And what about how meaning changes based on intonation? For example: “Do you want to go to a party?” “Yeah!”—a definite yes. “Oh, yeah” means “Oh, I forgot.” “Yeah, sure” is sarcastic.

To summarize the basic concepts and terminology, Ann cautioned us to remember that speech is linear and fleeting, so to lessen the cognitive load on the user, speech must have a narrow information stream.

How VUI Is Different from GUI

Ann taught us that a VUI needs to:

• anticipate—flow
• influence—prompt
• respond to—flow and prompt

When wireframing a VUI, you need to conduct the following steps:

1. Gather requirements.
2. Create high-level dialog flows.
3. Generate sample dialogs—that is, VUI mocks.
4. Verify the design.
5. Create a VUI Design Spec.

VUI design needs to consider the user’s requests. For example, “How would the user get out if he got stuck?”

Together, we learned how to wireframe an example dialog flow, using the following principles:

• What does user want?
• What does system need to fulfill that request?

Ann instructed us, “Your dialog flow should show how to get from A to B for the user’s main path and reflect their mental model through dialog.”

Steps to Wireframing a VUI Dialog Design

Taking the example of weather, Ann said, “There are many ways of asking: ‘How is the weather?’” For example, the system needs to know when and where you are. It is easy to make assumptions, but the system doesn’t know where you are—for example, you could be on Skype. Nor does the system know when you need the weather—now or tomorrow.

Ann taught us, “Bits of information can be thought of as slots.” In the weather example, location and time are slots.

We did an exercise together to learn how to wireframe a VUI for a weather app, as follows:

1. Create a grid, with a column for each case.
2. Plot out the user’s needs—to get weather for a location and time.
3. Do we know the location? If not, how can we get it?
4. Do we know what day? If not, how can we get it?
5. Do we have enough information to finish?
6. If yes, what is the resulting action?

Workshop Exercises

The workshop attendees then tried designing their own dialog flows. Some questions that came up included the following: “What about ambiguous words like soon or nearby?” “The answer depends on context,” replied Ann. “We have these amazing devices that promise more than they can pull off.”

As we worked on our flows, Ann taught us some best practices for VUI design:

• Minimize cognitive load through clarity and efficiency.
• Use terms users say naturally.
• List the most common commands first.
• Allow the user to control the information flow.
• Maximize accuracy and graceful recovery.
• Provide users with an easy way out—for example, by listing Exit as one of their options.
• Don’t blame the user.
• Escalate error handling, but know when to stop.
• Use discourse markers, natural language, and personas.
• Discourse markers are words such as now, next, and alright.
• Say sure or okay. A little goes a long way.
• Use spoken language, not written style—for example, use contractions such as you’re versus you are.
• Follow rules of turn taking.
• Do use a preposition at the end of a sentence.
• Define a system persona. Which persona is right for your VUI? “The answer,” said Ann, “is that you’re conveying something to the listener. Think about what you want to convey. Don’t be creepy. Don’t know too much about the user.” Ann showed some creepy examples—hacked Billy Bass and skeleton Alexa.

Flowcharting

The exercise then moved on to plotting a flowchart and creating sample dialogs. Workshop attendees paired up and discussed what the system would say and what the user would say. Ann gave us some pointers: “Be brief. Say only what you need to, when you need to. All prompts should have a consistent style.”

Prototyping

We then learned how to prototype our flows by recording sample dialogs, following these steps:

• Use Mac dictation to generate text-to-speech (TTS).
• Use the built-in voices Tom or Samantha. “It is important that you record with two contrasting voices,” said Ann.
• Record your voice with QuickTime player.
• Set up playback for key commands, so you can respond to your flows quickly.

Wizard of Oz Testing

The final step after creating a prototype would have been to interact with sample users as though we were the computer. While we didn’t really get to this step during the workshop, the techniques we learned seemed very useful.

In Wizard of Oz testing, simulate the application logic and the speech recognition engine. To do this, participants are given task scenarios. Ann taught us that the designer has to be the wizard. Have a moderator observe and listen to the user. Ann cautioned: “Be sure to respond accurately and appropriately—for example, simulate barge-in.” The wizard plays pre-recorded prompts in response to whatever the participant says.

In Summary

The workshop went by very quickly. From workshop attendees’ level of engagement and the enthusiastic discussions afterward, I could tell that everyone thought it was a useful and worthwhile workshop.

Conclusion

In 2017, the O’Reilly Design Conference was much improved over the previous year’s inaugural conference. O’Reilly presented a great variety of excellent content and the conference’s production values were superb.

While I’ve been looking forward to attending their next Design Conference, it seems O’Reilly may not be putting on a Design Conference in 2018. If you’re a designer, the next time O’Reilly puts on a Design Conference, I recommend that you attend it.