The Art of User Science

Brent: I'm Brent Tworetzky, Executive Vice President for Product at XO Group.

Brian: I'm Brian Smith, VP of Marketing at UserTesting.

Suzanne: So we're here at INDUSTRY Conference. The two of you just gave a super exciting talk on user science, and I wish for our audience that they were able to attend and hear it. But for their benefit, why don't we just start right there, because this is ... Is this a proprietary term, by the way that you're starting to polish up, user science?

Brent: The term user science emerged from a few things that I'd been learning. I originally picked up the term consumer science when I learned from the former head of product at Netflix. I was at a company called Chegg. He was the chief product officer there and I learned a lot from him about doing qualitative research, interviews, observational studies. I combined his qualitative research with some AB testing work that I’d learned from other product schools of thought, surveying and mixing it all together. I really liked the idea of calling it a science, calling it a real field, but not all of our users are consumers. So to make it more broad, not just consumer science, we called it user science because small businesses are users and companies are users too.

Suzanne: Yeah, we talk only to product managers on this show, and one of the themes that has come up is this tension between being quant-driven versus qual-driven. And there's sort of there's a school of folks that say, "You have to get out and talk to people. You have to be qualitative." And then there are no shortage of great platforms and folks that say, "You must be data-driven. You must have numbers. You must have those insights." What I really like about this user science framework as you've set it up is that it's actually a celebration of both those things. Both those things really being a necessity. What I especially liked in your talk was that you presented it in a 2x2 grid, which as we know is a favorite tool of product managers everywhere. "Let's put it in a 2x2 grid and see what happens." Maybe you could just take a moment and walk our listeners through kind of that 2x2 grid framework and how user science really seeks to bring qualitative and quantitative methods together holistically.

Brent: To start, user science is about fully understanding your users, both their needs, desires, wants as well as their actual behaviors and interactions. As you said, sometimes we do UX research and we can get really great into the qual, and understanding them. Sometimes we really get quanti. To bring it all together into one field is user science.

Imagine if you would a 2x2. On the left side is user needs and intents. On the right side is their actual behavior.

On the top row is qualitative research, meaning talking to people individually, a couple dozen folks, really getting the human story. On the bottom row is quantitative research, converting people to numbers, surveying thousands of people, looking at your Google Analytics or Mixpanel to see millions of sessions.

So across that 2x2 of both intent on the left, action on the right, qual on the top, quant on the bottom, that's how you can think about placing your tools. For example, one on one interviews, focus groups, are on the top left because they're a small number of people and they're about understanding intent. Surveys, or looking at SEO results are on the bottom left. You still get into human intent, but you do it at scale.

On the right side of their behavior, the way that you look at behavior in a small amount is through watching people. Watching people, use your prototype. Have them come in to your office and use your products. Remote usability testing is a great tool for that. And on the bottom right are the tools for looking at behavior at scale, tools that product managers are often very comfortable with. AB testing, and analytics platforms.

There are more advanced tools that go across all of these things together such as looking at customer feedback. It can be small or large, it can be about their needs and their behaviors. You can do beta tests which stick in the top row, the qualitative, but can take you from their needs to their behaviors. And my personal favorite, diary studies, which is a method of following users over a long period of time, understanding what their needs are, how they actually use products and what are the trigger points along their journey that make them make decisions and take actions.

Suzanne: Beautiful. And speaking of tools, Brian, you represent usertesting.com. I hope all of our listeners are already well aware of your product. But in case they're not, can you just tell us what is usertesting.com?

Brian: Sure. So, user testing enables product managers and others, user experience researchers, designers, marketers, executives ... to create great products, incredible products, by leveraging this power of human insights. So what Brent’s talking about with qual versus quant, we see all of these things coming together. So user testing, we are very much the qual. We have quant components, but it's about seeing the emotional reaction to what you put in front of them, whether that's a sketch up, whether it's an inVision prototype, whether it's a real live site or app or mobile site or even like an in-world experience, like an unboxing.

We enable you to see people's reactions and hear them talk out loud. Their wants, their needs, their emotions, their real feelings about what the product experience is. So for example, we provide videos and audios of people using your actual product or tool.

Suzanne: Yeah. And it's so important because number one, I think the reason that a lot of people don't get out of the building, right, so to speak, is ... Well, one, we can forget. I actually have a hypothesis that the better we are at product management, the susceptible we are to forgetting to go through the steps. When you're learning, you almost have that checklist. When you know it all, you can sometimes be a danger of going too quick. We don't get out of the building or we struggle like, "Where am I going to find someone? I don't want to the grab somebody at Starbucks and show them a paper prototype. And then we allow that fear to talk us out of doing user testing altogether, which for precisely the reasons, Brent, that you're sharing, isn't the right approach. Usertesting.com actually allows product teams to conduct important usability tests really at scale saying, "We will find the people for you. We will allow you to facilitate it." Candidly you can still sit at your desk if that's comfortable to you, but don't skip on the testing.

Brian: Sure. Two points there. One is, product managers or anyone in the company, you don't know what you don't know. So you need to get out there. And then second, most product managers are not creating a product which is for them. So how do they know what to build if they're not getting out of the building and talking to people? As for our platform: UserTesting absolutely has revolutionized how people think about gathering this human understanding. It is intimidating for people to think, "Oh, wait, I have to get in front of someone at Starbucks. Sure, I'm giving them the $10 gift card, but you know, I'm bugging them and is this real?" And truthfully, it's not that real. They're not necessarily in their normal environment at that point, where it's like, "You're going to use this at home. Why am I bugging you at Starbucks?"

So UserTesting allows you to interact with our panel of a million people. You can also bring your own employees or your own customers in as well. The beautiful thing about having access to a million people, our panel, or to use your employees or your customers, is that you can target people exactly. You can find that person in Cleveland, Ohio who has this salary range, who has this background, with Web or app experience, and really understand their problems, and therefore solve for those.

Suzanne: Brent, you and your product team use usertesting.com. It's part of your toolkit?

Brent: That's right. I've been using UserTesting since 2010. I love remote usability testing. I've been using it since before usertesting.com because capturing a user in the context of your product allows you to get a signal so much more quickly than building out the whole thing. And bring them into the lab, and doing that a bunch of times. When UserTesting came on and said, "We can do this quickly for you," and at the time it was one-twentieth of the cost of using an external facilitator, it was revolutionary.

I love seeing users using our products. I love getting that feedback quickly. I brought UserTesting to four companies that I’ve worked for, and the platform is fantastic, Brian, but it's also the notion of remote usability testing. It's accessible, it's human, watching the video and the audio is visceral on seeing how users stumble over your product and you feeling embarrassed when you see that, or loving your product and giving you confidence of the direction to go in.

Suzanne: To what extent does usertesting.com augment my ability to prepare the questions, and sort of prepare the test? Because I think this is also another fear is, "Alright, so I've got a prototype. I can do that. I sketch something, or put together some wireframes ... What I don't have is an experienced researcher or an experienced interviewer who knows how to take a subject through the different cues. Is that part of the service offering?

Brian: Sure. So, UserTesting, and we can actually drop the “dot com” because we rebranded, and we're really excited that people are talking about us as UserTesting because we're in so many different places right now. But to answer your question, the great thing is that you don't have to have this knowledge of user research. It's a platform which allows anyone to complete a study. We call it a study. That can even be intimidating.

The great thing is once you log in to the system, and are like "Okay, I want to ask questions," we have templates for that. So we found through our own user testing, that people came in and were like, "Uh-oh, what do we do now?" And as a business person, running marketing, I'm like, "Wait, how are people engaging with the platform because I have to make sure that a year later, after they've subscribed, it's like, hey, they're renewing and they're growing with us as well.

So what we've done is bring templates into the process very early on so you can say, "Hey, what am I actually trying to do here? What problem am I trying to solve for? What stage of development am I at?" And we prompt you with all of these great templates which we've developed with user experience researchers over the years. So like we know the right way to ask a question, and what questions to ask in different circumstances.

So really easy to get started. That's not to say that you shouldn't be working with a trained user experience professional, but we can get you going. The great thing is going out there in the world, talking to product managers and giving them a quick introduction to this, or even marketers or executives, they're like, "Wait, I can just click a button and get going and within a couple of hours, get this great insight back?" They start doing more and more of these tests, and they learn throughout the process, "Wait, there's a great way to ask a question where you're not biasing people.” And we prompt you along the way to prevent that from happening.

Suzanne: Awesome. Brent, most people probably don't know what is XO Group. Can you tell our audience what is XO Group, or maybe more importantly, what are the products that live inside this organization?

Brent: XO Group is a 21 year old tech company. We are a life stage, media and tech company, primarily helping couples through life's most challenging and exciting moments. We're particularly known for The Knot, which is America's leading wedding planning app and site. Also, The Bump which is one of America's leading pregnancy apps. And we have a few other great services that help couples navigate, enjoy life's moments together.

Suzanne: One of the things I heard you speak about is the scale of your product teams. So you have a handful of products, and then you have about 10x the number of product managers. 30 product managers on your team? Is that where you're at?

Brent: That's about right.

Suzanne: That's a huge team.

Brent: We have a wonderful large team, and part of that, is because wedding planning is very, very complex. We're tackling one of the hardest things that people do in their lives. And our role in that journey is to help couples pull off their perfect wedding with confidence, connecting them with the pros, and serving the pros that help them pull off that day with confidence, and helping them work with, communicate with, manage their guests who are coming there.

Wedding planning is a 300-hour project and event-planning exercise that very few people have ever done before by the time they come to planning their weddings. It is hard. It is complex. There's the inspiration side of things, how do I articulate my style, what type of wedding do I want ... there's the planning and project management side of things. How do I stay on track? What should I do next? What should my budget be? There's the vendor management side of things. Well, I got to pick a venue, but then I also on average need to get seven other vendors. How much do I pay? Who's the one who's going to be right for me? Do they work well together? Are they going to pull off my style? And then there's also managing your guests, which is how do I communicate with them? How do I find gifts for them to buy me ... and all the other products that I need to buy along the way.

It is such a complex journey. It's mostly digitally under-served today, so people kind of stumble through it along the way. And the stakes couldn't be higher. Not only is it one of the most complex things that people have ever done, it's one of the most expensive things they've ever done, and it is the most important weekend of their lives. At the time, that's what they're feeling. And many couples, many successful couples would say that it was the happiest weekend of their life. It's your show. It's one of the few times in life where everyone you care about is with you there in that moment. It's hard, it's important, it's expensive. We have a lot of products to build to help people.

Suzanne: I'm curious listening to you speak, especially given that the suite of products are really centered around key events in a person's timeline, which is different from say a platform like UserTesting where if I'm working actively in product or I have a product, I can you know, be using that tool regularly and ongoing, and assuming you continue to deliver value for me, Brian, month over month over month, never leave. How do the metrics that matter change when the products that you're running aren't really about retention over longer periods of time?

Brent: The way we think about our metrics is, we want to help all couples pull off their perfect weddings with confidence. That's a phrase that we use internally that brings it all together. We're successful if every year we can have every couple become members with us. If we can serve them in all their key use cases. If they stay with us, and if we're providing them real value, and we delight them, that's our goal. Along the way, we're collecting information from them so that we can provide better recommendations both to them and to future couples going through the same journey.

So those are the audiences that they'll come, they'll use us and they'll leave. And we also have audiences that do stick with us, particularly wedding pros. We're the leading way online to help couples connect with pros. So we are one of the main sales marketing branding ways that wedding venues, wedding photographers, wedding DJs connect with their customers online. We take them seriously as users as well. They have needs that we hope that we're serving, being great partners to them, giving them business, giving them branding, giving them insights, so that they can run their businesses more successfully.

Suzanne: Can you recall a particularly exciting revelation, piece of insight that you gained specifically from user testing for one of your products?

Brent: For sure. We use user science all the way through, top to bottom, end to end, but also at the macro level and the micro level. So we've run diary studies that have helped us say, "Gosh, Brent, you guys are solving the wrong problems at the wrong time.' When people first get engaged, their biggest problems are, "Gosh, what do I do next?" And "What is my style" and "How do I find my dress?"

In the middle of the journey, it's "How do I find the rest of my vendors?" "How do I collect my guest list and navigate the hairy things of who's coming, who's not coming?" Starting to create a registry ... and at the end of the journey, it's "Gosh, this big event's coming. How do I make sure I pull off the day?" So project management for the day, handling all the guests, and the last inputs.

User research has helped us figure out what problem to solve at what time, which are the most emotional problems that need different types of product support, what are the problems that we can solve that other people can't solve versus things that are well met out there. It's found gaps for us such as tools to help people articulate their style. You know, Pinterest does a great job at collecting images, but it doesn't help you put that into a way that helps you articulate it to your vendors and your florist and your photographers so you get what you want. So we have tools that help our users solve that particular challenge that no one has helped them before.

We found out through user research that was an unmet need. We've always had great tools like a checklist and a budgeter, but we have learned that users need a better way to manage their vendors and find vendors as part of a team. So we built tools that help you assemble the right team for you. And so on and so on.

We learned people need a stronger guest list manager that's more social, that needs to be heavily mobile because collecting information about addresses and phone numbers and emails is better done from your ... from all the contacts you already have in your phone, and so on and so forth.

Suzanne: Great. Brian, I'm sure you have no shortage of case studies given how many clients kind of come through the platform. But are there any that stick out particularly in your mind as big wins that clients were able to achieve, specifically from engaging user testing?

Brian: Like you said, there are a lot of examples. So my first love was B to C, and especially around eCommerce. So I was in marketing at lots of eCommerce companies. And eCommerce companies are so metrics focused where it's like, hey, every single step along the way, whether it's the login experience, search, browse, navigate, and especially that checkout flow is just so important. We're tracking everything through there, so that checkout flow especially ... that last page or those couple pages leading up to it, are so critical.

And I think because of that, lots of eCommerce and retailer companies adopted us very early on. So we have a plethora of stories there like we work with plenty of other industries. But it was really interesting to see eCommerce companies take this from day one. And it wasn't the small businesses. It was the large Walmarts and Overstocks and, you know, eBays and Amazons of the world because one percentage increase in conversion rate for them can be millions, tens of millions, hundreds of millions of dollars over the course of a year.

So our best stories are along those lines where it's like, "Hey, if we actually change this button to say something else or how it appears or where it appears," those learnings are incredible. So in a specific case, because I know you’re like, you're going to drill me down into that area anyways ...

StubHub, so, ticketing. They presented the checkout option in such a way that people weren't sure, "Wait, am I going to buy there or is there another step afterwards?" Through user testing, they were able to figure out this problem there, and come up with a creative solution. And this will sound small. They increased their conversion rate on that page on ... it was a mobile checkout page ... by I think it was 2.6, it might have been 2.4%. But that small increase accounts for million of dollars of sales for someone like that. So that's one of my favorite examples.

Brent: I have another fun example of using user testing, remote usability to uncover something surprising. One of the most recent trends in weddings and wedding gifting is couples wanting to receive cash. Digital cash is a large growing trend. There are already many cultures out there today that primarily give cash for weddings. But it's also very popular for people to register at large retailers, and we support those cases very well.

We wanted to build a digital cash offering for our users. But it's such a sensitive topic. How do you say, "You know, I'm getting married. Give me cash. Come online, put down your credit card and choose to give me cash, and do you cover the fee or not?" It feels impersonal at such a personal moment. We worked with remote usability testing. We put our users through and we tested a number of different ways of how might someone broach the topic with their friends and communities to get gifts? What's right wording to use? So saying, "Give us cash, cash fund, cash account," felt uncomfortable for the couples. They didn't know how to articulate that.

But we stumbled on, listening to them, in their words, we'd ask them, "How would you describe what it is that you're looking for?" And we heard them eventually come up with "funds." And we came across the word, "Give us funds," give us what we would ultimately call "the newlywed fund." But for the purpose of a skiing lesson or scuba diving lesson, when you took the word "cash" out, and you replace it with "fund," and when you attached it to buying something specific so the giver knows what it's for and can feel more connected to the couple, we iterated our way to figuring that out and we launched that earlier this year, what we call the Newlywed Fund. We make sure that you attach your asks to experiences or products. It has gone remarkably well. Had we just taken our best guess of "How do I articulate this?" we wouldn't have come up with those insights and this would've been a less successful product for us.

Suzanne: I mean, hearing you go through all of these features is making me lament just a little that I'm already married and didn't have the benefit of all of this functionality when I was going through the process.

Brent: Just you wait. The Knot is making planning weddings so much easier and more fun. And we've got lots of great stuff coming.

Brian: I wanted to echo what Brent said. He could've AB tested 20 different versions of that phrase that have actually became Newlywed Fund. And he might have eventually got there, but part of the value, part of the beauty of something like UserTesting is you uncover these hidden gems. You're asking people to talk out loud about their experience which means they veer off the path a lot, and we've found that product managers especially, love that veering off the path. Maybe it goes back to your previous question like, "Oh, how hard is this to do? It's kind of intimidating. What if I ask the wrong question?"

The great part is, you can do it as a recorded session, but you can also just have a live interview. You can say, "Hey, in three days, I want to talk to someone like this" and then just start talking to them. Don't even send them to a page, as you’re ... to say, "Look at this," just start talking to them and be like, "Wait, okay, how do you do that? Can you show me that?" And they can start going to Google and searching for, you know, wedding planning. So you can uncover these hidden gems and usually, it's those hidden gems between all the actual activities or tasks which provide tremendous value.

Suzanne: Yeah, what you're hinting at before too with the numbers and the AB testing brings us back to that tension that we spoke about at the open, which is if I'm too data-driven, then I'm looking at numbers and I'm perhaps getting insights. So this is broken. Retention is broken or it needs to be increased. Or churn needs to be decreased. But you speak in the framework of this marriage of the what and the why, and both in terms of the overall discipline of user science. But even in the putting together of your product toolkit, that the tools that you have are supporting kind of both of these things.

Would either of you like to share a little bit about what are some of those other tools in your toolkit, and how do you sort of bounce them off of each other to ultimately get to the insights that you need?

Brent: At XO Group, we have a toolkit that we have rolled out that everyone uses. It's specific choices. For building things, we start with sketch files, which we then turn into interactive prototypes with InVision. Like our other tools, we train everybody on how to use this so it is a shared tooling and language throughout our internal product community.

From there, we do iteration, we use it internally, but then we’ll release it through remote usability testing on UserTesting as platform, or in other ways. And so we are all trained on how to do that. We then use Mixpanel and Google Analytics for tracking, and Optimizely for AB testing, as well Taplytics and our mobile platform. Everybody's trained on how to use all of these tools, which just makes it so much easier when you’re in our community, when you’re in our company, to learn from one another. There are other great tools out there we are checking regularly, but that's a great set of tools that meets an organization our size. The most recent tool we've started playing with is Lookback, and it's overlapping to UserTesting. It fills a slightly different need, which is interacting with the users as they're using your site live. Some of our teams are using that tool now.

Brian: The one thing I'd add which I think ... When Brent brings up user science, I love it because it's like the complete solution. You're seeing the 360 degree view of your users, customers, guests, those humans. You're getting that human understanding. And so with the qual plus quant, you're really getting there and going back and forth throughout this 2x2 grid. It's amazing.

One thing which you guys don't use as much I think as some other companies which we’re seeing out there are surveys. So SurveyMonkey is one platform out there. Qualtrics is another platform. And you're talking about how all these things interact. I think we've heard some examples from Brent already and from me. With surveys, it's really interesting. You will have companies like Delta send out a survey after every single flight. Imagine how many millions of surveys they’re getting ... There's this incredible quantitative data in there, and people love that. They go gaga over that. Things are brought up with managers because of that. We make changes because of that.

But there's ... When people are filling out a survey, I do it myself, it's like, "Well, it's a five-point scale." I'm like, "Is it a 3? Is it a … ?” So you hear these thoughts out loud if you can record them in that moment. So I think there's a beautiful opportunity for you to also look at, "Hey, how do we bring surveys into our user science methodology, into this super power that we have?" And mixing it again with these interviews? So with live interviews or remote interviews. Just that human understanding, behind the surveys, it's the "what" that you got with the surveys, but it's the "why" that brings it to life.

Brent: You're right about that, Brian. I left our surveys. We use Typeform for our longer-form surveys, and Qualaroo for our short-form surveys. We use them not as much as I've used them in other companies. But surveys can be tremendously valuable.

Brian: One other tool out there which most companies use ... I think most companies use ... but I'm not sure if they're using it correctly all the time ... is the NPS Survey. So everyone seems to love it. I'm not sure if ... The interesting thing about NPS for me is it's a moment in time as opposed to, "Hey, doing a diary study. We're learning over time." And XO Group has great examples there. You can think with a product like Amazon Echo. Like you buy it. You start using it. You're asking for weather every single second or play a song. And then over time, your usage changes. Same with NPS surveys. It's like you're just capturing one moment in time. You want to look over time to really understand what your users are thinking, how their expectations are changing.

Brent: We could probably do an entire session on NPS.

Suzanne: Well, you know, hearing you say that, I'm reminded of the anecdote, Brent, that you shared before in your talk, of your time at Udacity. And this idea that if we rely on surveys sort of at the beginning and at the end, we miss a lot of the why that happens in the middle ... Is that something that you can elaborate on for our listeners here?

Brent: Sure. There's a tool that I love called the Diary Study, and I've used it several times now. What a diary study is, is you enroll a small number of people, say 10 to 20, and you agree to check in with them over a period of time: 3, 6, 12 months. It's quite effortful. You set this up because you want to understand how your product fits into a person's life. So it's not about your product per se, but it's actually about the person's life, with your product being secondary in their life.

So you might say, "Hey," at The Knot, "we want to talk to people who recently got engaged. We want to learn about what their biggest needs, anxieties, challenges are that week. What wedding related tasks they did that week. What digital tools they're using and how do they choose to use them."

If we ask those same questions every two weeks, every two weeks along their 12-month, 14-month, wedding planning journey, you get to understand how they're feeling, what their triggers are, what their unmet needs are, as well as how do they select different products, and what products they're using. In it, we've discovered for example, that wedding planning is primarily done with Google Docs and paper. Our competition is Google Docs and paper. We have to be better than that.

Suzanne: Yeah.

Brent: Google Docs and paper are pretty good. But they're not wedding specific. So how can we build something that fits in people's lives? It serves as a foundation for the work that we do. We use this at Udacity to figure out how do we build an offering in classes so that people are willing to do hard work of learning, putting 5 to 10 hours a weekend. We do this on The Bump, figuring out what are people's pregnancy journeys, and what are the different needs and pain points. That's a fun one because there's a consistency of what people are going through week by week in their pregnancy journey.

Suzanne: Yeah. I think what really resonated for me when you shared the example is the reason that people give for leaving isn't always the real reason for leaving. And so if we rely on ... and it's qualitative feedback, but if we rely solely on qualitative feedback at a single point in the journey, then we may not be getting the true story, which in the Udacity example you shared is, you know, "I felt guilty that I wasn't putting in the effort, and then it stopped making sense for me to be paying for this platform," which really isn't the same type of insight as "It cost too much money." But if you as the product manager, take away that insight, then you go down this rabbit hole of AB testing different pricing strategies and really chasing the wrong type of change.

Brent: In our product community, we want to build amazing products that change the world. That's what's lights the fire in our bellies. So when we work on solving the wrong problem, and we put energy into something that doesn't move the needle for our users or our metrics, gosh, that's wasted opportunity. It's not what we wake up to do in the morning. So the better we can understand our users, a greater percentage of our time is spent solving the right problems as effectively as possible. You know, we're making the world a better place as they say, and that's why user science is such a critical toolkit.

Suzanne: Yeah, I think you very much live into that mission around building better products and building better product people. You talked a little bit before about having this toolkit, having it be standardized, having it be trained across the entire organization. I'm with you as an advocate for you can't just train the product managers if the UX designers and the engineers and the marketers are all operating on their own process.

I know at XO Group, there's this product university. You put all of your employees through training. Can you tell us a little bit about how that works starting from day one?

Brent: Sure. I run the product organization, which includes product managers, product designers, product user researchers. We also have other people who help build the product very closely: our product marketers and our engineers. And the way that we're set up is we have dedicated squads, and teams focused on solving a problem. That team has a product manager, product designer, often a product marketing manager, a tech lead and engineers in it. And I want to empower that team to be as successful as possible, and quite autonomous.

So the way to make sure that that team has what they need to be successful, to solve the most important problems is, is having the product people in there, be able to be mature enough and set up a success that I don't need to be coaching and mentoring them for them to solve the right problems. I need great people so that I can provide this autonomy.

The way that we build the great people in our organization ... Part of it is hiring, of course. And then once we get there, we put all of our product people and product adjacent people through product university. In their first three months, they in a cohort go through a two-day training, going deep into our methodologies, our philosophies, and how we work. In their individual fields, our product managers and product designers, they go through an onboarding exercise, which is a multi-day exercise of using all of our tools, launching their first UserTesting test, launching their first Optimizely test, working within our system.

So that's how we get everyone to having a baseline. We're also doing continuous learning. Every three to four weeks, we have a class, which we call product school. It's around some of the different elements of the way that we do product management. So sometimes it's about design thinking, sometimes it's about quantitative work, sometimes it's about interpersonal dynamics, communication and collaboration. We have that curriculum that we do every three to four weeks. There are materials for that. We record the videos and then when new people come in, or when anybody needs a refresher, they go back and look at those materials and the video.

So now when you combine, I've got this onboarding product university. I've got these ongoing courses that I can access. It's quite a good curriculum. To parrot that, we also have a skills matrix for people to figure out where are they in their careers, what are the things that they should know, in order to grow, but also have more autonomy, independence and scope.

The product university product school helps them get there but be also need one on one learning and customized learning. The way that we help augment that, one is there's just a very clear way of understanding what you need to do to keep growing, which is why we have these skills grids. But then we do things such as mentoring. We do something called pair pm'ing which is somewhat a pair programming, A pm will shadow another pm who is particularly good at a skill that they have. We provide learning opportunities for individual skills. I want to become a better public speaker, I want to become a better facilitator, a negotiator, et cetera.

And then we have a learning budget for each of our product people so they can keep growing in these areas. By using a skills grid and by looking at people's progress, every six months, we're showing that people are progressing their product careers and getting more skilled and we then look at our product results, and see that our squads with more autonomy are coming up with better and stronger ideas and executing better.

Brian: One thing which I ... I want to repeat everything that Brent said. But I need to focus in. So what's special about what Brent’s talking about is he's leveling a playing field, or leveling up every single new person which comes on board. What I have seen typically at organizations throughout the years is, you learn A or you learn B, or maybe it's A and D, and maybe it’s B and Q in terms of the tools or platforms that you have access to. Oftentimes, those tools and platforms are also restricted. They're not democratized throughout their organization.

So the beautiful thing about what they're doing is, they're democratizing it. They're empowering every person who joins the product group to have a similar framework, a similar language, and have the tools to succeed. You're giving them the tools to succeed and then getting out of their] way. That's beautiful. It's empowerment. What we see elsewhere is no, that doesn't always happen. I don't have access to Optimizely or Adobe Target. That's some other group. That scares me.

I think that's the difference between a group like Brent's and this idea of user science being a superpower and putting it into reality versus people talk about this all the time. It's like, "Yeah, we have that, we have that," all these things are great. But they're not actually implementing them together and giving an empowering everyone to use them. What's special is, you know, it's a level playing field. You've empowered everyone and that can enable everyone, the whole squad, to work together to create these great products.

Brent: We have a user research lead. We have products, analytics folks. We have product copywriter. We have our product specialists. But for my team of 30 product managers and 20 product designers, I've got two years of researchers. So that's why I pair that with teaching everybody on how to do these things and use these product and user science skills.

Another organization may have chosen to not teach people those skills, and had ten years of researchers. I've seen various different designs. Our approach works really well if we want to empower our teams to move quickly and have a lot of autonomy because that way they don't need to rely on a third party in an ivory tower having all the skills and tools.

Suzanne: Yeah, I mean I really only see two problems with the approach that you're presenting. The first is, anybody listening in is going to become immediately dissatisfied with their current configuration because they're going to say, "How come I don't have this team-based learning and ongoing investment opportunity?" that's problem one. Problem two is, they may all show up at your door, shortly after we air this episode looking to get into that community so you’re gonna have an influx of prospective pm's.

Brent: You're very kind. It's an intentional investment in our organization. We want to be one of the best places to do product in New York. But it is an investment and some people love to grow their careers and improve their skill set. But it's not for everyone. Not everyone wants to grow that way. And it is time and attention from our leadership team that we've chosen to put in this way. And we love the results we're getting from it, but there are other ways to invest in your people.

Suzanne: All right, given that you can't physically support training the entire 100PM community, maybe you both can help us out with a segment we like to do. It's called 'Get the Job, Learn the Job, Love the Job." And I'll just call up the question and whoever feels called to answer, we'll go from there. The first thing I want to know is, what would you tell somebody who is currently working in a PM adjacent role, be it design or marketing, user experience research ... that really wants to become a product manager? Or frankly even somebody who's not in a PM adjacent role but really wants to become a product manager, but doesn't know how to get started.

Brian: So I'd say the best way to do it, you can read everything online, you can take courses, you can knock on Brent's door and I recommend that. But the most important thing, go out and find a product manager. If you are a designer or you're in UX or you're in marketing, and you look at something new, go out, find the best product manager who you know or your friends know. And sit down and talk with them, and learn about their world. I'd say that product management isn't for everyone. It's this unique awesome field where you have an opportunity to be a superhero. You're creating these great products, which hundreds, thousands, millions of people would use. You're having this incredible impact out there. So sit down with someone and just learn about their world. Learn about their daily responsibilities. It's not all unicorns and roses all the time. There are failures there that you ought to learn from. So sit down and just ask questions.

Brent: Getting into product management, I find is not as mysterious or challenging as some portray it. To be a great product manager requires being able to be strategic, to be a strong executor, and to understand your users. I generally bucket these things. And to be strategic, which just means to decide what to work on and what direction to go in, you could do that and have that experience in any function. To be a strong executor, the way we think about it at XO Group is, communication, collaboration, and technicals and quality and getting results. You can also do that in any function. So great account managers and finance folks and designers, engineers can do strategy and can be great executors.

Learning user science, how to really understand your users, what are the tools of that trade, is something that is specific to the domain of product management or product professionals like UX researchers. I'd encourage someone to practice learning that skill and then pretend to be a product manager. If you were the product manager of Snap or Twitter or Facebook or Open Table or something crazy that you already use in your daily life, how would you change it? It is so easy to go into User Testing and come up with a hypothesis and test an existing product. And it's a fun thing to do.

For anyone listening to this, they could go into 100PM the site and say, "How might I change the site to get more people to look at it?" Run visitors through the site and see are they surprised by what they're finding? Are the viewers and visitors understanding the right information, taking the right action? Getting started is not that hard.

Suzanne: Actually, I love that advice in particular because you know, we've been ... Our platform's been live a little over a year. We're at the part where we're exploring some usability improvements of our own. I thought it was a project I was going to have to create time for. But we're just going to crowdsource it. To the aspiring PM's listening in, say, go to UserTesting, run our site through some tests and then come and tell me how we can make it a better experience going forward.

Brent: I think the first test is free. Is that right?

Brian: We'll work out something. I love that idea. To Brent's point. It's not rocket science but this is like to get the level that we're talking about, it takes time and effort. So just start. Put something out there. Lots of these platforms have some sort of free trial. So just getting familiar with it and getting in that mindset, is a great step forward.

Suzanne: Brian, in your answer before, you talked about it sort of not all being unicorns and roses and that there are hard aspects to product management. What have you seen as typical pitfalls or places where pm's fall down in practice because it is harder to do it than just to talk about doing it?

Brian: Sure. I think the main places where I've seen people fall or trip, they often get back up which is exciting. But fall and trip, is they're not thinking about it holistically. The best example is, they're building something for themselves because they think they know what the user wants. That's not understanding your user. That's not human insights. That's not human understanding. That's you understanding. You're building something for you versus the world out there. That is the biggest pitfall we see. And that's why we're such proponents of what Brent's talking about, this idea of continuous customer learning, continuous human understanding. If you're not doing that, you might get lucky. You might build an exceptional product. But it's "might" and as I've seen working with product managers, might is not acceptable. It's like "We must do this" and this is they own the P&L or they own ... like they are very adamant about their product winning, succeeding. So you want to be as confident as possible, so where they fail is cutting a corner here and there, "Oh, we have to release this just because of x or y" which seems somewhat extraneous. It's no, there's a certain way to do this which is going to give you more confidence and that you're going to build the right product and that product is going to succeed.

Brent: Building on what Brian said, talking about pitfalls of product managers, one of the biggest things I see are folks who are output oriented instead outcome oriented. When you're outcome oriented, you say, "How do I get to the right results?" as quickly as possible, as successful as possible. When you're output oriented, you say, "How do I build it and launch it?' When you're output oriented, you don't scrutinize to make sure you're being as successful as you can at every step. You may not think to shortcut, "How do I get a signal faster? Or how do I optimize where I'm going?" And so that’s something can be a difference between a product manager whose goal is to get to release, which is an output versus someone who's a product manager who says, "I want to double conversion or double our growth rate." And that person is outcome oriented. It's a mindset change.

Suzanne: What about why you love it? I mean, both of you have been in this field a long time. You're advocates for the development of product people and product management as a discipline. Like why even be in this role?

Brian: So I ... to be honest, I'm a marketer but I think that I love product managers more than other types of roles out there because at the end of the day, they're trying to get that complete picture. They're using every single opportunity to understand and I viewed in my career that I haven't always done that. I thought I was the smartest marketer in the room for years because I was so data-metrics oriented. And I have on my resume, I'm a data oriented metrics, blah blah blah, and it was all about the numbers, and I was proud of my use of pivot tables and Excel to find an answer.

What I realized is I have a limited point of view. And when I talk to really smart product managers, product leaders like Brent, there's a more global meta point of view about how to launch something properly. And we live in this incredible world right now where we have rising consumer expectations, because we're all using amazing new apps which come out of nowhere like Snap or Instagram like Slack like Netflix keeps on raising the bar, Amazon ...

So you have these raising consumer expectations, or customer expectations and the product leaders have an opportunity to go beyond as opposed to, "Hey maybe how I did things in the past," it was a limited point of view. I'm just trying to tweak this. No, product managers are thinking, "Well, wait, everyone now has an Echo device. How are we going to change our product to work well with that?" And that's a bigger point of view. So that's what I love about product managers and product in general. You have an opportunity to just create something which no one's thought of before and get adoption by millions of people.

Suzanne: Are we ever going to convert you, Brian, or are you just firmly planted in the marketing world?

Brian: So maybe secretly I’ve been studying Brent’s processes and I need to understand all these platforms and then maybe, maybe. No, I truthfully love marketing. And if ever anyone wants to talk marketing, I'll create a 100 marketing podcast also.

Suzanne: Start with ten, take my word for it. 100 is a lot.

Brent: Brian is very good at what he does.

Brian: Thank you.

Brent: Why do I love product management? Gosh, we're an idealistic bunch. We want to change the world. In a world with where Marc Andreessen says, "Software's eating the world," we are changing industries, changing the way that we live. Product managers by being the facilitators and in the heart of what companies build, what experiences we build, we're the ones helping figure out how to change the world. I've been very lucky. I've worked in education, healthcare, personal finance, and now weddings. And as at Mint, in personal finance, we help people understand their financial situations. Are they safe? Are they well set up? We help reduce anxiety and help people make better decisions with their money and just be more aware of their money. That felt great. We changed what people did.

When I was at Chegg, we launched a textbook rental business that has changed how people buy textbooks and spend less money on textbooks. And when we launched a digital textbook, we changed how people studies so they spend less time to get more effective outcomes.

At Udacity we figured out how to make high quality effective online education in technical fields. Hadn't been done that much before, and now we are finding people going through a one to three month program at Udacity increase their earning potential and their actual earnings by $40,000. It's not a happen one time. It happens for a lot of people. We're changing these people's lives.

Here at The Knot, we're helping people celebrate, enjoy, navigate life's most important moments together. Planning your wedding is really crazy, stressful, amazing, and we're helping people get through it. Helping couples stay stronger together. Knock wood we're going to lower the divorce rate by reducing stress on one of life's stressful most important events. I mean, that's changing the world.

Brian: Brent and product managers often change the world. Like looking at Mint, this awesome company from years ago? Now we have like robo advisors doing all this great work for you, looking at all this data. But Mint was this first platform where it's like, "Wait, I can get this overview of how I'm doing." That's new, that's different. I bet when Brent's kids grow up, they will not own a textbook. Chegg changed how people think about education.

Udacity, same thing. The idea that we're all going to these four-year schools, I might find value in it, but not everyone finds value in it. And companies like Lynda which I was at. Udacity, Udemy, other platforms, they're changing the way education happens in this world. So you ask me if I'll ever be a product manager? Maybe, maybe not. But I'm an entrepreneur at heart. And the idea of an entrepreneur is very much the mindset of a product manager. We're out there to change the world. So that's what's really exciting.

Suzanne: Beautifully said. I especially love Brent that you are also helping to change and shift the landscape of what's happening in tech and in product in New York. As you know, the 100 Product Managers podcast will be coming to New York very soon. So we're going to circle back to XO Group, hear from some of your great product managers I hope and really start to shine a light on some of the cool stuff that's happening out there.

Brent: It's wonderful to see what 100 Product Managers is doing for the community. Thank you.

Suzanne: Brian, Brent, thank you so much for being part of our project.

Brian: Thank you.

Brent: What a pleasure.