Have you ever wondered how to turn experiments into customer delight?
20 Feb 2022
The first rule of skiing is falling is good. Falling means you’re learning. Falling also means you’re trying and pushing your skills to improve. Similar statements can be applied to product management. Just as learning to ski is tricky, so is learning to accept that a majority of product experiments will not go as planned. What’s important, in skiing and in product management, getting up and trying again. As a Product leader (or ski instructor), you are responsible for creating a space where it is safe to experiment, learn, and try again (yes, I’m a fan of psychological safety).
While some individuals create boards in Pinterest of their dreams and aspirations, I save articles (see post #2). The following articles reflect the approach I am working on with my team as we increase the amount of experimentation in product. To learn more about the practices I am applying, register for the March cohort of People-First Product Leadership.
Optimize to be Wrong, not Right
Barry O’Reilly always has posts that make one stop and think differently, such is the power of unlearning. His themes stand the test of time, as seen from this article dating back to 2017. Instead of seeking to always be right, gain strength from identifying and removing weaknesses right at the start of a project. The end result will be better due to the early stage learnings.
People tend to forget that the measure of progress for innovation is not how many “good” ideas you validate, but actually how many you invalidate quickly and inexpensively.
…making multiple small investment to experiment—to dig deeper and find out whether your hypothesis is true or false (or somewhere in between)—means at worst you’ve lost a small amount of time, money and resources but learned a lot. From there, you can synthesize what you have learned and feed it forward into your next round of experiments while continuing to run relatively inexpensive experiments with small risk that cover a broad base of hypotheses.
The method Barry recommends is called Set-Based Concurrent Engineering (SBCE) or set based design, in which a team considers multiple ideas rather than a single idea. With each experiment, the team builds up knowledge that allows them to narrow down ideas through eliminating those with unsatisfactory outcomes. The earlier you experiment and obtain learning, the lower the cost to the business in time, energy, and resources.
If you only explore one idea (called point based design), there is less chance for learning. Then when new information arises, you have to start all over again - maybe even go back to the start. With set based design, learning is gained earlier and may require stepping back one space vs. going all the way back to the beginning.
Do Things that Don't Scale
What do you think of when someone says “wizard of oz”? Do you think of the movie?
If you’re a product manager who is a fan of experimentation, running a wizard of oz experiment might come to mind.
The Wizard of Oz method is a process that allows a user to interact with an interface without knowing that the responses are being generated by a human rather than a computer.
In this article by Paul Graham, the founder of Y Combinator, shares advice given to would-be founders - which reminded me of product leaders running diverse teams. He recommends using manual (human-powered) efforts - aka running “wizard of oz” tests - early on. Why?
At the early stage you have less to lose.
You’ll learn directly from the end users.
Insight, both good and bad, will inform how you will do things differently when your audience grows and you gradually switch to less manual methods.
Yes, the idea of an experiment going wrong can be scary, especially when one is an early stage startup. Yet, that’s the safest time to fail. You can launch faster, you can iterate on feedback quicker, and you’ll know what to build as you’ll have the “muscle memory” of doing it yourself.
Consider how long your team can explore an idea manually, and then push them to see if they can do it for a bit longer. Eventually bottlenecks will arise - and that’s a good thing as it means an increasing number of customers are using your offering. When that happens, it’s a signal to start automating the bottlenecks.
Your first users should feel that signing up with you was one of the best choices they ever made. And you in turn should be racking your brains to think of new ways to delight them. …And incidentally, when it does, you'll find that delighting customers scales better than you expected. Partly because you can usually find ways to make anything scale more than you would have predicted, and partly because delighting customers will by then have permeated your culture.
Don’t listen to the naysayers who bring up established players. They have simply forgotten the rough starts all companies, even the popular ones today, have faced. But do listen to your team. Ensure they don’t get discouraged when things go wrong. Instead remind them to focus on the potential, the savings of catching gotchas early on, and what will happen when an experiment goes right and customers experience delight.
Eigenquestions: The Art of Framing Problems
Ok, we now know things can, and will go, wrong teams practice experimentation. We know it’s best to experiment early and often. We know serving customers manually in our early stage experiments provides signals which inform subsequent automation and enable growth. How do we determine what to experiment on? Hello to framing.
Framing is the process of breaking down a problem into a set of choices, trade offs, and options that enable a team to make a call and move forward.
In this article by the founder of Coda, Shishir Mehrotra, he shares how to approach framing. The three techniques he recommends are
Technique 1: Eigenquestions
"Eigenquestion" is a made-up-word from linear algebra. It is the question where, if answered, it likely answers the subsequent questions as well.
Technique 2: The Power of a Great Diagram
Framing is both a right brained and a left brained activity. Using diagrams with a consistent pattern and coloring system helps create shared understanding.
Technique 3: Framing with your Team
Framing is not meant to be done alone as great ideas can come from anywhere. Great framers find ways to incorporate diverse perspectives.
Apply these techniques to help your team work together more effectively. And, to encourage the right environment for your team to approach framing, he wraps up the article with a few tips.
Start with a positive mindset. In Japan, it's good luck to find a problems as it's a chance to improve something. This positive context can be highly motivating. It moves us from a defensive and fearful attitude to an optimistic and proactive one.
Search for the underlying logic of the problem. Be both a great anthropologist (capture the details and summarize a discussion) and a great detective (what's the real issue here) by asking “why” until you get to the root of an issue.
Acknowledge it when you're stuck. If you get stuck on a problem, recognize the way in which you are stuck. Get perspective through simplifying. Peel back the layers until you reach the core of the problem.
Hypothesis-Driven Development (HDD)
When working through a problem you are dealing with uncertainty. To help navigate, you should have explicit, testable ideas and focus on disproving them quickly so you can turn your attention to exploring a potentially “better idea”.
The author labels this approach as HDD and uses the following diagram to define the conditions as pivot (disproven) or persevere (proven). What’s even better, as you progress an idea that appears to have potential, the diagram still applies.
For determing what to test, the author recommends using the ‘double diamond’ framing of ‘right problem’ vs. ‘right solution’ which I also use for People-First Product Leadership. By applying the double diamond you can address three big questions:
How do you test that you’ve found the ‘Right Problem’?
How do you test that you’ve found demand and have the ‘Right Solution’?
How do you test that you’ve designed the ‘Right Solution’?
The only way to find out is, first to go out and talk to customers and find out! Second, you need to ask non-leading questions. The goal being to determine ‘Who is our customer, and what’s important to them?’.
We’ll wrap by making the seemingly impossible, possible. Here’s a way to make meetings fun!
Ask This Question at Your Next Meeting
What if meetings accelerated the progress they now stifle? Rethink your meeting approach by asking the question: What are you stuck on?
Incorporating this question into team meetings introduces a collaborative forum for creative problem-solving, and provides a wealth of compelling benefits [, including the following:
Inviting reflection to step back and think deeply about what they are really trying to achieve, as well as the barriers getting in the way.
Communicating that feeling challenged is not an experience that needs to be hidden or feared. It’s expected. That perspective helps to foster resilience.
Introducing an opportunity for colleagues to provide input — allowing teammates to mentor each other, harness one another’s strengths, discover untapped expertise and nurture greater collaboration.
Motivating growth as asking the question offers leaders a tool for communicating that pushing yourself and finding your limits is desirable, without coming across as preachy or out-of-touch.