Innovation at Scale, #9: Testing Your Assumptions
Innovation at Scale, #9: Testing Your Assumptions
How hypothesis-driven design can accelerate your progress
Last week I shared a link to Ben Holliday's Everything is hypothesis driven design. That short post is a summary of a few key lessons from Jeff Gothelf and Josh Seiden’s book Lean UX.
This week, I want to zoom in on one key lesson from Lean UX: any business idea includes Assumptions, and these assumptions need to be turned into Hypotheses so they can be effectively tested.
Defining Assumptions
Lean UX contains some excellent tools for revealing unstated assumptions, but I typically start by using an even simpler prompt: “What would have to be true for this idea to work?”
Your answers to this question will begin as assumptions, which we can then convert to hypotheses.
For example, if you’re exploring a wearable health monitor, you’ll need to specifically note that one of your Assumptions is “People will wear the device.”
There could be lots of reasons why people would or would not wear the device, but this is the required user behavior to enable your success. A good assumption focuses on the required user actions. You should avoid assumptions like “People will think it’s fashionable,” because that might or might not be sufficient for your users to take the right actions. What if it’s fashionable but uncomfortable?
The list of Assumptions should be comprehensive — it needs to include all of the things that are truly required for this idea to work. To ensure you have covered all of the required Assumptions, you should examine what would have to be true across the 3 main categories of inquiry: Desirability, Viability, Feasibility.
The DVF Digression
I’ve yet to offer a comprehensive look at the DVF framework, but in short:
Desirability is related to users and their behaviours. Are we solving the key user problems? Do we understand their goals? Does the offering appeal to users?
Viability is about the commercial model. Do the unit economics work? What are the costs/revenue models? Can this be profitable?
Feasibility is about our capability to build this solution. Can we deliver, within our constraints of cost and time?
Turning Assumptions into Hypotheses
At this point, your Assumptions are probably statements that sound like facts. To create Hypotheses, we need to shift toward a framing based on our beliefs. “Users will wear the device” should become “We believe that users will the device.” However, this doesn’t take us far enough, because it’s not yet testable. To make a testable hypothesis, we need to be able to answer the question “How will we know if we are right?”
The next step is to include success criteria for your hypotheses. The hypothesis format in Lean UX is a good one:
We believe [this statement is true].
We will know we’re [right/wrong] when we see the following feedback from the market:
[qualitative feedback] and/or [quantitative feedback] and/or [key performance indicator change].
Prioritizing Hypotheses
Once we have these hypotheses, we need to understand How sure we are that these are true as well?” as “How impactful is it to our success if it is true or false?”
You can answer these questions by mapping these Hypotheses on a Prioritization matrix like this one:
(source: Lean UX)
Once these assumptions are mapped, you can design experiments for the upper-right quadrant to explore Hypotheses that are both Unknown and High Risk. We should define High Risk as “of highest material impact for our future success” — it doesn’t have to do with likelihood, it has to do with impact.
I recommend taking all of the hypotheses in the top right quadrant and ranking them in order of risk. This will be the prioritized risk of which hypotheses most need to test, which will help order your experiments.
(An aside: Lean UX suggests that this prioritization should happen with a list of Assumptions. I prefer to have teams move to the hypothesis framing first, to help solidify the idea that we have questions rather than answers. )
Testing Hypotheses
It’s not always easy to know the best tests to run to get strong answers for your hypotheses. Each product, and each hypothesis, will require different approaches to getting valid answers to your questions. Good experiment design is an art that requires more time to teach than a couple of paragraphs, but I’ll leave you with two principles that can help inform your experiment designs:
Look for valid responses
Every experiment can be designed to offer results that are more or less valid indicators of what you’re actually trying to measure. If you ask your best friend “Do you like this?” there’s a good chance that she will be supportive and say Yes. If you ask strangers in your target audience to actually buy something from you, this is a *much* stronger signal of your target audience’s intent to buy. Try to push further down the spectrum toward an actual commercial exchange of value when possible.
Set (the right) success criteria in advance
Success criteria, be they qualitative or quantitative, must be agreed before the experiment begins, snd they need to be set at the right level. If you say to your team “as long as 25% of people say that they’re willing to wear our device, we can consider this Hypothesis proven,” then that’s your success criterion. Sometimes teams say “We want 90% of people to be willing to wear our device,” but wishing for something to be true is not the same as making effective decisions based on that criterion. Set the threshold at the minimum level that would allow you to greenlight the project. If your threshold is 25%, then 24% is a clear “No,” not an opportunity for debate. You must go back to the drawing board.
I’ll have more to say about effective hypothesis testing in the coming weeks.
*
Next week, I’ll focus on effective decisionmaking in the digital age, including lessons from Jeff Bezos, Dave Girouard, and a related lesson from Lean UX.
Reading List
Each week, I'll include links to articles, books, or podcasts related to corporate innovation, that can help you accelerate the knowledge and progress of your teams.
*This week, two posts about possible futures, and one cautionary quote about why innovation has to go beyond predicting the future and focus on inventing it*
This post, Premonition (by Toby Shorin, Drew Austin, Kara Kittel, Kei Kreutler, & Edouard Urcades) is full of potentially-prescient predictions on what COVID-19 means for how our society and culture work; this starts from cutting-edge trends and projects outwards to a widely-distributed future.
COVID and forced experiments by Benedict Evans looks instead at what COVID-driven changes might mean for the late adopters, and how this might be what pushes their behaviors more fully online. (One concrete example: I’ve been trying to convince my parents to set up on Instacart, and it may actually work. Will they grow to prefer the convenience, or will they still prefer to go to the shop for the ritual of it, and the excuse to leave the house, when things settle down?)
Finally, an excerpt from Eric’ Ries’ forward to Lean UX on why understanding these futures is insufficient to create value for your business:
A personal note
I’m currently looking for new, interesting projects to explore, and people to meet. I’m especially interested in working with very large companies on building and leading innovation teams, and in helping startups to scale from a nascent Product function to a high-performing larger team. Please don’t hesitate to reach out to discuss either of the above.
*
A question for those who made it this far: Which concepts would you like me to explain in further detail in future issues? Thanks so much for reading.