How to define a fail or success condition for your Lean experiment

Doing Lean experiments is a great way to rapidly and inexpensively test innovative business ideas.

Moreover, if done consistently and correctly, it helps you accumulate proprietary reusable knowledge—directly contributing to your organisation's competitive advantage.

The purpose of running Lean experiments is to learn more about critical assumptions behind your idea—whatever they might be.

In other words, a clear learning goal is always the starting point. From there we carefully design the experiment, and then conduct it.

Final stage is making sense of the data collected from the experiment.

Because our beautiful minds are susceptible to numerous cognitive biases and logical fallacies, it's immensely helpful if we define success and failure before we attempt to interpret the data.

In the guide for designing Lean experiments, I outline three ways to do so:

• Extrapolate from the business case. Start from the market size, and work your way to a metric that's appropriate to your maturity level.
• Base it on the industry benchmark. You can use data like the industry average, market leader's performance, and performance of your competitors or alternative solutions.
• Hippocratic oath. You can use your current performance as a minimum success criteria. In other words, the change you are experimenting with should produce better results than the current solution.

As you might have observed, I keep writing "fail or success criteria." That's because they are different sides of the same coin, albeit with slightly different implications.

In practice, the difference would look like this:

• Fail criterion: Less than 20% of interviewees spent more than $600 on painting their doors in the last three months.
• Success criterion: More than 20% of interviewees spent more than $600 on painting their doors in the last three months.

Setting fail criteria is more in tune with the scientific method, but is a bit more cumbersome. Setting success criteria usually seems more optimistic and natural.

I suggest you always strive for fail criteria, and go for success criteria only if you have a good reason (e.g. organisational culture where failure is tabu).

And remember that regardless of which you opt for—it always needs to be connected to the learning goal at hand. Everything starts and finishes with what do you want to learn in order to make this idea a great success.