Test twice, run once

You've just finished designing your lean experiment: the learning goal is clear, as well as whom are you going to learn it from. The experiment steps are outlined, as are success criteria and time boundary.

You are ready to go live! Or are you?

Measure twice, cut once, is an apt proverb for this case. To be more direct, it's better to check the experiment design twice, than to run it and quickly find out something is amiss.

Checking your experiment design isn't about getting an early insight into whatever you've put forward as your learning goal, but rather about understanding if it run as intended—mechanically speaking.

Let's say that you want to learn how would your existing customer base react to a new product. You devise a lean experiment that would basically send a new offer to 500 customers, and a regular offer to other 500 customers. You decide to measure open and conversion (i.e. a purchase) rates to determine if it's a success or failure.

Checking in the above example would refer to:

• making sure that the right people will receive the email (e.g. existing customers that have done at least one purchase, a single customer doesn't receive both offers, etc),
• making sure that all links works as intended (e.g. tracking parameters, directed at correct product pages, etc), and
• making sure that everybody in the team know how the experiment will be executed (e.g. what happens first, who sends out the emails, how to terminate the experiment if necessary, etc).

In essence, you want to check if your experiment will produce data as intended, not which data will it produce. You will analyse the latter once the experiment has been been concluded.

Mistakes happen, but it's better to catch them early. That is especially true for larger companies that have recognisable brands, where even small mistakes can harm their reputation and have financial repercussions.

So, make sure to test your lean experiment at least twice before running it.

For a complete guide on how to design, run, and learn from Lean Experiments, head to:

Step-by-step guide for designing Lean Experiments

Designing sound experiments is critical to creating valid and reusable knowledge. This guide will help you achieve that.

Bruno PešecBruno Pešec