Experiments vs Pilots: Your Experiment is a Pilot. Start with an Experiment.

What’s the difference between an experiment and a pilot?

An experiment tests one risky assumption at small scale and low cost, while a pilot tests the whole solution in front of paying customers. Experiments are quick, quiet, and easy to change course on. Pilots are expensive, highly visible, and hard to turn. Run a pilot only after you have proven a compelling fit of customer, problem, and solution. Until then, you’re rehearsing on opening night.

There's nothing more exciting than trying something new. You feel the opening-night jitters; a mixture of fear and anticipation that flutters the heart and the stomach alike. In theatre, opening night is the culmination of months of hard work designed to reduce the risk of that first official performance for the public. Casting, rehearsals, understudies, set design, dress rehearsals, and finally private performances help ensure that the only question remaining on opening night is viability—will buzz and word-of-mouth keep the paying public coming for a long theatrical run. Opening night is a pilot; all the important experiments around feasibility and desirability have already been run.

Such is your project. I hear a lot of people using the word "experiment", when in fact, they are really piloting. Piloting is a timeless tactic of corporate America. Its aim is usually "proof-of concept". The typical metric for a pilot is customer uptake. Sometimes internal systems' ability to manage the new product is also measured. I can't imagine anything riskier. Why? Because you are rehearsing in front of a paying audience. If you fail, it will be headline news tomorrow. And it's expensive, and you don't learn a lot if you are measuring the complex behavior of purchase.

Don't get me wrong. There's a time for pilots, but it's only after you have proven a compelling fit of customer-problem-solution. And that's rarely at the beginning. Start with the desirability piece of the puzzle. Is there an early-adopter customer out there that is already trying to solve for the problem you have identified? What can you learn from those customers? Could you adapt their solution and offer it to a small group of similar customers? Then, do they use it? Why or why not?

Next, adjust your prototype and offer to a larger group of people—say 25 or so. What can you learn about desirability in this setting? What you learn about feasibility; it is even possible to offer this at scale? When you have those answers it comes time to start thinking about viability. Why think about it now? Because what you learned about desirability may be so compelling that the answer to "what should we be involved in" may change with new information. Likewise, feasibility: if you can show a jaw-dropping market opportunity, resources can magically appear to make the seemingly impossible happen.

The Discount Program Example

Let's say you are going to test a customer discount program with your customers. You ran some surveys and some people said that such a program would be helpful.

Do you: a) contract with a vendor and send an invite to a state's worth of customers and inform retailers of the program, or b) talk to a group of high-value customers about what they shop for and how they do it?

Correct, (b). Remember, before Angie's list was a website, it was a stapled list that Angie schlepped all over town. She learned a ton by doing it, and it lit the path to subsequent success in the scaled online world.

Brainstorm the biggest risky assumptions facing your project. Think in terms of desirability, feasibility, and viability. Rank them. Hint: desirability risks are usually first. Then feasibility. Then viability. Now test your top risk in the simplest way possible. Remember, these experiments aren't testing your solution, it's testing the behaviors that must be present for your customers to derive value from a solution. That's why pilots aren't right at this time; measuring sales doesn't tell you a thing about the why or why not of the purchase.

Looking at the Discount Program, after brainstorming risky assumptions, you came up with a couple of doozies. First, "customer will accept a discount/coupon". Second, "customer will use the coupon to buy something".

Now experiment. Send out your coupons. Track email open rates. Now measure redemptions. Talk to people that didn't open the email. Talk to people that didn't redeem. Talk to people that did redeem. Learn everything that you can. Then re-evaluate the offer. Discuss viability in the context of the customer response. Given the customer response, should you be doing this? Is it worth the reputational risks? Could your company be good at it? Is it a good use of finite company resources?

This is just the beginning. But there is an ending. One of two things will happen. You'll be unable to generate compelling evidence to move forward, or you will get good evidence. If customers don't want it, that's great—do something else with those resources! The only failures occur when we charge ahead without evidence. The important thing is that you took the risk out of your offer long before the bright lights of opening night.

Frequently asked questions

When should I run a pilot instead of an experiment?

Run a pilot only after you have proven a compelling fit of customer, problem, and solution — and that is rarely at the beginning. A pilot's usual aim is proof-of-concept, measured by customer uptake and sometimes by whether internal systems can handle the new product. Prove desirability first, then feasibility, then viability.

Should I test desirability, feasibility, or viability first?

Test desirability first, almost every time — find out whether anyone actually wants this. Brainstorm the riskiest assumptions facing your project, sort them into desirability, feasibility, and viability, then rank them. Test the top risk in the simplest way you can. What you learn about desirability can be compelling enough to change what your company should be involved in at all.

Why is it risky to run a pilot too early?

Piloting too early puts you on stage rehearsing in front of a paying audience. Pilots are large, expensive, and highly visible, so a stumble is headline news tomorrow, and changing course gets hard once stakeholders are watching. There is a time for pilots — after the fit is proven, not before.

What does a good first experiment look like?

A good first experiment is small, cheap, and aimed at one behavior instead of the whole solution. Take a customer discount program: rather than contracting a vendor and inviting a state's worth of customers, talk to a group of high-value customers about what they shop for and how they do it. Then send coupons to a small group, track open rates and redemptions, and talk to the people who didn't open, didn't redeem, and did redeem. Every one of those conversations costs less than a rollout.

Do experiments test the solution or the customer?

Experiments test the behaviors that must be present for customers to derive value from a solution, not the solution itself. That is why a pilot is the wrong tool this early: it measures the purchase and misses the why. Watch what customers do, then ask them why they did it.

What counts as failure in an experiment?

The only failure is charging ahead without evidence. Every experiment ends one of two ways: you can't generate compelling evidence to move forward, or you get good evidence. If customers don't want it, that's great news — put those resources somewhere better.