Skip to main content
DiscoveryIntermediate6 min read

Experiment-Driven Development

How to validate assumptions before investing development time

Most teams build features based on assumptions they've never tested. Someone important says "users want X," the team builds X, and three months later the data shows nobody uses it. Experiment-driven development is the discipline of testing assumptions before investing significant development time.

The goal is to maximize the odds that what you build will actually drive the outcomes you're chasing - for the business and for the people using the product.

Why experiments matter

  • Reduce time-to-learning - test assumptions quickly instead of guessing
  • Avoid over-investing in features that may not meet user needs
  • Separate problems from solutions - the problem worth solving (discovered through user-centered design) and the solution that works are two different questions
  • Prioritize based on evidence - build what the data supports, not what the loudest stakeholder wants

The experiment process

1. Identify business objectives

What are you trying to achieve? Expanding a customer segment? Improving retention? Increasing revenue? Every experiment should connect to a business objective - an outcome, not an output.

2. Establish a discovery cadence

Schedule discovery calls with real customers or prospects every week. Consistently document findings with a focus on actionable opportunities.

3. Surface ideas and assumptions worth testing

Use ideation techniques - crazy eights, opportunity scoring, the new venture question list below. The goal is to separate what you know from what you assume.

4. Define hypothesis, test method, and success metrics

Use this template:

We believe that [doing this] for [these people] will achieve [this outcome]. We'll know this is true when we see [qualitative or quantitative signal] that improves [this KPI].

5. Run the experiment

Focus on the smallest, cheapest way to validate. Usually you don't need to build anything in code (see MVP strategy and launch for how to take validated experiments to market):

  • Paper prototype or Figma prototype
  • Signup or landing page with engagement metrics
  • Prove demand by asking for something of value - prepayment, contact info, a commitment
  • Time-box the activity so it doesn't drag on indefinitely

Before you reach for an A/B test, run the math

Whether a split test can finish depends on three numbers together: your baseline conversion rate, the smallest lift worth acting on, and weekly users (not sessions). For 95% confidence and 80% power, each arm needs roughly:

n = 7.85 × [p₁(1 − p₁) + p₂(1 − p₂)] ÷ (p₂ − p₁)²

weeks = 2n ÷ weekly users

Worked through at 1,000 users a week, looking for a 25% relative lift:

Baseline (p₁) Target (p₂) Users per arm Weeks to finish
2% 2.5% 7.85 × (0.0196 + 0.0244) ÷ 0.005² ≈ 13,800 27.6
5% 6.25% 7.85 × (0.0475 + 0.0586) ÷ 0.0125² ≈ 5,330 10.7
20% 25% 7.85 × (0.16 + 0.1875) ÷ 0.05² ≈ 1,090 2.2

Most key actions on small products sit in the 2% to 5% row. That's why the rule of thumb "you can test once you have 1,000 visitors a week" fails: traffic alone doesn't decide it. If the answer is more than four to six weeks, or longer than your season, don't run the test.

What to do instead:

  • Ship one change at a time with a dated changelog entry, and compare matched whole weeks before and after (same weekday mix, no holidays, bot spikes removed).
  • Use an untouched page or a sibling product as a control where you have one.
  • Watch five real sessions. At low traffic, qualitative evidence moves faster than any test.
  • Save true A/B tests for a planned spike like a launch or a seasonal peak, and only on a high-baseline step.

When you do run one, fix the sample size up front and don't stop early because it looks like it's winning. Checking repeatedly inflates false positives badly (Evan Miller).

6. Document and share results

For validated ideas, link the experiment results to backlog tickets and write stories with clear acceptance criteria. For invalidated ideas, document the learning - it's just as valuable. Killing a bad idea early saves weeks of wasted development.

7. Continuously evaluate released features

Don't assume the solution is "good forever." Continually evaluate released features and prune what's not working. Solutions evolve with the product.

Principles of good experiments

Principle Why it matters
Test one thing at a time Isolate the variable so results are interpretable
Define success metrics in advance Avoid moving the goalposts after seeing results
Smallest, cheapest validation Build the minimum needed to learn - usually not code
Time-box the activity Set a deadline so experiments don't drag on
Trace everything to a customer need If you can't trace a feature back to real user evidence, question whether to build it

The culture that makes experiments work

Process alone doesn't make a team experiment-driven. Two team values do, and without them the process quietly dies:

  • Share experiments. Run hypothesis-driven experiments and share every outcome - validated or not - so the whole team owns the learning instead of one person holding the context. A result that only the PM sees is a result the team can't act on.
  • Celebrate failure. A failed experiment is an invalidated hypothesis, not a mistake to bury. The sooner a team produces failures, the sooner it steers off a doomed path. Teams that treat failed experiments as embarrassing stop running the risky ones - which are exactly the ones worth running.

These are two of the four balanced-team values. The best product teams I've worked on discovered great products piece by piece through cheap experiments, and treated the ones that didn't pan out with curiosity rather than blame.

The new venture question list

For any new product or large feature set, answer each question below. Then rate your confidence (rock-solid fact or risky assumption?) and severity (if wrong, does it sink the project?). Sort by low confidence + high severity - those are your most dangerous assumptions. Run experiments on those first.

Question
My target customer will be:
The problem my customers want to solve is:
My customer need can be solved with this product/solution:
The measurable outcome my customer wants to achieve is:
My primary customer acquisition tactic will be:
My earliest adopters will be:
I will make my money by:
My primary competition will be:
I will beat my competitors primarily because of:
My biggest risk to financial viability is:
What other assumption, if wrong, could cause failure:
What other critical risk could cause failure:

How to use this list

  1. Answer each question as best you can
  2. Add a confidence level to each answer (fact vs. assumption)
  3. Add a severity level (if you're wrong, how bad is it?)
  4. Map on a 2x2 - identify the ones that are both low confidence and high severity
  5. Run experiments on those first

Common pitfalls

  • Running experiments after the decision is made - if leadership has already decided to build it, the experiment is theater
  • Experiments without clear success criteria - "let's see what happens" isn't an experiment, it's hope
  • Over-building the test - if your experiment takes more than a week to set up, you're building a feature, not testing an assumption
  • Ignoring invalidating results - the whole point is to learn. If the experiment says no, listen
  • A/B testing without the sample-size math - at a 2% baseline and 1,000 users a week, detecting a 25% lift takes over six months. Run the formula before you build the variant

Try this today

Pick one feature your team is planning to build in the next quarter. Write a hypothesis using the template above. Then ask: what's the cheapest thing we could do in the next week to test this? If the answer is "nothing - we'd have to build the whole thing," the feature isn't well enough understood yet.

Sources

  • "Share experiments" and "celebrate failure" are two of the four Pivotal balanced-team values I coached from for four years. The broader case that continuous experimentation is how modern product teams reduce guesswork is well made in The Product Economy (Product School, 2020).
  • Evan Miller, "How Not To Run an A/B Test", on fixing the sample size in advance and the cost of peeking. The per-arm formula is the standard two-proportion approximation at 95% confidence and 80% power, where 7.85 = (1.96 + 0.84)².

Skills for this topic

AI skills you can run with Claude or Codex to put this practice to work.

Apps for this topic

Real, free tools on this site that do this work for you right now.

See this in practice

Real engagements where this practice did the work.

Want help with experiment-driven development?

I coach teams on this practice. Let's talk about your situation.

Get in touch