Run the Sample-Size Math Before You A/B Test

Every strategy I wrote for one of my small apps this year had an A/B test somewhere in it. Test the homepage headline. Test the call to action. Test the share card. And every time someone actually ran the numbers, the answer came back in months. One came back at 28 weeks. Another needed more users than the whole season would bring.

The plan wasn't wrong to want evidence. It was wrong about what a split test needs.

The rule of thumb that misleads

The advice you'll find most often says you can start testing once you have about 1,000 visitors a week. It's a reasonable starting point for a high-converting page. It's badly wrong for most of what small products care about, because traffic is only one of three numbers that decide whether a test can finish:

• Your baseline rate for the action you're testing.
• The smallest lift you'd actually act on.
• Weekly users reaching the step, not sessions.

The formula

For the usual 95% confidence and 80% power, each arm of a two-variant test needs roughly:

n = 7.85 × [p₁(1 − p₁) + p₂(1 − p₂)] ÷ (p₂ − p₁)²

where p₁ is your baseline rate and p₂ is the rate you're hoping to reach. The 7.85 is (1.96 + 0.84)², the two z-scores for those confidence and power levels. Then:

weeks = 2n ÷ weekly users

That's it. A spreadsheet cell can hold it.

Three worked rows

Here it is at 1,000 users a week, looking for a 25% relative lift, which is already a big improvement to hope for:

Same traffic, same lift, and the time to finish swings from two weeks to more than half a year. The 20% row is a high-converting step like "clicked the main button on the landing page." The 2% row is what most products actually care about: signed up, paid, came back, invited someone.

So "we have 1,000 visitors a week" happens to be fine for the bottom row and useless for the top one. Run the formula, not the rule.

What I do instead

If the answer is more than four to six weeks, or longer than the season you're in, I don't run the test. That isn't the same as giving up on evidence.

Ship one change at a time, with a dated changelog entry. Then compare matched whole weeks before and after: same weekday mix, no holidays, bot spikes removed. It's an interrupted time series, and it's far more honest than a split test that never reaches significance.

Use a control you already have. An untouched page, or a sibling product with the same audience and seasonality, gives you something to compare the change against.

Watch five real sessions. At low traffic, watching people use the thing finds the big problems faster than any test could confirm the small ones.

Save real A/B tests for a planned spike. A launch, a seasonal peak, a premiere. Run them only on a high-baseline step, where the math says you can finish inside the spike.

If you do run one

Two rules from Evan Miller's classic piece on A/B testing mistakes still matter most:

1. Fix the sample size before you start. Use the formula above.
2. Don't stop early because it looks like it's winning. Checking repeatedly and stopping at the first significant result inflates false positives badly. Committing to the sample size up front fixes it.

Before you propose the next test

Write four numbers next to it: the baseline, the minimum lift worth acting on, weekly users at that step, and the weeks it will take. If nobody can fill in the first three, the test isn't ready. If the fourth is bigger than your season, the test isn't the right tool.

For the wider practice of testing assumptions cheaply, see Experiment-Driven Development, and A/B testing beyond the basics covers how to run a test well once the math says you can.

Sources

• Evan Miller, "How Not To Run an A/B Test", on fixing the sample size in advance and the false positives that come from peeking.