A/B Testing for Product Managers: Beyond the Basics

Why Experiments Fail Without Strategy

I've seen PMs run dozens of experiments that never changed a roadmap decision. Not because the results were wrong, but because the test wasn't tied to a strategic bet.

At Kaiser, my role as a product coach often started with reframing: "This isn't a test to see if blue beats green. It's a bet to see if simplifying the way to get an insurance coverage estimate reduces call center load and improves member satisfaction." That shift made experiments easier to prioritize and much harder to ignore.

When you design an A/B test, start by answering:

• Which business outcome is at stake?
• What decision will this result inform?
• What's the opportunity cost of running this instead of something else?

If you can't answer all three, you are probably testing too low in the value chain.

Building a Test That Teaches You Something

The core structure is simple, but skipping a step will cost you:

1. Write a clear hypothesis - "If we X, then Y will happen because Z."
2. Pre-commit to metrics - One primary metric for success, plus guardrails so you don't win in one area and damage another.
3. Set your sample and timeline - Decide in advance to avoid peeking and biasing results.
4. Document context - Future teams should understand why you ran this and what you learned.

At a civic-tech startup, we tested a new municipal service onboarding flow. We thought our simplified version would get more first-time users to complete sign-up. It didn't. Completion dropped by 12 percent. The cause? Our simplification hid key eligibility info, so people bailed mid-way. That was humbling and valuable.

The goal isn't to win every test. It's to learn enough to make better calls next time.

When Not to A/B Test

Not every question needs a split test. Skip it if:

• The change is irreversible or has ethical concerns.
• You don't have enough traffic to reach significance in a reasonable time.
• The result wouldn't change your decision.

Tie this back to opportunity cost. Every experiment you run is time you aren't testing something else. Make it count.

Bringing Stakeholders Along

A well-run test still fails if no one believes the results. Make your findings stick by:

• Framing them in terms leaders care about, such as cost savings, revenue, or risk reduction.
• Using visuals that show impact at a glance.
• Sharing both surprises and confirmations. Your credibility grows when you admit when you were wrong.

Want my full Experimentation Toolkit, including templates and the decks behind these examples? Download it here and start running tests that matter.