Check This Table Before You Build the A/B Test

Here is a sprint I've watched get burned more than once, and it usually starts with a ticket rather than a bad idea.

Someone asks for an A/B test. You wire up the feature flag, split the traffic, build the variant, add the events, ship it. Six weeks later the readout comes back "no significant difference," everyone shrugs, and the variant gets deleted. The work was real. The answer was never obtainable with the traffic that page gets — and that was knowable on day one, in about thirty seconds, before a single line of test code existed.

I'm a growth person, not an engineer, but this is the part of my job that most directly wastes engineering time, so it's the part worth handing over. Before I run any experiment I calculate the required sample size — not as a formality, as a feasibility check. At least 30% of the test ideas I've seen would have required six months to run properly given the available traffic. Knowing that upfront saves time, money, and false confidence.

So here's the table. Screenshot it, put it in the ticket template, and make "which cell is this?" the first question asked in refinement.

The sample size table most calculators won't show you

Every sample size calculator asks for four inputs and returns one number. That's useful when you already know your baseline. It's useless for the question that actually comes first: is this test worth planning at all?

Each cell is the visitors needed per variant at 95% confidence and 80% power, for a given baseline conversion rate and a given relative minimum detectable effect (MDE).

Three things to read off it before you build anything.

Double every number for a two-arm test. These are per-variant figures. A control and one variant at a 3% baseline chasing a 10% lift needs 53,000 each — 106,000 total. This single misreading is the most common planning error I see, and it's why so many tests get called at roughly half the power their owners believe they have.

The relationship is quadratic, not linear. Halving your MDE roughly quadruples the sample size. Look across the 3% row: 53,000 at a 10% MDE becomes 208,000 at a 5% MDE. Same test, same page, same traffic — asking for twice the sensitivity costs four times the visitors.

Low baselines are brutal. A 1% baseline needs roughly eleven times the traffic of a 10% baseline for the same relative lift. If you're optimizing a low-converting flow, your MDE has to be large or the test will never finish.

If your traffic can't support the cell you land in, the answer isn't to run the test anyway. It's to change the MDE, change the page, or change the method.

Why an underpowered test is worse than no test

An underpowered test doesn't have enough data to reliably detect the effect you're looking for — even when that effect is real.

Say you're testing a new product page design. The new design genuinely improves conversion by 8% relative. But the test only reached 40% of the required sample size before someone called it. It shows no significant result. You kill the variant. You just deleted a real winner.

That's a Type II error — a false negative. And unlike a false positive (shipping a winner that wasn't), a false negative is invisible. Nobody files a bug for it. Nobody knows what was lost.

The rate of false negatives is controlled by statistical power. At 80% power and your required sample size, you'll correctly detect a true effect 80% of the time. At 40% of the required sample size, power falls to roughly 40–45%. You are worse than a coin flip at finding effects that are genuinely there.

This is the argument for treating the sample size calculation as a build gate rather than a reporting detail. It's the cheapest check in the whole pipeline and it runs before any code.

The four inputs, and how each one gets fumbled

Every credible calculator — Optimizely's, Evan Miller's, AB Testguide — needs exactly four inputs. Get one wrong and the whole number is wrong.

1. Baseline conversion rate. The current rate of the metric you're optimizing, on the specific page being tested — not a site-wide average. Testing checkout? Use checkout conversion rate, not overall site CVR. This is the input most often pulled from the wrong dashboard.

2. Minimum Detectable Effect (MDE). The smallest relative lift you want the test to be able to detect. Set it at the minimum lift you'd actually ship. Check whether your calculator uses relative or absolute MDE — this guide is relative throughout. Mixing the two silently changes the answer by a lot.

3. Statistical significance threshold. Almost always 95% (alpha = 0.05, two-tailed). Some teams use 90% for exploratory tests to reduce sample size. Use 99% when stakes are high — pricing tests, checkout changes on high-revenue pages.

4. Statistical power. Almost always 80% (beta = 0.20). Use 90% for high-stakes tests where missing a real effect is costly.

Worked example

New checkout page layout on an e-commerce site.

• Baseline CVR: 3.0%
• MDE: 10% relative (detect a lift from 3.0% to 3.3% or larger)
• Significance: 95% (two-tailed)
• Power: 80%

Through a standard two-proportion z-test: ~53,000 visitors per variation (~106,000 total across both arms).

Now convert to duration: divide by your weekly traffic per arm, then round up to whole weeks so you capture weekly cycles. If that lands past your planning horizon, you've just saved the sprint.

A few things that surprise people

A 5% MDE is expensive. On a 3% baseline you need roughly 208,000 visitors per variant — about 416,000 total. On a 10% baseline it drops to about 58,000 per variant. Because sample size scales quadratically with MDE, a 5% MDE costs roughly four times what a 10% MDE costs on the same page. Most teams chasing a 5% MDE don't have the traffic to resolve it and would be better served testing a bolder change.

More variants cost more than you'd guess, but not the way most people guess. Every additional variant needs its own full per-variant allocation, and multiple comparisons raise false-positive risk unless you apply a correction such as Bonferroni. Adding a third variation costs about 21% more per variant — roughly 1.8× the total traffic, not 3×.

"We'll run it for two weeks and see" isn't a plan. Duration alone tells you nothing. Two weeks on a high-traffic homepage may be well powered; the same two weeks on a 500-session-per-week page produces a fraction of the data you need. Calculate the sample size first, convert it to weeks at your traffic level, and run for that long.

Stopping early inflates false positives no matter what the p-value shows at the moment you peek. Significance isn't a fixed visitor count — it's the point at which your observed difference is unlikely to be noise given your sample size and effect.

The one-line version for your ticket template

Which cell of the table is this test in, and does our weekly traffic reach it inside the planning horizon?

If the answer is no, the useful output isn't a test. It's a bigger change worth a bigger MDE, or a different method entirely. Either way you found out before the feature flag, which is the whole point.

---

I built free tools for exactly this: a sample size calculator to find how many visitors each variant needs, and an MDE calculator to check whether the lift you're chasing is even detectable given your traffic. Both are free, no signup.

Correction (July 2026): an earlier version of this guide reported the worked-example sample sizes as per-variant figures when they were closer to the combined two-arm totals — understating the per-variant requirement by roughly half. It also overstated the cost of a third variation as 3× the per-variant sample size; the correct figure is about 21% more per variant, or roughly 1.8× total traffic. Every number here has been recalculated from the two-proportion z-test at 95% confidence and 80% power, and cross-checked against a power simulation.