How Much Data Do You Need?

Imagine testing whether a coin is slightly biased, by flipping it ten times. You'd learn nothing. Many business experiments are just like that. They're too small to ever detect the effect that they're looking for, and they just burn time.

• A coin, flipped ten times
• You'd learn nothing
• Many tests are too small

Power analysis

Power is the chance that your test will find a real effect. Eighty percent is the usual target. We ask how many users we need, to detect three different lifts, from a base rate of ten percent. Run it. Look at those numbers. A small lift needs a huge number of people, and a bigger lift needs far fewer. Halving the effect means four times the data. So before starting, ask what's the smallest lift worth finding. If you can't get enough users, don't run it.

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

for lift in [0.002, 0.01, 0.02]:
    effect = proportion_effectsize(0.10 + lift, 0.10)
    n = NormalIndPower().solve_power(effect, power=0.8, alpha=0.05)
    print(f"To detect {lift:.1%} lift: {n:>9,.0f} people per group")

To detect 0.2% lift:   356,326 people per group
To detect 1.0% lift:    14,744 people per group
To detect 2.0% lift:     3,835 people per group

The danger of peeking

Here's the most common mistake in A/B testing. Checking every day, and stopping as soon as it looks significant. Let's simulate that, with tests that have no real effect at all, peeking ten times each. Run it. Instead of five percent false wins, we get eighteen percent, more than three times that. Each peek is another chance to be fooled. So decide the length in advance, or use methods designed for continuous monitoring.

import numpy as np
from scipy import stats

rng = np.random.default_rng(0)
wins = 0
for _ in range(1000):
    a, b = rng.normal(0, 1, 2000), rng.normal(0, 1, 2000)
    for n in range(200, 2001, 200):
        if stats.ttest_ind(a[:n], b[:n]).pvalue < 0.05:
            wins += 1
            break

print(f"No real effect, but peeking 'won' {wins / 1000:.0%} of the time")

No real effect, but peeking 'won' 18% of the time

Testing many things

Test one thing, and you've got a five percent false alarm rate. Test twenty things, and a false win is likely. We simulate it. Run it. It's around sixty five percent. If you slice your results by region, age, and device, something will always look exciting. A simple fix is the Bonferroni correction. Divide your threshold by the number of tests. That brings it back to about five percent.

import numpy as np

rng = np.random.default_rng(0)
tests = 20
trials = 5000

false = rng.random((trials, tests)) < 0.05
print(f"At least one false win in {tests} tests: {false.any(axis=1).mean():.0%}")

strict = rng.random((trials, tests)) < 0.05 / tests
print(f"With a Bonferroni limit: {strict.any(axis=1).mean():.0%}")

At least one false win in 20 tests: 65%
With a Bonferroni limit: 5%

Summary