Cutting Test Noise

Customer spending varies hugely. Some people spend ten dollars a month, and some spend three hundred. All that natural variation drowns out the small effect that you're trying to measure. But you know how much each customer spent before the test began. Can you use that? Yes, and it's one of the cleverest tricks in experimentation.

• Spending varies hugely
• Noise drowns small effects
• You know each person's past

Removing what you already know

We simulate four thousand customers, with a true effect of two dollars. Their spending now is strongly linked with their spending before. Cuped subtracts out the part that past spending can predict. Because this test was randomised, that can't introduce bias. It just removes noise. Run it. Look at the error. It's about half as large, for the same data. Halving the error is worth four times as many users. That can turn a month long test into a week. That's why big tech companies use this on almost every experiment.

import numpy as np

rng = np.random.default_rng(1)
n = 4000
before = rng.gamma(2, 15, n)
group = rng.integers(0, 2, n)
after = before * 0.9 + rng.normal(0, 12, n) + 2 * group

theta = np.cov(before, after)[0, 1] / before.var()
adjusted = after - theta * (before - before.mean())

for name, y in [("Plain", after), ("CUPED", adjusted)]:
    gap = y[group == 1].mean() - y[group == 0].mean()
    se = np.sqrt(y[group == 1].var() / (group == 1).sum() + y[group == 0].var() / (group == 0).sum())
    print(f"{name}: effect {gap:.2f}, error {se:.2f}")

Plain: effect 1.58, error 0.70
CUPED: effect 1.39, error 0.37

Summary