Bayesian A/B Testing

You tell the founder that the p value is point zero four. The founder asks, so what's the chance that the new version is better? And the honest answer is, a p value doesn't tell you that. It's a shame, because that's exactly what everybody wants to know. Bayesian analysis answers it directly.

• What's the chance it's better?
• A p-value doesn't say
• Bayesian analysis answers directly

Chance to beat

For a conversion rate, the Beta distribution describes what we believe about the true rate, given the wins and losses that we've seen. We draw two hundred thousand possible rates for A, and the same for B. Then we simply count how often B comes out ahead. Run it. Look at what we can now say, in plain words. There's about a ninety eight percent chance that B is better. The expected lift is about one point four points. And if we choose B and we're wrong, we'd expect to lose almost nothing. A founder can act on that directly.

import numpy as np
import pandas as pd

df = pd.read_csv("ab_test.csv")
s = df.groupby("group")["converted"].agg(["sum", "count"])

rng = np.random.default_rng(0)
a = rng.beta(1 + s.loc["A", "sum"], 1 + s.loc["A", "count"] - s.loc["A", "sum"], 200000)
b = rng.beta(1 + s.loc["B", "sum"], 1 + s.loc["B", "count"] - s.loc["B", "sum"], 200000)

print(f"Chance B beats A: {(b > a).mean():.1%}")
print(f"Expected lift: {(b - a).mean():.2%}")
print(f"Chance the lift is over 1 point: {((b - a) > 0.01).mean():.1%}")
print(f"Expected loss if we pick B: {np.maximum(a - b, 0).mean():.3%}")

Chance B beats A: 97.7%
Expected lift: 1.42%
Chance the lift is over 1 point: 72.5%
Expected loss if we pick B: 0.006%

Two ways of reading a test

Which approach is right? Both are valid. Classic tests are simple, widely understood, and give strict error rates. Bayesian results are easier to explain, and they let you state the size of the risk. Results are often very similar on large samples. And beware. Bayesian methods don't make peeking automatically safe. Choose one, and agree the rules before you start.

Summary