A/B Testing

Harbor Coffee is testing a new app design. Half of the users see the old version, and half see the new one, chosen at random. That random coin flip is one of the most powerful ideas in all of science. It cancels out every hidden difference between the groups, so that any gap left over must come from the change.

• Half old, half new
• A random coin flip
• It cancels hidden differences

Reading an A/B test

Eight thousand users. We count the conversions in each group, and compare the rates with a proportions test. We also get a confidence interval for B. Run it. B converts better, and the p value is just under five percent, so luck is a less likely explanation, though it's not a landslide. Now we ask the business question. Is that lift big enough to be worth the cost of switching? Statistics can't decide that. You have to.

import pandas as pd
from statsmodels.stats.proportion import proportions_ztest, proportion_confint

df = pd.read_csv("ab_test.csv")
s = df.groupby("group")["converted"].agg(["sum", "count"])
print(s.assign(rate=lambda d: (d["sum"] / d["count"]).round(4)).to_string())

z, p = proportions_ztest(s["sum"].values, s["count"].values)
low, high = proportion_confint(s.loc["B", "sum"], s.loc["B", "count"])
print(f"p-value: {p:.4f}")
print(f"B's rate is between {low:.3f} and {high:.3f}")

       sum  count    rate
group
A      430   4005  0.1074
B      486   3995  0.1217
p-value: 0.0448
B's rate is between 0.112 and 0.132

Designing an experiment properly

A good test is designed before it starts. Pick one main metric. Decide the sample size. Randomise properly. Run it for full weeks, so that weekdays and weekends both count. And don't stop early. Check that the two groups really are the same size, because a mismatch is a sign of a broken experiment.

Summary