AI Hints
Statistics & Math for ML · Lesson 119 of 138

Practice: An Experiment End to End

From question to answer.

⏱ 2 min read▶ Video at 2:22:18🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

The weekend difference looks real, but is it? Let's answer properly, with a test, an interval, and an effect size. Three numbers that together tell the full story.

Practice: An Experiment End to End
From the video: Practice: An Experiment End to End
  • Is the difference real?
  • A test, an interval, an effect size
  • The full story

Test, interval, effect

First, a t test with unequal spreads. Then a bootstrap interval for the difference in means. And finally, Cohen's d for the effect size. Run it. The p value is tiny, the interval is well above zero, and d is about zero point seven, a medium to large effect. So weekend customers really do spend more, by about two and a half dollars.

experiment.pypython
import numpy as np
from scipy.stats import ttest_ind
from cafe import df

we = df[df["day"] == "weekend"]["spend"].to_numpy()
wd = df[df["day"] == "weekday"]["spend"].to_numpy()
print(f"p = {ttest_ind(we, wd, equal_var=False).pvalue:.1e}")

rng = np.random.default_rng(0)
diffs = [rng.choice(we, len(we)).mean() - rng.choice(wd, len(wd)).mean() for _ in range(5000)]
print("95% CI for the difference:", np.percentile(diffs, [2.5, 97.5]).round(2))
d = (we.mean() - wd.mean()) / np.sqrt((we.var(ddof=1) + wd.var(ddof=1)) / 2)
print(f"Cohen's d = {d:.2f}")
Output
p = 2.5e-14
95% CI for the difference: [1.93 3.16]
Cohen's d = 0.73
Also uses cafe.py
import numpy as np
import pandas as pd

rng = np.random.default_rng(7)
n = 600
day = rng.choice(["weekday", "weekend"], n, p=[0.7, 0.3])
temp = rng.uniform(5, 30, n)
spend = rng.lognormal(1.8, 0.45, n) + np.where(day == "weekend", 2.0, 0) + 0.05 * temp
df = pd.DataFrame({"day": day, "temp": temp.round(1), "spend": spend.round(2)})

Reporting it

A good report says what you found in plain words. Weekend customers spend about two and a half dollars more, somewhere between two and three dollars, and it's very unlikely to be chance. Then it adds the caveats. This is observational data, so the cause could be different customers, not the day itself.

keyvalue
Effectabout +$2.50 at weekends
Intervala range, not one number
Evidencetiny p-value
Caveatobservational, not causal

Summary

A full answer combines a test, an interval, and an effect size, plus honest caveats. Next, building a classifier from scratch.

Key takeaways
  • t-test
  • Bootstrap interval
  • Effect size
Next up: Practice: A Model from Scratch — logistic regression by hand