AI Hints
Statistics & Math for ML · Lesson 123 of 138

Simpson's Paradox

When groups flip the trend.

⏱ 2 min read▶ Video at 2:27:07🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A treatment works better for young patients, and also better for old patients. But combined, it looks worse. How can that be? It's called Simpson's paradox, and it's real.

Simpson's Paradox
From the video: Simpson's Paradox
  • Better for young
  • Better for old
  • Worse overall?

Two hospitals

Treatment A was mostly given to severe cases, and treatment B mostly to mild ones. Within each group, A has the higher recovery rate. But overall, B looks better. Run it. The severity of the cases is a confounder that flips the overall comparison.

simpson.pypython
import pandas as pd

df = pd.DataFrame({
    "treatment": ["A", "A", "B", "B"],
    "severity":  ["mild", "severe", "mild", "severe"],
    "patients":  [100, 400, 400, 100],
    "recovered": [95, 280, 360, 60],
})
df["rate"] = df["recovered"] / df["patients"]
print(df[["treatment", "severity", "rate"]].to_string(index=False))

overall = df.groupby("treatment")[["recovered", "patients"]].sum()
print((overall["recovered"] / overall["patients"]).round(3).to_string())
Output
treatment severity  rate
        A     mild  0.95
        A   severe  0.70
        B     mild  0.90
        B   severe  0.60
treatment
A    0.75
B    0.84

The lesson

When groups are different sizes, and a hidden variable decides who ends up in which group, combined numbers can mislead. Always ask whether there's a variable that should be split out. In machine learning, the same thing happens when a feature behaves differently in different groups of data.

keyvalue
Within groupsA is better
CombinedB looks better
Causeunequal group mixes
Fixsplit by the confounder

Summary

Simpson's paradox shows that combining groups can reverse a trend. Look for hidden variables before you trust totals. Next, ways to draw a sample.

Key takeaways
  • Compare within groups
  • Watch the mix
  • Find the confounder
Next up: Sampling Methods — random, stratified, cluster