AI Hints
Home›Statistics & Math for ML Tutorial›Effect Size and Pitfalls
Statistics & Math for ML · Lesson 65 of 138

Inference Pitfalls

P-hacking and more.

⏱ 2 min read▶ Video at 1:16:06🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Statistics is powerful, but it's easy to fool yourself, and others. Here are the mistakes that show up again and again, in research papers, business reports, and machine learning projects.

Inference Pitfalls
From the video: Inference Pitfalls
  • Easy to fool yourself
  • Common mistakes
  • Avoid them

P-hacking

P hacking means trying analysis after analysis until something gives a p value under five percent, then reporting only that one. Removing some data, adding a variable, or stopping the experiment at a lucky moment. Each step seems harmless, but together they almost guarantee a false result.

keyvalue
Trying many analysesone will 'work'
Stopping earlyat a lucky moment
Reporting onehiding the rest
Fixplan the analysis first

Other traps

Correlation isn't causation, so a link doesn't prove one thing causes the other. A biased sample gives biased answers, no matter how big it is. Simpson's paradox can reverse a trend when you combine groups. And extrapolation, predicting far outside your data, is risky.

keyvalue
Correlationisn't causation
Biased samplesstay biased when big
Simpson's paradoxgroups can flip a trend
Extrapolationrisky beyond the data

Summary

Plan your analysis in advance, report everything, and be suspicious of results that look too good. Next, a quiz on inference.

Key takeaways
  • Don't p-hack
  • Mind sampling bias
  • Watch for confounders
Next up: Inference Quiz — test yourself