Inference Pitfalls
P-hacking and more.
Statistics is powerful, but it's easy to fool yourself, and others. Here are the mistakes that show up again and again, in research papers, business reports, and machine learning projects.

- Easy to fool yourself
- Common mistakes
- Avoid them
P-hacking
P hacking means trying analysis after analysis until something gives a p value under five percent, then reporting only that one. Removing some data, adding a variable, or stopping the experiment at a lucky moment. Each step seems harmless, but together they almost guarantee a false result.
| key | value |
|---|---|
| Trying many analyses | one will 'work' |
| Stopping early | at a lucky moment |
| Reporting one | hiding the rest |
| Fix | plan the analysis first |
Other traps
Correlation isn't causation, so a link doesn't prove one thing causes the other. A biased sample gives biased answers, no matter how big it is. Simpson's paradox can reverse a trend when you combine groups. And extrapolation, predicting far outside your data, is risky.
| key | value |
|---|---|
| Correlation | isn't causation |
| Biased samples | stay biased when big |
| Simpson's paradox | groups can flip a trend |
| Extrapolation | risky beyond the data |
Summary
Plan your analysis in advance, report everything, and be suspicious of results that look too good. Next, a quiz on inference.
Key takeaways- Don't p-hack
- Mind sampling bias
- Watch for confounders