AI Hints
Home›Statistics & Math for ML Tutorial›Percentiles and Outliers
Statistics & Math for ML · Lesson 18 of 138

Finding Outliers

The IQR rule.

⏱ 2 min read▶ Video at 19:11🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

An outlier is a value far from the rest. It might be a typing mistake, or it might be the most interesting point in the data. Either way, you need to find it first. A simple rule uses the quartiles.

Finding Outliers
From the video: Finding Outliers
  • Far from the rest
  • A mistake, or a discovery?
  • Find it first

The IQR rule

The inter quartile range, or IQR, is Q three minus Q one, the width of the box. Anything more than one and a half IQRs below Q one, or above Q three, is called an outlier. Run it. Two outliers. The typing mistake, nine hundred and eighty, and a suspiciously low two.

iqr.pypython
import numpy as np

ages = np.array([23, 25, 31, 35, 29, 41, 38, 27, 980, 33, 2, 30])

q1, q3 = np.percentile(ages, [25, 75])
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr

print("keep between", low, "and", high)
print("outliers:", ages[(ages < low) | (ages > high)])
Output
keep between 12.625 and 49.625
outliers: [980   2]

What to do with outliers

Never delete outliers automatically. First, ask why. An age of nine hundred and eighty is clearly an error, so fix it or remove it. But a huge purchase might be fraud, which is exactly what you're looking for. Some models, like linear regression, are very sensitive to outliers. Others, like decision trees, barely care.

keyvalue
IQRQ3 − Q1
Outlierbeyond 1.5 × IQR
Firstask why
Errorsfix or remove

Summary

The IQR rule flags values more than one and a half IQRs outside the box. Then you decide what to do with them. Next, histograms, and the shape of data.

Key takeaways
  • IQR = Q3 − Q1
  • 1.5 × IQR fences
  • Ask why before deleting
Next up: Histograms — the shape of data