Outliers

Look at the average occupancy in our housing data. Most districts have two or three people per home. But one district has over twelve hundred people per home. Is that a typo? A prison? A dormitory? Extreme values like this are called outliers, and they can pull a linear model badly out of shape.

• Most homes hold two or three people
• One district shows over a thousand
• Real, or a mistake?

Finding outliers

A standard way to find them uses the middle of the data. Q one and Q three mark the middle fifty percent. The gap between them is called the I Q R. Anything far above Q three, usually one and a half gaps, is flagged. Run it. The normal range ends at under five people. Over seven hundred rows are above that, and the largest are enormous. They are clearly not normal homes.

import numpy as np
from sklearn.datasets import fetch_california_housing

df = fetch_california_housing(as_frame=True).frame
col = df["AveOccup"]

q1, q3 = col.quantile([0.25, 0.75])
iqr = q3 - q1
high = q3 + 1.5 * iqr

print(f"Normal range tops out near {high:.1f} people per home")
print("Rows above that:", (col > high).sum())
print("Largest value:  ", round(col.max(), 1))
print("Largest values: ", col.nlargest(3).round(0).tolist())

Normal range tops out near 4.6 people per home
Rows above that: 704
Largest value:   1243.3
Largest values:  [1243.0, 600.0, 502.0]

Four ways to respond

You have four options. Investigate first, because an outlier may be a data entry error that you can fix. Remove it, but only if you are sure it is a mistake, not just an unusual truth. Clip it, by capping values at, say, the ninety ninth percentile. Or use a model that does not care, because tree models, which you will meet soon, barely notice outliers at all.

Does clipping help?

Let's test it, instead of guessing. We measure the cross validated score on the raw data. Then we clip every feature at its ninety ninth percentile, and measure again. Run it. The score goes up, from one small change to the data. Notice what happened here. We changed no model, and no setting. We only cleaned the data. This is the lesson that professionals never forget.

import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score, KFold

df = fetch_california_housing(as_frame=True).frame
y = df.pop("MedHouseVal")
folds = KFold(5, shuffle=True, random_state=0)

before = cross_val_score(LinearRegression(), df, y, cv=folds).mean()

fixed = df.copy()
limits = fixed.quantile(0.99)
fixed = fixed.clip(upper=limits, axis=1)
after = cross_val_score(LinearRegression(), fixed, y, cv=folds).mean()

print(f"R2 before: {before:.3f}")
print(f"R2 after:  {after:.3f}")

R2 before: 0.604
R2 after:  0.673

Summary