Handling Outliers

Average spend jumps fourfold overnight. Is it a huge success, or a data bug? Three strange numbers, among a thousand, have dragged your average up into the sky. How you handle them can completely change your conclusion, so it's worth thinking about carefully.

• Average spend jumps fourfold
• Success or a bug?
• Three strange numbers

Means, medians, and trimming

We add three giant values to a thousand ordinary ones. Run it. The mean leaps from ten to forty six. The median barely moves. That's what makes it robust. A trimmed mean drops the top and bottom one percent, and it recovers a sensible value. And the box rule flags values that are far beyond the middle half of the data. It's found thirteen outliers. Those include our three, along with ten real big spenders. So be careful. Those ten are real customers, and they might be your most valuable ones.

import numpy as np

rng = np.random.default_rng(0)
spend = rng.lognormal(2.2, .6, 1000)
broken = spend.copy()
broken[:3] = [9000, 12000, 15000]

print(f"Mean: {spend.mean():.1f} -> {broken.mean():.1f}")
print(f"Median: {np.median(spend):.1f} -> {np.median(broken):.1f}")

low, high = np.quantile(broken, [.01, .99])
print(f"Trimmed mean: {broken[(broken >= low) & (broken <= high)].mean():.1f}")

q1, q3 = np.quantile(broken, [.25, .75])
print("Outliers by the box rule:", ((broken > q3 + 3 * (q3 - q1))).sum())

Mean: 10.4 -> 46.4
Median: 8.6 -> 8.7
Trimmed mean: 10.3
Outliers by the box rule: 13

What should you do?

The right response depends on the cause. A data error should be fixed, or removed. A real but extreme value should usually be kept, and you should use robust statistics. If the outliers are the whole story, such as big spenders, study them on their own. And always report what you did, and how the answer changes without them.

Summary