Why Data Goes Missing

Income is blank for a quarter of your customers. The easy thing would be to drop those rows. But why is it blank? It turns out that high earners skip that question more often. So if you drop those rows, you've quietly removed your richest customers, and every number you report is now wrong.

• Income blank for a quarter
• Drop those rows?
• High earners skip the question

Missing for a reason

We simulate spending that depends on income, and high earners are much more likely to have a blank. The true mean is about twelve. If we just drop the missing rows, we get under eleven. That's a real bias. If we fill in the gaps, using the average for people with a similar income, we get very close to the truth. Run it. Twelve point oh two is the truth. Dropping gives ten point nine five. Filling by income band gives eleven point eight three. So always ask why it's missing. If it depends on something that you've got, you can adjust for it.

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
n = 5000
income = rng.normal(50, 15, n)
spend = 2 + .2 * income + rng.normal(0, 2, n)
missing = rng.random(n) < (income > 55) * .7
seen = pd.Series(np.where(missing, np.nan, spend))

print(f"True mean spend:    {spend.mean():.2f}")
print(f"Drop missing:       {seen.mean():.2f}")
print(f"Fill by income band: {seen.fillna(seen.groupby(pd.cut(income, 6), observed=True).transform('mean')).mean():.2f}")
print(f"Missing share: {missing.mean():.0%}")

True mean spend:    12.02
Drop missing:       10.95
Fill by income band: 11.83
Missing share: 26%

Three kinds of missing

Statisticians name three kinds. Missing completely at random is pure accident, and dropping rows is fine. Missing at random depends on something that you've measured, so you can adjust for it. Missing not at random depends on the missing value itself, such as high earners hiding their income, and that needs extra care. Always add a column that marks what was missing, as the gap itself can be useful.

Summary