Selection Bias

A business writer studies hundreds of successful cafes, and finds something strange. Among successful cafes, the ones with the best coffee tend to have the worst locations. So is a bad location good for coffee? Of course not. But the pattern's really there. Where does it come from?

• Studying successful cafes
• Best coffee, worst locations
• Where does this come from?

Creating a pattern from nothing

We simulate a hundred thousand cafes. Coffee quality and location quality are completely unrelated. But a cafe survives if the two together are good enough. Then we look at the correlation, for all cafes, and for only the survivors. Run it. Across all cafes, the correlation is zero, as we designed. Among the survivors, it's minus point six. A strong pattern has appeared, from nothing. Why? A surviving cafe with a bad location must have great coffee, or it'd be dead. The act of selecting survivors created the link. This is called collider bias.

import numpy as np

rng = np.random.default_rng(0)
n = 100000
coffee = rng.normal(0, 1, n)
location = rng.normal(0, 1, n)
survived = coffee + location > 1

print(f"All cafes ever opened: {np.corrcoef(coffee, location)[0, 1]:+.2f}")
print(f"Only the survivors:    {np.corrcoef(coffee[survived], location[survived])[0, 1]:+.2f}")

All cafes ever opened: +0.00
Only the survivors:    -0.61

Where it hides

It's all around you. Among film stars, good looks and acting skill seem unrelated, because you needed one or the other to get in. Hospital patients show odd links between diseases, as you only see people who were ill enough to go. And among your job applicants, skills may seem to trade off. Whenever you filter data, ask how that filter could create a link.

Summary