Survivorship Bias

In the Second World War, the US military studied bombers returning from missions, marking where the bullet holes were. Most were on the wings and the tail. So they planned to add armour there. The statistician Abraham Wald told them they had it backwards. Those were the planes that came back. Armour the places with no holes.

• Bullet holes on returning planes
• Add armour there?
• Those are the ones that came back

Listening to survivors

Here's the business version. Only customers who stayed answer your survey. We simulate twenty thousand customers, each with a different experience. Those with a good experience stay. And a few of those answer. Run it. The average experience across all customers is zero. Among survey responders, it's plus point five six. Your survey looks wonderful, because the unhappy people have already gone. You can't learn about the people who aren't there. To understand them, you have to go and find them, with exit surveys and cancellation data.

import numpy as np

rng = np.random.default_rng(0)
n = 20000
quality = rng.normal(0, 1, n)
returned_ok = rng.random(n) < 1 / (1 + np.exp(-quality * 1.5))
talked = returned_ok & (rng.random(n) < .2)

print(f"All customers, average quality: {quality.mean():+.2f}")
print(f"The ones who answer a survey:   {quality[talked].mean():+.2f}")

All customers, average quality: +0.00
The ones who answer a survey:   +0.56

Summary