Natural Experiments

In eighteen fifty four, John Snow suspected that dirty water caused cholera. He couldn't run an experiment. But two water companies served the same streets, and one had moved its intake upstream. He compared their customers, and he proved it. That's a natural experiment, and you can use the same logic.

• 1854: cholera in London
• Two water companies
• A natural experiment

Difference in differences

Oslo gets a new menu, and Bergen doesn't. Oslo's sales go up. But maybe everything rose because of the summer. So we use Bergen as a comparison. Take Oslo's change, and subtract Bergen's change. Run it. Oslo rose by one hundred and eighty, but Bergen rose by ninety anyway. So the menu's real effect is about ninety a day, half of the naive answer. This only works if both cafes would've moved in parallel without the change.

import pandas as pd

data = pd.DataFrame({
    "group": ["Oslo", "Oslo", "Bergen", "Bergen"],
    "period": ["Before", "After", "Before", "After"],
    "daily_sales": [1000, 1180, 900, 990],
})

t = data.pivot(index="group", columns="period", values="daily_sales")
print(t[["Before", "After"]].to_string())

change_oslo = t.loc["Oslo", "After"] - t.loc["Oslo", "Before"]
change_bergen = t.loc["Bergen", "After"] - t.loc["Bergen", "Before"]
print("Effect of the new menu:", change_oslo - change_bergen, "per day")

period  Before  After
group
Bergen     900    990
Oslo      1000   1180
Effect of the new menu: 90 per day

Other natural experiments

Other designs exist. Regression discontinuity compares people just above and just below a cut-off, such as a loyalty tier. Instrumental variables use something that nudges behaviour at random. Geographic tests switch something on in some cities, and not in others. Look for these. Companies often run accidental experiments, without noticing.

Summary