Regression Discontinuity

Spend five hundred dollars, and you get Gold status. Spend four hundred and ninety nine, and you don't. Are those two customers really different? Of course not. One just happened to buy one more coffee. So that cut-off gives you something very close to a random experiment.

• $500 gets you Gold
• $499 doesn't
• Nearly random at the cut-off

Measuring the jump

Gold customers spent more to begin with, so comparing all Gold customers with everyone else is hopeless. Instead, we only look near the line, and we fit a straight line on each side. The effect is the jump at the line. Run it. The naive gap is over a hundred dollars. The jump is about twenty six, and that's close to the true twenty five that we built in. Nearly all of that naive gap was just who these customers already were. Rules like this are everywhere, in pass marks, age limits, and credit scores.

import numpy as np
import pandas as pd
import statsmodels.formula.api as smf

rng = np.random.default_rng(0)
spend = rng.uniform(300, 700, 8000)
gold = (spend >= 500).astype(int)
next_year = 100 + 0.5 * spend + 25 * gold + rng.normal(0, 40, 8000)
df = pd.DataFrame({"s": spend - 500, "gold": gold, "y": next_year})

print("Naive gap:", round(df[df.gold == 1].y.mean() - df[df.gold == 0].y.mean(), 1))

near = df[df.s.abs() < 100]
m = smf.ols("y ~ gold + s + gold:s", near).fit()
print("Jump at the cut-off:", round(m.params["gold"], 1))
print("95% range:", m.conf_int().loc["gold"].round(1).tolist())

Naive gap: 124.4
Jump at the cut-off: 25.9
95% range: [20.9, 30.8]

Checking the design

This design has one key condition. People mustn't be able to control which side they land on. If customers can buy one extra coffee to reach Gold, then those just above aren't the same as those just below. So check for a pile up of people just over the line. And remember that the answer only applies to people near the cut-off.

Summary