Instrumental Variables

Do more visits make people spend more per month? You can't just compare. Keen customers both visit more and spend more. But what if you'd randomly emailed half of your customers, and that email nudged some of them into visiting more? You can use that random nudge to work out the real effect of a visit.

• Do visits increase spending?
• Keen customers do both
• Use a random nudge

Using an instrument

In this simulation, the true effect of a visit is point eight of a dollar, and keenness confuses things. The email was random, and it only affects spend through visits. That's the key requirement. The method's beautifully simple. How much did the email raise spending? And how much did it raise visits? Divide one by the other. Run it. The naive model says one point nine six, which is more than double the truth. The instrument says point eight three, which is very close. It's only two averages, and one division. Economists won a Nobel Prize for developing this way of thinking.

import numpy as np
import statsmodels.api as sm

rng = np.random.default_rng(0)
n = 20000
keen = rng.normal(0, 1, n)
got_email = rng.integers(0, 2, n)
visits = 2 + 1.5 * got_email + keen + rng.normal(0, 1, n)
spend = 5 + 0.8 * visits + 3 * keen + rng.normal(0, 1, n)

naive = sm.OLS(spend, sm.add_constant(visits)).fit().params[1]
first = spend[got_email == 1].mean() - spend[got_email == 0].mean()
second = visits[got_email == 1].mean() - visits[got_email == 0].mean()

print(f"Naive effect of a visit: {naive:.2f}")
print(f"Instrument estimate:     {first / second:.2f}  (true 0.80)")

Naive effect of a visit: 1.96
Instrument estimate:     0.83  (true 0.80)

When it works

A good instrument must do two things. It has to really change the treatment. And it must affect the outcome only through that treatment. That second part can't be tested with data, so you have to argue for it. Random promotions, lotteries, and weather have all been used. And if the instrument's weak, the answer becomes unreliable.

Summary