Fair Comparisons

Customers who hold a loyalty card leave far less often. So is the card working? Remember, nobody assigned those cards at random. Keen customers sign up, and they'd have stayed anyway. How can you compare fairly, when people chose for themselves?

• Card holders leave less
• Nobody assigned cards at random
• Keen customers sign up anyway

Inverse propensity weighting

We simulate customers, where frequent visitors are more likely to take a card, and more likely to stay. We've built in a small real benefit from the card, so we know the true answer. First, the naive comparison. Then we model the chance that each person has a card. We call that a propensity. And we give extra weight to the surprising people, such as the rare visitor who took one. Run it. The naive gap says that the card cuts churn by seventeen points. After weighting, most of that vanishes, down to four. So the card does help a little, but most of the gap was just the type of person who signs up. A claim of seventeen points would have badly misled the founder.

import numpy as np
import pandas as pd
import statsmodels.formula.api as smf

rng = np.random.default_rng(0)
n = 6000
visits = rng.gamma(2, 2, n)
card = (rng.random(n) < 1 / (1 + np.exp(1.5 - 0.5 * visits))).astype(int)
leave = (rng.random(n) < 1 / (1 + np.exp(-0.2 + 0.35 * visits + 0.3 * card))).astype(int)
df = pd.DataFrame({"visits": visits, "card": card, "leave": leave})

print("Naive gap:", round(df[df.card == 1].leave.mean() - df[df.card == 0].leave.mean(), 3))

df["p"] = smf.logit("card ~ visits", df).fit(disp=0).predict(df)
df["w"] = np.where(df.card == 1, 1 / df.p, 1 / (1 - df.p))
treated = np.average(df[df.card == 1].leave, weights=df[df.card == 1].w)
control = np.average(df[df.card == 0].leave, weights=df[df.card == 0].w)
print("Weighted gap:", round(treated - control, 3))

Naive gap: -0.173
Weighted gap: -0.04

The limits of weighting

This method only adjusts for what you measured. If keenness also depends on something that isn't in your data, you'll still be fooled. So check that groups look similar after weighting. Be wary of extreme weights. And wherever you can, run a real experiment.

Summary