From Scores to Dollars
Choosing the threshold that pays.
Here's the question that the finance team really cares about. Who should get a discount? An offer costs forty dollars. If it keeps a customer, that's worth four hundred. But if we send it to someone who was staying anyway, we've wasted forty dollars. The best threshold isn't a matter of statistics. It's a matter of money.
- Who gets a discount?
- Offer costs $40, saved customer is $400
- The threshold is about money
The money curve
Let's build this into code. Offer is the cost of each offer. Saved is the value of keeping a customer. And accept is the share who stay because of the offer. For each threshold, we find who we'd target, and work out the profit. Money gained from real leavers, minus the cost of all the offers. Run it. At a low threshold, we send too many offers. At a high one, we miss people. There's a clear best point in the middle, and the company now has a real dollar figure. This is how you make your boss's boss care about machine learning.
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
rng = np.random.default_rng(7)
n = 7000
df = pd.DataFrame({
"tenure": rng.integers(1, 72, n),
"monthly": rng.uniform(20, 120, n).round(2),
"contract": rng.choice(["monthly", "yearly", "two-year"], n, p=[.55, .25, .2]),
"support_calls": rng.poisson(1.5, n),
"autopay": rng.choice([0, 1], n, p=[.45, .55]),
})
score = (-2.2 + 0.05 * df["monthly"] - 0.04 * df["tenure"]
+ 0.45 * df["support_calls"] - 0.6 * df["autopay"]
+ df["contract"].map({"monthly": 0.9, "yearly": -0.6, "two-year": -1.5}))
df["left"] = (rng.random(n) < 1 / (1 + np.exp(-score))).astype(int)
df.loc[rng.random(n) < 0.05, "monthly"] = np.nan
y = df.pop("left")
X_tr, X_te, y_tr, y_te = train_test_split(df, y, stratify=y, random_state=0)
nums = ["tenure", "monthly", "support_calls", "autopay"]
prep = ColumnTransformer([
("n", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), nums),
("c", OneHotEncoder(), ["contract"]),
])
model = make_pipeline(prep, LogisticRegression(max_iter=1000)).fit(X_tr, y_tr)
chance = model.predict_proba(X_te)[:, 1]
OFFER, SAVED = 40, 400 # cost of an offer, value of a saved customer
ACCEPT = 0.3 # share who stay because of the offer
best = (0, 0)
for t in np.arange(0.05, 0.95, 0.05):
target = chance >= t
hits = (target & (y_te == 1)).sum()
profit = hits * ACCEPT * SAVED - target.sum() * OFFER
print(f"threshold {t:.2f} offers {target.sum():>5} profit ${profit:>8,.0f}") if round(t * 100) % 20 == 0 else None
best = max(best, (profit, t))
print(f"Best threshold {best[1]:.2f} earns ${best[0]:,.0f}")threshold 0.20 offers 1359 profit $ 55,440 threshold 0.40 offers 1066 profit $ 55,160 threshold 0.60 offers 813 profit $ 48,840 threshold 0.80 offers 518 profit $ 35,320 Best threshold 0.30 earns $56,320
Think in decisions
Notice what changed. We stopped asking, how accurate is the model? And we started asking, what decision does it support, and what is that decision worth? Every model exists to change a decision. There's one more subtle trap. A customer most likely to leave isn't always the best one to target, because some will leave no matter what. Uplift modelling finds the people that an offer will actually persuade.
Summary
Translate your model into dollars, and everyone in the company will listen. Your two projects are finished, and they show the two most common kinds of business problem. Next, let's look at how the world's best practitioners work, when competing in the top contests.
Key takeaways- Value the outcomes
- Pick thresholds by profit
- Models exist to change decisions