Project Two: Customer Churn
Who is about to leave?.
A subscription company has a leak. Every month, customers quietly cancel. Winning a new customer costs several times more than keeping an old one. If you could tell who is about to leave, you could reach out first, with an offer. This is called churn prediction, and it's one of the most valuable models in business.
- Customers quietly cancel
- Keeping them is cheaper
- A very valuable model
Meet the customers
Here's the customer data, which we simulate from known rules, so that we can check our work. There's tenure, monthly charge, contract type, support calls, and whether they pay automatically. Some charges are missing, as they would be in real data. Run it. About a third of customers left. Now look at the contracts. People on monthly contracts leave far more often than those on two year deals. A simple group by, and we already have our first insight for the business.
import numpy as np
import pandas as pd
rng = np.random.default_rng(7)
n = 7000
df = pd.DataFrame({
"tenure": rng.integers(1, 72, n),
"monthly": rng.uniform(20, 120, n).round(2),
"contract": rng.choice(["monthly", "yearly", "two-year"], n, p=[.55, .25, .2]),
"support_calls": rng.poisson(1.5, n),
"autopay": rng.choice([0, 1], n, p=[.45, .55]),
})
score = (-2.2 + 0.05 * df["monthly"] - 0.04 * df["tenure"]
+ 0.45 * df["support_calls"] - 0.6 * df["autopay"]
+ df["contract"].map({"monthly": 0.9, "yearly": -0.6, "two-year": -1.5}))
df["left"] = (rng.random(n) < 1 / (1 + np.exp(-score))).astype(int)
df.loc[rng.random(n) < 0.05, "monthly"] = np.nan
df.to_csv("churn.csv", index=False)
print(df.head(3).to_string())
print()
print("Customers who left:", f"{df['left'].mean():.1%}")
print("By contract:")
print(df.groupby("contract")["left"].mean().round(2).to_string())tenure monthly contract support_calls autopay left 0 68 61.21 monthly 1 0 0 1 45 110.85 yearly 3 1 0 2 49 116.95 two-year 3 1 0 Customers who left: 54.1% By contract: contract monthly 0.67 two-year 0.32 yearly 0.44
Models and the right metric
Accuracy is the wrong score here, so we use the area under the precision recall curve. We build a column transformer, then compare logistic regression and boosting, with cross validation. Run it. Look at this surprise. They're almost tied, and the simple model may even edge ahead! Why? Because we built this data from a formula that's close to logistic. Real data isn't always that kind, but it's a great reminder. Never assume that the complex model will win. Always test.
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
rng = np.random.default_rng(7)
n = 7000
df = pd.DataFrame({
"tenure": rng.integers(1, 72, n),
"monthly": rng.uniform(20, 120, n).round(2),
"contract": rng.choice(["monthly", "yearly", "two-year"], n, p=[.55, .25, .2]),
"support_calls": rng.poisson(1.5, n),
"autopay": rng.choice([0, 1], n, p=[.45, .55]),
})
score = (-2.2 + 0.05 * df["monthly"] - 0.04 * df["tenure"]
+ 0.45 * df["support_calls"] - 0.6 * df["autopay"]
+ df["contract"].map({"monthly": 0.9, "yearly": -0.6, "two-year": -1.5}))
df["left"] = (rng.random(n) < 1 / (1 + np.exp(-score))).astype(int)
df.loc[rng.random(n) < 0.05, "monthly"] = np.nan
y = df.pop("left")
X_tr, X_te, y_tr, y_te = train_test_split(df, y, stratify=y, random_state=0)
nums = ["tenure", "monthly", "support_calls", "autopay"]
prep = ColumnTransformer([
("n", make_pipeline(SimpleImputer(strategy="median"), StandardScaler()), nums),
("c", OneHotEncoder(), ["contract"]),
])
models = {
"Logistic": make_pipeline(prep, LogisticRegression(max_iter=1000)),
"Boosting": make_pipeline(prep, HistGradientBoostingClassifier(
max_iter=150, learning_rate=0.05, random_state=0)),
}
for name, m in models.items():
s = cross_val_score(m, X_tr, y_tr, cv=5, scoring="average_precision").mean()
print(f"{name:<9} PR AUC {s:.3f}")
print(f"Share who left: {y_tr.mean():.3f}")Logistic PR AUC 0.879 Boosting PR AUC 0.855 Share who left: 0.541
Summary
A little exploring gave us our first insights, and a fair test told us that a simple model is enough here. But the company doesn't care about precision recall curves. It cares about money. Next, we turn our model into dollars.
Key takeaways- Churn: who is about to leave
- Use a metric that fits
- Test before assuming complex wins