Metrics That Matter

A model is only as useful as the metric used to judge it. Pick the wrong one, and you can optimise yourself into a worse business. In this lesson, you will turn a model's scores into business value, choose a decision threshold with real costs, and learn how to run a fair A B test.

• Metrics must match the business
• Thresholds have costs
• Experiments need enough data

From model metrics to business metrics

There are two kinds of metrics. Model metrics, like precision, recall, and the area under the curve, tell you how well the model separates cases. Business metrics, like customers retained, revenue, or hours saved, tell you whether the model creates value. A model can improve its metric, and still not move the business, for example, if the retention offers it triggers are too expensive. So always connect the two. Estimate what each prediction outcome is worth, and judge the model in money, or in whatever the business counts.

1. Model metricprecision, recall, AUC
2. Cost of each outcomewhat a miss or a false alarm costs
3. Business metriccustomers kept, money saved
4. Decisionship, tune, or rethink

Choosing a threshold by cost

Here is how costs choose a threshold. A model gives a churn probability. We must decide above which probability to call the customer. Suppose a retention call costs five dollars, and a customer who leaves, costs two hundred dollars of lost value, but a call saves only a third of them. For each possible threshold, we compute the expected profit, from the calls we make, and the customers we keep. Then we print the best threshold. It is not the default of fifty percent. It is much lower, because missing a leaving customer costs far more, than a call. The right threshold comes from money, not from habit.

import numpy as np
from sklearn.linear_model import LogisticRegression
from churn import make_data

train, test = make_data(4000, seed=1), make_data(3000, seed=2)
features = ['tenure', 'charges', 'calls', 'contract']
model = LogisticRegression(max_iter=500).fit(train[features], train['churn'])
prob = model.predict_proba(test[features])[:, 1]

CALL_COST, CHURN_COST, SAVE_RATE = 5, 200, 1 / 3

def profit(threshold):
    called = prob >= threshold
    saved = called & (test['churn'].to_numpy() == 1)
    return saved.sum() * CHURN_COST * SAVE_RATE - called.sum() * CALL_COST

for t in (0.5, 0.3, 0.2, 0.1, 0.05):
    print(f'threshold {t:.2f}  profit ${profit(t):8.0f}')
best = max(np.arange(0.02, 0.6, 0.01), key=profit)
print('best threshold:', round(float(best), 2))

threshold 0.50  profit $    3623
threshold 0.30  profit $    9700
threshold 0.20  profit $   12693
threshold 0.10  profit $   16480
threshold 0.05  profit $   16345
best threshold: 0.07

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

Summary