Choosing the Threshold
You decide which mistakes to make.
Here is a secret that many beginners never learn. The default cut-off of point five is just a default. Nothing makes it right. If the model is thirty percent sure that a tumour is malignant, would you send that patient home? Of course not. The threshold is a business decision, and it is yours to make.
- 0.5 is only a default
- Would you risk thirty percent?
- The threshold is your decision
Moving the threshold
We flip the labels, so that one now means malignant, the case we must catch. We take the model's chances. Then we try four thresholds. Read the output from top to bottom. With a strict threshold of point nine, precision is high, but we miss cases. As we lower it, recall climbs. At point zero five, we catch every malignant tumour in this test, at the cost of more false alarms. It is a trade. The clinic would rather run an extra test than miss a tumour, so we choose a low threshold.
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
y = 1 - y # now 1 means malignant, the case we must catch
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.25, stratify=y, random_state=0)
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_tr, y_tr)
chance = model.predict_proba(X_te)[:, 1]
for t in [0.9, 0.5, 0.2, 0.05]:
p = (chance >= t).astype(int)
print(f"threshold {t:<5} precision {precision_score(y_te, p):.3f}"
f" recall {recall_score(y_te, p):.3f}")threshold 0.9 precision 1.000 recall 0.887 threshold 0.5 precision 1.000 recall 0.962 threshold 0.2 precision 0.852 recall 0.981 threshold 0.05 precision 0.779 recall 1.000
The ROC curve
To compare models across every possible threshold, we draw a curve. Along the bottom is the false alarm rate, and going up is the share of real cases we catch. As we lower the threshold, we move along the curve. A useless coin flip gives the diagonal line. A great model bends up towards the top left corner. The area under this curve, called the A U C, gives one number. One is perfect, and point five is a coin flip.
When classes are rare
When the rare class is very rare, the R O C curve can look too kind. Then we prefer the precision recall curve. In this example, only eight percent of cases are positive. On the left, the R O C curve looks strong. On the right, the precision recall curve shows a more sober picture. For rare events like fraud, trust the one on the right.
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import RocCurveDisplay, PrecisionRecallDisplay, roc_auc_score
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=4000, weights=[0.92], class_sep=0.8,
flip_y=0.02, random_state=1)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=0)
model = LogisticRegression().fit(X_tr, y_tr)
fig, ax = plt.subplots(1, 2, figsize=(10, 4.5))
RocCurveDisplay.from_estimator(model, X_te, y_te, ax=ax[0])
PrecisionRecallDisplay.from_estimator(model, X_te, y_te, ax=ax[1])
plt.tight_layout()
plt.savefig("curves.png", dpi=120)
print("AUC:", round(roc_auc_score(y_te, model.predict_proba(X_te)[:, 1]), 3))AUC: 0.896
Summary
You now choose your mistakes on purpose. This is a real professional skill, and one that many data scientists skip. The clinic's tool now catches more tumours. But there is still that imbalance problem. Next, we go directly after rare classes.
Key takeaways- The threshold is a business choice
- ROC AUC compares models
- Use precision recall for rare events