Calibration
Can you trust the probability?.
A weather app says seventy percent chance of rain. If it's well calibrated, it should rain on about seventy percent of the days it says that. Many models give probabilities that are too confident, or not confident enough. Calibration checks them.

- 70% chance of rain
- Does it rain 70% of the time?
- Calibration
A reliability curve
We train two models, naive Bayes and logistic regression, and compare their predicted probabilities with what actually happened. Calibration curve groups predictions into bins, and finds the real share of positives in each. Run it. Logistic regression stays close to the diagonal. Naive Bayes wanders far away from it, so its probabilities can't be trusted.
import matplotlib.pyplot as plt
from sklearn.calibration import calibration_curve
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=20000, n_features=20, n_informative=2, n_redundant=10, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=0)
for model in [LogisticRegression(max_iter=1000), GaussianNB()]:
real, pred = calibration_curve(yte, model.fit(Xtr, ytr).predict_proba(Xte)[:, 1], n_bins=10)
plt.plot(pred, real, "o-", label=type(model).__name__)
plt.plot([0, 1], [0, 1], "--", color="#ffffff"); plt.legend(); plt.title("Calibration")
plt.savefig("calibration.png")Fixing calibration
If a model ranks well, but its probabilities are off, you can recalibrate it. Scikit learn's calibrated classifier CV learns a correction, using Platt scaling or isotonic regression. Calibration matters whenever the actual probability is used, like pricing risk, or deciding when a doctor should double check.
| key | value |
|---|---|
| Calibrated | 70% means 70% |
| Check | calibration_curve |
| Fix | CalibratedClassifierCV |
| Matters for | risk, medicine, pricing |
Summary
A calibrated model's probabilities match reality. Check with a calibration curve, and fix with calibrated classifier CV. Next, a cheat sheet of every formula.
Key takeaways- calibration_curve
- Diagonal is perfect
- CalibratedClassifierCV