AI Hints
Home›Statistics & Math for ML Tutorial›Going Deeper: Probability in Models
Statistics & Math for ML · Lesson 135 of 138

Calibration

Can you trust the probability?.

⏱ 2 min read▶ Video at 2:41:34🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A weather app says seventy percent chance of rain. If it's well calibrated, it should rain on about seventy percent of the days it says that. Many models give probabilities that are too confident, or not confident enough. Calibration checks them.

Calibration
From the video: Calibration
  • 70% chance of rain
  • Does it rain 70% of the time?
  • Calibration

A reliability curve

We train two models, naive Bayes and logistic regression, and compare their predicted probabilities with what actually happened. Calibration curve groups predictions into bins, and finds the real share of positives in each. Run it. Logistic regression stays close to the diagonal. Naive Bayes wanders far away from it, so its probabilities can't be trusted.

calibration.pypython
import matplotlib.pyplot as plt
from sklearn.calibration import calibration_curve
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=20000, n_features=20, n_informative=2, n_redundant=10, random_state=42)
Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=0)
for model in [LogisticRegression(max_iter=1000), GaussianNB()]:
    real, pred = calibration_curve(yte, model.fit(Xtr, ytr).predict_proba(Xte)[:, 1], n_bins=10)
    plt.plot(pred, real, "o-", label=type(model).__name__)
plt.plot([0, 1], [0, 1], "--", color="#ffffff"); plt.legend(); plt.title("Calibration")
plt.savefig("calibration.png")

Fixing calibration

If a model ranks well, but its probabilities are off, you can recalibrate it. Scikit learn's calibrated classifier CV learns a correction, using Platt scaling or isotonic regression. Calibration matters whenever the actual probability is used, like pricing risk, or deciding when a doctor should double check.

keyvalue
Calibrated70% means 70%
Checkcalibration_curve
FixCalibratedClassifierCV
Matters forrisk, medicine, pricing

Summary

A calibrated model's probabilities match reality. Check with a calibration curve, and fix with calibrated classifier CV. Next, a cheat sheet of every formula.

Key takeaways
  • calibration_curve
  • Diagonal is perfect
  • CalibratedClassifierCV
Next up: Formula Cheat Sheet — everything in one place