AI Hints
Statistics & Math for ML · Lesson 114 of 138

ROC and AUC

Every threshold.

⏱ 2 min read▶ Video at 2:15:31🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A classifier outputs probabilities, and we pick a threshold to say yes or no. Change the threshold, and precision and recall change too. The ROC curve shows the model's performance at every threshold at once.

ROC and AUC
From the video: ROC and AUC
  • Probabilities, then a threshold
  • Change it, results change
  • See every threshold at once

Drawing the ROC curve

We train a logistic regression on a synthetic dataset, and get probabilities for the test set. ROC curve tries every threshold, and records the true positive rate and false positive rate. Run it. A good model's curve bows towards the top left corner. The dashed diagonal is random guessing.

roc.pypython
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=2000, n_features=8, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=0)
probs = LogisticRegression().fit(Xtr, ytr).predict_proba(Xte)[:, 1]

fpr, tpr, _ = roc_curve(yte, probs)
plt.plot(fpr, tpr); plt.plot([0, 1], [0, 1], "--", color="#ffffff")
plt.title(f"ROC curve, AUC = {roc_auc_score(yte, probs):.3f}")
plt.savefig("roc.png")

AUC in one number

The area under the ROC curve, called AUC, sums the whole curve up in one number. One is a perfect model, and zero point five is random guessing. It also has a nice meaning. It's the chance that the model gives a random positive example a higher score than a random negative one.

keyvalue
ROCTPR vs FPR at every threshold
AUC = 1perfect
AUC = 0.5random guessing
MeaningP(positive ranked above negative)

Summary

The ROC curve shows performance at every threshold, and AUC sums it up. Next, cross validation, for even more reliable scores.

Key takeaways
  • roc_curve
  • roc_auc_score
  • 0.5 is random
Next up: Cross-Validation — many splits