ROC and AUC
Every threshold.
A classifier outputs probabilities, and we pick a threshold to say yes or no. Change the threshold, and precision and recall change too. The ROC curve shows the model's performance at every threshold at once.

- Probabilities, then a threshold
- Change it, results change
- See every threshold at once
Drawing the ROC curve
We train a logistic regression on a synthetic dataset, and get probabilities for the test set. ROC curve tries every threshold, and records the true positive rate and false positive rate. Run it. A good model's curve bows towards the top left corner. The dashed diagonal is random guessing.
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.model_selection import train_test_split
X, y = make_classification(n_samples=2000, n_features=8, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=0)
probs = LogisticRegression().fit(Xtr, ytr).predict_proba(Xte)[:, 1]
fpr, tpr, _ = roc_curve(yte, probs)
plt.plot(fpr, tpr); plt.plot([0, 1], [0, 1], "--", color="#ffffff")
plt.title(f"ROC curve, AUC = {roc_auc_score(yte, probs):.3f}")
plt.savefig("roc.png")AUC in one number
The area under the ROC curve, called AUC, sums the whole curve up in one number. One is a perfect model, and zero point five is random guessing. It also has a nice meaning. It's the chance that the model gives a random positive example a higher score than a random negative one.
| key | value |
|---|---|
| ROC | TPR vs FPR at every threshold |
| AUC = 1 | perfect |
| AUC = 0.5 | random guessing |
| Meaning | P(positive ranked above negative) |
Summary
The ROC curve shows performance at every threshold, and AUC sums it up. Next, cross validation, for even more reliable scores.
Key takeaways- roc_curve
- roc_auc_score
- 0.5 is random