AI Hints
Statistics & Math for ML · Lesson 113 of 138

Precision and Recall

Beyond accuracy.

⏱ 2 min read▶ Video at 2:14:19🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Only one percent of transactions are fraud. A model that always says not fraud is ninety nine percent accurate, and completely useless. For rare classes, accuracy hides the truth. We need precision and recall.

Precision and Recall
From the video: Precision and Recall
  • 1% are fraud
  • Always say no: 99% accurate
  • Useless! Accuracy hides it

The confusion matrix

Here are a hundred real labels, with five frauds, and a model's predictions. The confusion matrix counts true negatives, false positives, false negatives, and true positives. Precision asks, of the ones we flagged, how many were fraud? Recall asks, of the real frauds, how many did we catch? Run it.

precision.pypython
import numpy as np
from sklearn.metrics import confusion_matrix, precision_score, recall_score

actual = np.array([0] * 95 + [1] * 5)
predicted = np.array([0] * 92 + [1] * 3 + [1] * 4 + [0] * 1)

print(confusion_matrix(actual, predicted))
print("accuracy: ", (actual == predicted).mean())
print("precision:", round(precision_score(actual, predicted), 3))
print("recall:   ", recall_score(actual, predicted))
Output
[[92  3]
 [ 1  4]]
accuracy:  0.96
precision: 0.571
recall:    0.8

Reading the numbers

We caught four of the five frauds, so recall is zero point eight. But of the seven we flagged, only four were fraud, so precision is about zero point five seven. There's always a trade off. A stricter model has higher precision, but lower recall. The F one score combines both into one number.

keyvalue
PrecisionTP ÷ (TP + FP)
RecallTP ÷ (TP + FN)
Trade-offstricter: more precise, fewer caught
F1balance of both

Summary

Precision is how many flagged cases were right, and recall is how many real cases we caught. Use them for rare classes. Next, the ROC curve.

Key takeaways
  • confusion_matrix
  • Precision: flagged and right
  • Recall: caught
Next up: ROC and AUC — every threshold