Precision and Recall
Beyond accuracy.
Only one percent of transactions are fraud. A model that always says not fraud is ninety nine percent accurate, and completely useless. For rare classes, accuracy hides the truth. We need precision and recall.

- 1% are fraud
- Always say no: 99% accurate
- Useless! Accuracy hides it
The confusion matrix
Here are a hundred real labels, with five frauds, and a model's predictions. The confusion matrix counts true negatives, false positives, false negatives, and true positives. Precision asks, of the ones we flagged, how many were fraud? Recall asks, of the real frauds, how many did we catch? Run it.
import numpy as np
from sklearn.metrics import confusion_matrix, precision_score, recall_score
actual = np.array([0] * 95 + [1] * 5)
predicted = np.array([0] * 92 + [1] * 3 + [1] * 4 + [0] * 1)
print(confusion_matrix(actual, predicted))
print("accuracy: ", (actual == predicted).mean())
print("precision:", round(precision_score(actual, predicted), 3))
print("recall: ", recall_score(actual, predicted))[[92 3] [ 1 4]] accuracy: 0.96 precision: 0.571 recall: 0.8
Reading the numbers
We caught four of the five frauds, so recall is zero point eight. But of the seven we flagged, only four were fraud, so precision is about zero point five seven. There's always a trade off. A stricter model has higher precision, but lower recall. The F one score combines both into one number.
| key | value |
|---|---|
| Precision | TP ÷ (TP + FP) |
| Recall | TP ÷ (TP + FN) |
| Trade-off | stricter: more precise, fewer caught |
| F1 | balance of both |
Summary
Precision is how many flagged cases were right, and recall is how many real cases we caught. Use them for rare classes. Next, the ROC curve.
Key takeaways- confusion_matrix
- Precision: flagged and right
- Recall: caught