Why Accuracy Lies

A startup boasts of a fraud detector with ninety nine percent accuracy. Investors are impressed. Then someone reads the code. It says, no matter what the transaction is, answer, not fraud. Since only one percent of transactions are fraud, this lazy line of code is right ninety nine percent of the time, and it has never caught a criminal.

• 99 percent accurate
• It always says not fraud
• It catches nobody

The lazy model

Let's build that lazy model. Here are a thousand transactions, of which only ten are fraud. Dummy classifier always predicts the most common answer. Run it. Accuracy is point nine nine. Frauds caught, zero. This is called class imbalance, and it is everywhere that matters. Disease, fraud, machine failure, and spam. Whenever the thing you care about is rare, accuracy is a trap.

import numpy as np
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score, recall_score

y = np.array([0] * 990 + [1] * 10)
X = np.zeros((1000, 1))

lazy = DummyClassifier(strategy="most_frequent").fit(X, y)
pred = lazy.predict(X)

print("Accuracy:", accuracy_score(y, pred))
print("Frauds caught:", recall_score(y, pred))

Accuracy: 0.99
Frauds caught: 0.0

The confusion matrix

The cure is to count four outcomes, in a box called the confusion matrix. A true positive, the model said yes, and it was right. A true negative, it said no, and it was right. A false positive, a false alarm, where a healthy person is told to worry. And a false negative, a missed case, where a sick person is sent home. Those last two kinds of mistake have very different costs. From this box, we get precision and recall.

Precision and recall

Precision asks, when the model says yes, how often is it right? It protects against false alarms. Recall asks, of all the real cases, how many did we find? It protects against missed cases. A spam filter wants high precision, because losing a real email is terrible. A cancer screen wants high recall, because missing a tumour is far worse than a second test. F one combines the two, into a single number.

A full report

Scikit learn prints all of this for you. Confusion matrix gives us the four counts, and classification report gives precision, recall, and F one, for each class. Read the matrix. The diagonal holds the correct answers, and the corners hold the mistakes. There are only a few. But we still have missed some malignant tumours. For a clinic, even one of those matters. So can we choose to catch more of them? Yes.

from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=0)
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_tr, y_tr)
pred = model.predict(X_te)

print(confusion_matrix(y_te, pred))
print(classification_report(y_te, pred, digits=3,
                            target_names=["malignant", "benign"]))

[[50  3]
 [ 3 87]]
              precision    recall  f1-score   support

   malignant      0.943     0.943     0.943        53
      benign      0.967     0.967     0.967        90

    accuracy                          0.958       143
   macro avg      0.955     0.955     0.955       143
weighted avg      0.958     0.958     0.958       143

Summary