Rare Defects
When ninety-seven percent means nothing.
On a real line, only about three apples in a hundred are bruised. So a model that always says, fine, scores ninety seven percent, and it catches nothing at all. Rare events are what factories, hospitals, and security systems are all about. So accuracy is the wrong score.
- Three in a hundred are bruised
- 'Always fine' scores 97%
- Accuracy is the wrong score
Oversampling the rare class
We make a realistic training set, where only about three percent are bruised. We train one plain model. Then we train a second one, where each bruised example is repeated thirty times, so that the classes are balanced. And we count how many bruises each one found. Run it. Both have fantastic looking accuracy. But look at the count. The plain model found twenty one of twenty six bruises. The balanced one found twenty four. The accuracy figure barely moved, yet the real result improved. That's why you have to watch recall.
import torch
from patch import make, net, train
torch.manual_seed(0)
X, y = make(3000, bruise_rate=0.03)
print("Bruised in training:", y.sum().item(), "of", len(y))
Xt, yt = make(1000, seed=9, bruise_rate=0.03)
plain = train(net(), X, y)
rare = (y == 1).nonzero().ravel()
boost = rare.repeat(30)
balanced = train(net(), torch.cat([X, X[boost]]), torch.cat([y, y[boost]]))
for name, m in [("Plain", plain), ("Oversampled", balanced)]:
pred = m(Xt).argmax(1)
found = ((pred == 1) & (yt == 1)).sum().item()
print(f"{name:<12} accuracy {(pred == yt).float().mean():.3f}, found {found} of {yt.sum().item()}")Bruised in training: 77 of 3000 Plain accuracy 0.995, found 21 of 26 Oversampled accuracy 0.998, found 24 of 26
Also uses patch.py
import numpy as np
import cv2
import torch
def make(n, seed=0, light=1.0, bruise_rate=0.5):
"""Simulated apple photos. Label 1 means bruised."""
rng = np.random.default_rng(seed)
X, y = [], []
for _ in range(n):
img = np.full((48, 48, 3), (70, 75, 80), np.uint8)
c = tuple(int(v) for v in rng.integers(19, 30, 2))
r = int(rng.integers(15, 20))
cv2.circle(img, c, r, (35, 40, int(rng.integers(150, 225))), -1, cv2.LINE_AA)
bad = rng.random() < bruise_rate
if bad:
at = (c[0] + int(rng.integers(-8, 9)), c[1] + int(rng.integers(-8, 9)))
cv2.circle(img, at, int(rng.integers(3, 6)), (25, 45, 70), -1, cv2.LINE_AA)
img = img * light + rng.normal(0, 8, img.shape)
X.append(img.clip(0, 255) / 255)
y.append(int(bad))
X = torch.tensor(np.array(X), dtype=torch.float32).permute(0, 3, 1, 2)
return X, torch.tensor(y)
def net():
return torch.nn.Sequential(
torch.nn.Conv2d(3, 16, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
torch.nn.Conv2d(16, 32, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
torch.nn.Flatten(), torch.nn.Linear(32 * 12 * 12, 2))
def train(model, X, y, epochs=8):
opt = torch.optim.Adam(model.parameters(), lr=2e-3)
for _ in range(epochs):
for i in torch.randperm(len(X)).split(64):
loss = torch.nn.functional.cross_entropy(model(X[i]), y[i])
opt.zero_grad(); loss.backward(); opt.step()
return model
def accuracy(model, X, y):
return (model(X).argmax(1) == y).float().mean().item()Other tools for rare defects
You've got more options. Weight the loss, so that rare mistakes count for more. Use focal loss, which focuses on hard examples. Copy and paste real defects onto good images, which is a very effective trick. Generate synthetic defects. And use anomaly detection, which needs no defect examples at all. We'll build that soon.
Summary
Watch recall, and rebalance your training. Next, a mistake that can make a broken model look perfect.
Key takeaways- Accuracy hides rare failures
- Oversample or weight
- Track how many you find