Finding the Unknown
Anomaly detection.
A new kind of disease appears on the apples, one that nobody's ever seen. Your bruise model was never trained on it, so it says, fine. And how would you ever collect examples of defects that haven't happened yet? There's a neat way round this. Stop learning what's bad. Learn what's normal.
- A disease nobody's seen
- The model says 'fine'
- Learn what's normal
Learning normal
We train on good apples only. There isn't a single bruise in the training set. PCA learns to compress and rebuild images of normal apples. It's good at that, because it's seen thousands. But it's bad at rebuilding something that it's never seen. So the rebuilding error becomes our score. We set a limit from good images only. Run it. Five percent of good apples are flagged, which is the limit that we chose. And it flags one hundred percent of the bruised apples, without ever having seen one. That's the power of this approach. Today's best tools, such as PatchCore, use the same idea, with deep features.
import numpy as np
from sklearn.decomposition import PCA
from patch import make
good, _ = make(1500, bruise_rate=0)
test_good, _ = make(200, seed=5, bruise_rate=0)
test_bad, _ = make(200, seed=6, bruise_rate=1)
flat = lambda x: x.reshape(len(x), -1).numpy()
pca = PCA(30, random_state=0).fit(flat(good))
def score(x):
a = flat(x)
return ((a - pca.inverse_transform(pca.transform(a))) ** 2).mean(1)
limit = np.quantile(score(test_good), 0.95)
print(f"Limit: {limit:.5f}")
print(f"Good flagged: {(score(test_good) > limit).mean():.0%}")
print(f"Bruised flagged: {(score(test_bad) > limit).mean():.0%}")Limit: 0.00186 Good flagged: 5% Bruised flagged: 100%
Also uses patch.py
import numpy as np
import cv2
import torch
def make(n, seed=0, light=1.0, bruise_rate=0.5):
"""Simulated apple photos. Label 1 means bruised."""
rng = np.random.default_rng(seed)
X, y = [], []
for _ in range(n):
img = np.full((48, 48, 3), (70, 75, 80), np.uint8)
c = tuple(int(v) for v in rng.integers(19, 30, 2))
r = int(rng.integers(15, 20))
cv2.circle(img, c, r, (35, 40, int(rng.integers(150, 225))), -1, cv2.LINE_AA)
bad = rng.random() < bruise_rate
if bad:
at = (c[0] + int(rng.integers(-8, 9)), c[1] + int(rng.integers(-8, 9)))
cv2.circle(img, at, int(rng.integers(3, 6)), (25, 45, 70), -1, cv2.LINE_AA)
img = img * light + rng.normal(0, 8, img.shape)
X.append(img.clip(0, 255) / 255)
y.append(int(bad))
X = torch.tensor(np.array(X), dtype=torch.float32).permute(0, 3, 1, 2)
return X, torch.tensor(y)
def net():
return torch.nn.Sequential(
torch.nn.Conv2d(3, 16, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
torch.nn.Conv2d(16, 32, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
torch.nn.Flatten(), torch.nn.Linear(32 * 12 * 12, 2))
def train(model, X, y, epochs=8):
opt = torch.optim.Adam(model.parameters(), lr=2e-3)
for _ in range(epochs):
for i in torch.randperm(len(X)).split(64):
loss = torch.nn.functional.cross_entropy(model(X[i]), y[i])
opt.zero_grad(); loss.backward(); opt.step()
return model
def accuracy(model, X, y):
return (model(X).argmax(1) == y).float().mean().item()When to use anomaly detection
Use it when defects are rare or unpredictable, and normal products look alike. It's a perfect match for factories. It's less good when normal items vary widely. It can't tell you what the defect is, only that something's wrong. So teams often combine the two approaches. A classifier for known defects, and anomaly detection as a safety net.
Summary
Learn what's normal, and anything else stands out. Next, why you shouldn't trust how confident a model says it is.
Key takeaways- Train on good examples only
- Rebuilding error is the score
- Use it as a safety net