Overconfident Models
When ninety-eight percent sure means wrong.
We've planned a safety rule. If the model's less than ninety percent sure, send that apple to a person. It sounds very sensible. But it only works if the confidence numbers mean something. Let's test that.
- If unsure, ask a person
- Sounds sensible
- Do the numbers mean anything?
Testing confidence
We train a model, then test it in dim light, where we know that it struggles. We compare how confident it is with how often it's right. Then we try temperature scaling, which divides the outputs by a number, to soften them. Run it. Look at that first line. The model says that it's ninety eight percent sure. It's right only fifty nine percent of the time. It's wrong, and it has no idea. Temperature helps a little, but look how far it's still off. On unfamiliar data, confidence means very little. So our safety rule wouldn't have fired at all.
import torch
from patch import make, net, train
torch.manual_seed(0)
model = train(net(), *make(800), epochs=25)
X, y = make(1000, seed=9, light=0.7)
p = model(X).softmax(1).detach()
conf, pred = p.max(1)
right = (pred == y).float()
print(f"Average confidence {conf.mean():.3f}, real accuracy {right.mean():.3f}")
for T in [1.0, 2.0, 4.0]:
c = (model(X) / T).softmax(1).max(1)[0].mean().item()
print(f"Temperature {T}: confidence {c:.3f}")Average confidence 0.984, real accuracy 0.589 Temperature 1.0: confidence 0.984 Temperature 2.0: confidence 0.968 Temperature 4.0: confidence 0.939
Also uses patch.py
import numpy as np
import cv2
import torch
def make(n, seed=0, light=1.0, bruise_rate=0.5):
"""Simulated apple photos. Label 1 means bruised."""
rng = np.random.default_rng(seed)
X, y = [], []
for _ in range(n):
img = np.full((48, 48, 3), (70, 75, 80), np.uint8)
c = tuple(int(v) for v in rng.integers(19, 30, 2))
r = int(rng.integers(15, 20))
cv2.circle(img, c, r, (35, 40, int(rng.integers(150, 225))), -1, cv2.LINE_AA)
bad = rng.random() < bruise_rate
if bad:
at = (c[0] + int(rng.integers(-8, 9)), c[1] + int(rng.integers(-8, 9)))
cv2.circle(img, at, int(rng.integers(3, 6)), (25, 45, 70), -1, cv2.LINE_AA)
img = img * light + rng.normal(0, 8, img.shape)
X.append(img.clip(0, 255) / 255)
y.append(int(bad))
X = torch.tensor(np.array(X), dtype=torch.float32).permute(0, 3, 1, 2)
return X, torch.tensor(y)
def net():
return torch.nn.Sequential(
torch.nn.Conv2d(3, 16, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
torch.nn.Conv2d(16, 32, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
torch.nn.Flatten(), torch.nn.Linear(32 * 12 * 12, 2))
def train(model, X, y, epochs=8):
opt = torch.optim.Adam(model.parameters(), lr=2e-3)
for _ in range(epochs):
for i in torch.randperm(len(X)).split(64):
loss = torch.nn.functional.cross_entropy(model(X[i]), y[i])
opt.zero_grad(); loss.backward(); opt.step()
return model
def accuracy(model, X, y):
return (model(X).argmax(1) == y).float().mean().item()Getting honest uncertainty
So what can you do? Calibrate on data that looks like production, and check it again whenever things change. Use several models, and treat disagreement as a warning. Add a separate check for unusual inputs, like the anomaly detector that we just built. And never use raw confidence as your only safety net.
Summary
Confidence isn't accuracy, especially on new data. So how do you notice when the world changes, without any labels? Next, monitoring.
Key takeaways- Models can be sure and wrong
- Temperature helps only partly
- Add input checks