When the Light Changes
The failure that surprises everyone.
AppleEye passed every test in the lab. In the first week on the line, it was wonderful. Then the clocks changed, the evening shift started in darkness, and accuracy collapsed. Nothing had changed in the code. The world changed. That's called domain shift, and it's the number one killer of vision systems.
- It passed every test
- The clocks changed
- The world changed, not the code
Testing other conditions
We train normally, then test the same model at different light levels. Run it. At full light, ninety eight percent. At eighty percent light, it's eighty two. At sixty percent light, it's fifty three, which is no better than flipping a coin. Yet a person would hardly notice that change. The model had only ever seen one kind of lighting, so it learned details that depend on brightness. Your test set can't warn you about conditions it doesn't include.
import torch
from patch import make, net, train, accuracy
torch.manual_seed(0)
model = train(net(), *make(1500))
for light in [1.0, 0.8, 0.6, 1.3]:
X, y = make(500, seed=9, light=light)
print(f"Light x{light}: accuracy {accuracy(model, X, y):.3f}")Light x1.0: accuracy 0.976 Light x0.8: accuracy 0.820 Light x0.6: accuracy 0.526 Light x1.3: accuracy 0.990
Also uses patch.py
import numpy as np
import cv2
import torch
def make(n, seed=0, light=1.0, bruise_rate=0.5):
"""Simulated apple photos. Label 1 means bruised."""
rng = np.random.default_rng(seed)
X, y = [], []
for _ in range(n):
img = np.full((48, 48, 3), (70, 75, 80), np.uint8)
c = tuple(int(v) for v in rng.integers(19, 30, 2))
r = int(rng.integers(15, 20))
cv2.circle(img, c, r, (35, 40, int(rng.integers(150, 225))), -1, cv2.LINE_AA)
bad = rng.random() < bruise_rate
if bad:
at = (c[0] + int(rng.integers(-8, 9)), c[1] + int(rng.integers(-8, 9)))
cv2.circle(img, at, int(rng.integers(3, 6)), (25, 45, 70), -1, cv2.LINE_AA)
img = img * light + rng.normal(0, 8, img.shape)
X.append(img.clip(0, 255) / 255)
y.append(int(bad))
X = torch.tensor(np.array(X), dtype=torch.float32).permute(0, 3, 1, 2)
return X, torch.tensor(y)
def net():
return torch.nn.Sequential(
torch.nn.Conv2d(3, 16, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
torch.nn.Conv2d(16, 32, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
torch.nn.Flatten(), torch.nn.Linear(32 * 12 * 12, 2))
def train(model, X, y, epochs=8):
opt = torch.optim.Adam(model.parameters(), lr=2e-3)
for _ in range(epochs):
for i in torch.randperm(len(X)).split(64):
loss = torch.nn.functional.cross_entropy(model(X[i]), y[i])
opt.zero_grad(); loss.backward(); opt.step()
return model
def accuracy(model, X, y):
return (model(X).argmax(1) == y).float().mean().item()Augmentation fixes it
Now we train with varied brightness. Each image is copied, with a random brightness between half and one point four times the original, and half of them flipped. Same model, same data, with no new photos. Run it. Look at the dim light line. Fifty three percent has become ninety seven. And the normal accuracy has even gone up, to ninety nine. We taught it that lighting doesn't matter. Augment for the changes you expect in real life.
import torch
from patch import make, net, train, accuracy
torch.manual_seed(0)
X, y = make(1500)
gains = torch.empty(len(X), 1, 1, 1).uniform_(0.5, 1.4)
flips = torch.rand(len(X)) < 0.5
Xa = (X * gains).clamp(0, 1)
Xa[flips] = Xa[flips].flip(-1)
model = train(net(), torch.cat([X, Xa]), torch.cat([y, y]))
for light in [1.0, 0.8, 0.6, 1.3]:
Xt, yt = make(500, seed=9, light=light)
print(f"Light x{light}: accuracy {accuracy(model, Xt, yt):.3f}")Light x1.0: accuracy 0.994 Light x0.8: accuracy 0.992 Light x0.6: accuracy 0.966 Light x1.3: accuracy 0.994
Also uses patch.py
import numpy as np
import cv2
import torch
def make(n, seed=0, light=1.0, bruise_rate=0.5):
"""Simulated apple photos. Label 1 means bruised."""
rng = np.random.default_rng(seed)
X, y = [], []
for _ in range(n):
img = np.full((48, 48, 3), (70, 75, 80), np.uint8)
c = tuple(int(v) for v in rng.integers(19, 30, 2))
r = int(rng.integers(15, 20))
cv2.circle(img, c, r, (35, 40, int(rng.integers(150, 225))), -1, cv2.LINE_AA)
bad = rng.random() < bruise_rate
if bad:
at = (c[0] + int(rng.integers(-8, 9)), c[1] + int(rng.integers(-8, 9)))
cv2.circle(img, at, int(rng.integers(3, 6)), (25, 45, 70), -1, cv2.LINE_AA)
img = img * light + rng.normal(0, 8, img.shape)
X.append(img.clip(0, 255) / 255)
y.append(int(bad))
X = torch.tensor(np.array(X), dtype=torch.float32).permute(0, 3, 1, 2)
return X, torch.tensor(y)
def net():
return torch.nn.Sequential(
torch.nn.Conv2d(3, 16, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
torch.nn.Conv2d(16, 32, 3, padding=1), torch.nn.ReLU(), torch.nn.MaxPool2d(2),
torch.nn.Flatten(), torch.nn.Linear(32 * 12 * 12, 2))
def train(model, X, y, epochs=8):
opt = torch.optim.Adam(model.parameters(), lr=2e-3)
for _ in range(epochs):
for i in torch.randperm(len(X)).split(64):
loss = torch.nn.functional.cross_entropy(model(X[i]), y[i])
opt.zero_grad(); loss.backward(); opt.step()
return model
def accuracy(model, X, y):
return (model(X).argmax(1) == y).float().mean().item()Guarding against shift
Here's the habit. Before you collect data, list what might change. Time of day, season, fruit variety, dirty lenses, and a new camera. Collect examples of each one, if you can. Augment for the rest. Build separate test sets for each condition, and report them one by one. A single accuracy number hides exactly the failures that you care about.
Summary
Test across conditions, and augment for reality. Next, another trap. What happens when defects are rare?
Key takeaways- Domain shift kills vision systems
- Test every condition
- Augmentation teaches invariance