Labels That Survive

Last week, somebody bumped the belt camera at the packhouse. It's now tilted by a few degrees, and AppleEye has started making mistakes. Nobody changed the model, and the apples look the same. You can't photograph every possible tilt. But you can teach the model that a tilt doesn't matter.

• The camera got bumped
• Nothing else changed
• Teach what doesn't matter

Tilted test, tilted training

Here are real handwritten digits, and we train on only three hundred of them. This function tilts an image by a chosen angle. Our test images are all tilted by twenty degrees, which is just what a bumped camera does. First, a plain model. Then we make five versions of every training image, at different angles, and we train again. Run it. Without augmentation, accuracy falls to about seventy one percent. With it, we reach about ninety percent, with no new photos at all. We didn't collect anything. We only told the model what doesn't matter.

import cv2
import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

def turn(img, degrees):
    M = cv2.getRotationMatrix2D((3.5, 3.5), degrees, 1)
    return cv2.warpAffine(img.astype(np.float32), M, (8, 8))

X, y = load_digits(return_X_y=True)
X = X.reshape(-1, 8, 8)
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=300, random_state=0)
tilted = np.array([turn(x, 20) for x in X_test]).reshape(len(X_test), -1)

plain = LogisticRegression(max_iter=3000).fit(X_train.reshape(300, -1), y_train)
print("No augmentation:  ", round(plain.score(tilted, y_test), 3))

more = [turn(x, a) for x in X_train for a in (-20, -10, 0, 10, 20)]
more = np.array(more).reshape(1500, -1)
smart = LogisticRegression(max_iter=3000).fit(more, np.repeat(y_train, 5))
print("With augmentation:", round(smart.score(tilted, y_test), 3))

No augmentation:   0.712
With augmentation: 0.896

Changes that keep the label

Augmentation only works if the label stays true. An apple, turned upside down, is still an apple, so flips are safe. A digit is different. Flip a six and a nine gets confusing, and a flipped two isn't a digit at all. Heavy colour changes would destroy a ripeness grader. And a crop can cut away the only bruise in the picture. Before you add any change, ask whether a person would still give the same label.

Random or matched to reality?

Most tutorials copy the same long list of random changes. It's better to look at your own camera. What really varies? For a belt, that's lighting, small turns, and a little blur. It's not snow, and it's not mirror images of text. So match your augmentation to the changes that you can expect, and measure each one on a held out set.

Moving the labels too

For a classifier, a flip leaves the label alone. Detection and segmentation are harder. If you turn the image, then every box must turn too, and a turned box usually grows larger than the object. A mask needs the same change, with nearest neighbour resizing, so that you don't invent new classes along the edges. Keypoints are the trickiest. If you flip a person, the left wrist becomes the right wrist, so those labels must swap. Always draw a few augmented samples, and check them with your own eyes.

Augmenting at test time

There's another trick. At prediction time, show the model several versions of one image, perhaps the original and a flip. Then average the answers. It often gives a small but real gain in accuracy, and it costs several times the compute. So it suits a batch job, or a medical review, where each decision matters a great deal. It's rarely worth it on a live belt, where you've only got milliseconds.

Summary