Label the Hard Ones First

The packhouse has two million unlabelled apple photos, and the budget pays for just five thousand labels. Most of those photos show an easy, perfect apple, and labelling them teaches the model almost nothing. If you could choose which images get labelled, which ones would you pick?

• Two million photos
• Five thousand labels
• Choose wisely

Random against hardest first

We start with thirty random labelled digits. Then, in each round, we train a model, and we buy twenty more labels, until we have a hundred and fifty. There's only one difference between the two runs. The random run picks at random. The smart run asks the model how sure it is about each unlabelled image, and it picks the twenty that it's least sure about. Run it. With the same labelling budget, random reaches ninety one point four percent, and hardest first reaches ninety three point four. That's about a quarter fewer mistakes, for no extra money.

import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)
X_pool, X_test, y_pool, y_test = train_test_split(X, y, test_size=500, random_state=0)

def run(smart):
    rng = np.random.default_rng(1)
    picked = list(rng.choice(len(X_pool), 30, replace=False))
    while len(picked) < 150:
        model = LogisticRegression(max_iter=2000).fit(X_pool[picked], y_pool[picked])
        left = np.setdiff1d(np.arange(len(X_pool)), picked)
        if smart:
            sure = model.predict_proba(X_pool[left]).max(axis=1)
            picked += list(left[np.argsort(sure)[:20]])
        else:
            picked += list(rng.choice(left, 20, replace=False))
    model = LogisticRegression(max_iter=2000).fit(X_pool[picked], y_pool[picked])
    return model.score(X_test, y_test)

print("Random labels:   ", round(run(False), 3))
print("Hardest first:   ", round(run(True), 3))

Random labels:    0.914
Hardest first:    0.934

Traps in active learning

This idea has real risks. The model can focus on images that are simply blurry, and impossible to label. So filter those out. It can pick twenty near identical images, so ask for variety too. And your labelled set no longer matches the real world, so never use it to measure performance. Keep a separate test set, chosen at random. Early on, when the model knows nothing, its sense of confidence isn't worth much, so start with a random batch.

A labelling loop that never stops

In a real system, this becomes a loop. The model runs in production. Images that it's unsure about are saved. People label those, and the new labels go into the next training run. Each round, the model gets stronger in exactly the places where it was weak. That's how AppleEye keeps improving after launch.

1. Run the modelin production
2. Save unsure imageslow confidence
3. People label themhard cases
4. Retrainstronger where weak

Three ways to measure unsure

We used the simplest idea, the highest probability. There are others. Entropy looks at the whole spread of probabilities, and it helps when you have many classes. Margin looks at the gap between the top two answers. And disagreement trains a few models, and picks the images that they argue about, which is often the strongest method, although it costs more. For detectors, you can score each box, and then pick the images with the most doubtful boxes.

Did it pay off?

Don't assume that active learning has helped. Run a small test. Split your labelling budget, and spend half at random, and half with your chosen method. Then compare both on the same random test set, and note the cost per label too. Hard images often take people longer, and they may disagree more about them. Sometimes, cheap random labels turn out to be the better deal.

Summary