Active Learning

Labels are expensive. An expert's hour, a customer survey, a manual review. If you can label only a few hundred examples, which ones should you choose? Choosing at random is the default. But there is a smarter way. Label the examples, that the current model is most unsure about. This is active learning, and it can save a great deal of effort.

• Labels cost real money
• Choose which to label
• Ask about the unsure cases

The active learning loop

The loop is simple. Train a model on the few labels you have. Use it to predict the unlabelled pool, and find the examples where its confidence is lowest, the ones near the decision boundary. Send those to the labelers. Add the new labels to the training set, and train again. Repeat. Each round, the labels go where they teach the model the most, instead of being spent on easy cases, that it already understands.

1. Train on current labelsa small model
2. Find the unsure caseslowest confidence
3. Label thosehumans answer
4. Retrain and repeatlabels go where they help

Random versus uncertain

Here is an experiment. We have a pool of three thousand customers, with labels we can buy, a few at a time. We start with thirty labels. In each of six rounds, we train a model, and measure its F one score on a separate test set. Then we buy twenty five more labels. In the random strategy, we pick them blindly. In the uncertain strategy, we pick the customers whose predicted probability is closest to a half. The table shows both. With the same number of labels, the uncertain strategy gives a clearly better F one score, in most rounds. The labels were spent where they mattered.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
from churn import make_data

pool, test = make_data(3000, seed=1), make_data(2000, seed=2)
features = ["tenure", "charges", "calls", "contract"]
Xp, yp = pool[features].to_numpy(), pool["churn"].to_numpy()
Xt, yt = test[features].to_numpy(), test["churn"].to_numpy()

def run(strategy, rounds=6, step=25):
    rng = np.random.default_rng(0)
    labelled = list(rng.choice(len(Xp), 30, replace=False))
    scores = []
    for _ in range(rounds):
        model = LogisticRegression(max_iter=500).fit(Xp[labelled], yp[labelled])
        scores.append(f1_score(yt, model.predict(Xt)))
        unlabelled = np.setdiff1d(np.arange(len(Xp)), labelled)
        if strategy == "random":
            picks = rng.choice(unlabelled, step, replace=False)
        else:
            confidence = np.abs(model.predict_proba(Xp[unlabelled])[:, 1] - 0.5)
            picks = unlabelled[np.argsort(confidence)[:step]]
        labelled += list(picks)
    return scores

print("labels   random F1   uncertain F1")
random_scores, smart_scores = run("random"), run("uncertain")
for i, (a, b) in enumerate(zip(random_scores, smart_scores)):
    print(f"{30 + 25 * i:6d}   {a:.3f}    {b:.3f}")

labels   random F1   uncertain F1
    30   0.336    0.336
    55   0.316    0.381
    80   0.216    0.348
   105   0.156    0.331
   130   0.248    0.327
   155   0.211    0.256

import numpy as np, pandas as pd

def make_data(n, seed, shifted=False):
    rng = np.random.default_rng(seed)
    tenure = rng.integers(1, 72, n)
    charges = rng.normal(65 + (28 if shifted else 0), 18, n).clip(20, 140).round(2)
    calls = rng.poisson(1.5 + (1.5 if shifted else 0), n)
    contract = rng.choice([0, 1, 2], n, p=[.55, .25, .20] if not shifted else [.30, .30, .40])
    logit = -1.0 - 0.04 * tenure + 0.025 * (charges - 65) + 0.45 * calls - 0.8 * contract
    if shifted:
        logit = -0.2 - 0.01 * tenure - 0.01 * (charges - 65) + 0.2 * calls - 0.2 * contract
    churn = (rng.random(n) < 1 / (1 + np.exp(-logit))).astype(int)
    return pd.DataFrame({"tenure": tenure, "charges": charges, "calls": calls, "contract": contract, "churn": churn})

When active learning helps

Active learning shines in some situations. When labels are expensive, and the pool of unlabelled data is large. When the classes are imbalanced, because uncertain cases often include the rare class. When the task is stable, so that labels stay valid. It helps less when labels are cheap, or when the model's confidence is poorly calibrated, so that uncertainty is a bad guide. Keep some random samples in each batch, to avoid the model only learning about the borderline, and to keep an unbiased test set.

Summary