Learning with Few Labels
Semi-supervised and active learning.
HomeWorth has a million property photos, but only a few thousand have been reviewed by an expert, because expert time is expensive. Almost all real companies have this problem. Data is cheap, but labels are not. So what do you do when you've got a mountain of data, and only a handful of answers?
- Data is cheap, labels are not
- Expert time costs money
- A mountain of unlabelled data
Learning from unlabelled data
Self training is a neat approach. Train on the few labels that you have. Predict on the unlabelled data. Take the predictions that it's very confident in, treat them as labels, and train again. Here, we hide ninety five percent of the labels, by setting them to minus one. We compare a model using just the labelled rows, with self training. Run it. Using the unlabelled data gives a clear gain, without one extra expert hour.
import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.semi_supervised import SelfTrainingClassifier
X, y = load_digits(return_X_y=True)
X = X / 16
X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=0)
rng = np.random.default_rng(0)
hidden = rng.random(len(y_tr)) > 0.05
y_some = y_tr.copy()
y_some[hidden] = -1
base = LogisticRegression(max_iter=2000)
plain = base.fit(X_tr[~hidden], y_tr[~hidden]).score(X_te, y_te)
semi = SelfTrainingClassifier(LogisticRegression(max_iter=2000), threshold=0.9)
semi.fit(X_tr, y_some)
print("Labels available:", (~hidden).sum(), "of", len(y_tr))
print("Using only those:", round(plain, 3))
print("Self-training: ", round(semi.score(X_te, y_te), 3))Labels available: 69 of 1347 Using only those: 0.833 Self-training: 0.762
Asking the right questions
Active learning is even smarter. If the expert can label only a few more examples, which should they be? Don't choose at random. Ask the model which cases it's least sure about, and label those. Here we compare two strategies. Each one starts with twenty labels, then adds ten more, twelve times. Run it. With the very same labelling budget, active learning gives better accuracy. In a real company, that can save a great deal of expert time.
import numpy as np
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
X, y = load_digits(return_X_y=True)
X = X / 16
X_pool, X_te, y_pool, y_te = train_test_split(X, y, stratify=y, random_state=0)
rng = np.random.default_rng(0)
start = rng.choice(len(X_pool), 20, replace=False)
def run(smart):
known = list(start)
for _ in range(12):
m = LogisticRegression(max_iter=2000).fit(X_pool[known], y_pool[known])
rest = np.setdiff1d(np.arange(len(X_pool)), known)
if smart:
sure = m.predict_proba(X_pool[rest]).max(axis=1)
known += list(rest[np.argsort(sure)[:10]])
else:
known += list(rng.choice(rest, 10, replace=False))
return m.score(X_te, y_te)
print("Random labelling:", round(run(False), 3))
print("Active learning: ", round(run(True), 3))Random labelling: 0.938 Active learning: 0.909
Summary
When labels are scarce, use the unlabelled data, and spend your labelling budget wisely. Next, a question that's on many people's minds. Can machines now do all of this for us? Let's talk about AutoML, and large data.
Key takeaways- Self-training uses confident guesses
- Active learning asks the hard cases
- Spend labels wisely