Labels and Feedback Loops

A supervised model learns from labels, the right answers. Where do they come from? Sometimes from the business, a customer left, or did not. Sometimes from people, who label images, or text. Labels are expensive, and often noisy. And, once the model is live, it can change the very data you collect next. In this lesson, we manage labels, and avoid feedback traps.

• Labels are expensive and noisy
• People can label, with care
• Live models change the data

Where labels come from

There are four main sources. Natural labels, which the business produces anyway, like a purchase, a cancellation, or a loan repayment. They are cheap, but often delayed. Human labels, from experts, or crowd workers, which are accurate, but slow and costly. Weak labels, from rules, or other models, which are cheap and noisy, but can bootstrap a project. And user feedback, thumbs up, edits, and corrections, which arrives in production, and is excellent for improving models over time.

Labeling well

Good labeling is a process. Write clear guidelines, with examples of the tricky cases. Have several people label the same samples, and measure how much they agree. Low agreement means the task, or the guidelines, are unclear, and no model can fix that. Review disagreements, and update the guidelines. Start small, train a first model, and use it to find the examples where it is least certain. Labeling those first, which is called active learning, gets you the most improvement, from each label.

1. Clear guidelineswith tricky examples
2. Several labelersmeasure agreement
3. Review disagreementsimprove the guidelines
4. Active learninglabel the uncertain cases first

Feedback loops

The live model can poison its own future data. A recommender shows only what it predicts you will like, so it never learns about the rest. A fraud model blocks some transactions, so you never see whether they were fraud. A hiring model rejects candidates, whose outcomes you never observe. Fight these loops. Keep a small share of random decisions, for exploration. Log what the model did not choose. Compare against holdout groups, that never see the model. And remember, that the data collected after launch is no longer a neutral sample.

Summary