Labelling Well

Two experienced fruit graders look at the same apple. One says bruised. The other says fine. If your experts disagree with each other, then what's your model supposed to learn? Most teams spend months on architectures, when they should be spending a week on their labels.

• Two experts disagree
• What should the model learn?
• Fix the labels first

A labelling process

Start by writing clear rules, with pictures of borderline examples. Label a small batch, with two people working separately. Measure how often they agree. Discuss each disagreement, and improve the rules. Only then label the rest. And keep sampling, to make sure that quality doesn't drift.

1. Write ruleswith examples
2. Double labela small batch
3. Measure agreementfind confusion
4. Fix the rulesthen label all
5. Keep checkingquality drifts

Getting more from fewer labels

Labels cost real money, so use them well. Pre-label with your current model, as correcting is quicker than drawing. Sort by loss, and review the examples which the model finds most surprising, as many of those are label mistakes. And use active learning, where people label the images that the model's least sure about.

Tools you can use

You don't need to build this yourself. CVAT and Label Studio are popular open source tools. Roboflow covers the whole workflow, with hosted options. And models such as SAM can draw a mask from one click, which cuts the time for segmentation labels enormously.

Summary