Labelling with AI
Getting labels cheaply.
Labelling ten thousand tickets by hand would take weeks, and would cost a fortune. But a language model can do a first pass in an hour, for a few dollars. Then a person only needs to check. That's changed how NLP projects get started.
- Ten thousand labels by hand?
- An LLM first pass in an hour
- People check the hard ones
A model as labeller
We ask for a label and a confidence score, using a fixed schema. Run it. Three tickets are labelled, quickly and cleanly. Do this across your whole dataset, and you've got training data for a cheap, fast classifier. That's a powerful pattern. Use the big model to teach a small one, then run the small model at high volume.
import json
from google import genai
from google.genai import types
from pydantic import BaseModel
class Label(BaseModel):
label: str
confidence: float
client = genai.Client()
texts = ["where is my book", "the zip broke on my bag", "love this shop!"]
for t in texts:
r = client.models.generate_content(
model="gemini-flash-latest",
contents=f"Label as delivery, product, praise, or other: {t}",
config=types.GenerateContentConfig(response_mime_type="application/json",
response_schema=Label))
print(t, "->", json.loads(r.text))where is my book -> {'label': 'delivery', 'confidence': 0.98}
the zip broke on my bag -> {'label': 'product', 'confidence': 0.98}
love this shop! -> {'label': 'praise', 'confidence': 0.99}Checking the labels
You can't simply trust those labels. So take a random sample, have a person label it too, and compare. Run it. Agreement and kappa show how far you can trust it, and the confusion matrix shows where it goes wrong. In this sample, refunds and deliveries get mixed up. So now you know exactly what to improve in your prompt.
from sklearn.metrics import cohen_kappa_score, confusion_matrix
human = ["refund", "delivery", "refund", "question", "delivery", "refund", "question", "delivery"]
model = ["refund", "delivery", "delivery", "question", "delivery", "refund", "question", "refund"]
print("Agreement:", sum(a == b for a, b in zip(human, model)) / len(human))
print("Kappa:", round(cohen_kappa_score(human, model), 2))
print(confusion_matrix(human, model, labels=["delivery", "question", "refund"]))Agreement: 0.75 Kappa: 0.62 [[2 0 1] [0 2 0] [1 0 2]]
Asking about the hard ones
A person's time is precious, so spend it well. Train a quick model on what you have. Then look at the examples where it's the least sure, and ask a person about those. Run it. The three least certain messages rise to the top. Labelling these improves the model far more than labelling easy ones. This is active learning, and it can save huge amounts of effort.
import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression
enc = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")
labelled = [("parcel never came", 0), ("where is my order", 0),
("refund me", 1), ("money back please", 1)]
pool = ["the courier lost it", "charged twice", "love your shop",
"is it in stock", "give my cash back", "the delivery man left it outside"]
X = enc.encode([t for t, _ in labelled])
y = [l for _, l in labelled]
clf = LogisticRegression().fit(X, y)
sure = clf.predict_proba(enc.encode(pool)).max(1)
for i in sure.argsort()[:3]:
print(f"{sure[i]:.2f} ask a human: {pool[i]}")0.51 ask a human: charged twice 0.53 ask a human: is it in stock 0.54 ask a human: love your shop
Summary
Let a big model label, check it with people, and focus humans on the hard cases. Next, three traps that catch experienced engineers.
Key takeaways- LLMs give fast first-pass labels
- Check a random sample
- Humans label the hardest cases