Classifying with No Training
Describe the labels instead.
It's launch day, and there's no labelled data yet. You need to start routing messages right now. Here's a trick that seems like cheating. Don't train anything. Just describe each category in a sentence, and see which description each message is closest to.
- Launch day, no labels
- You need routing now
- Describe each category
Labels as sentences
Each label gets a short description. Note that these are full sentences, rather than just the word refund, as sentences give a better embedding. We embed them. Then for each new message, we pick the closest description. Run it. Six out of six, with no training examples at all. To add a new category tomorrow, you simply write one more sentence. Now you can begin with this on day one, and collect real labels as you go, then later train a proper classifier.
from sentence_transformers import SentenceTransformer
from tickets import QUERIES
model = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")
labels = {"delivery": "a problem with shipping or a parcel",
"refund": "the customer wants their money back",
"quality": "the book itself is damaged or badly printed"}
L = model.encode(list(labels.values()), normalize_embeddings=True)
keys = list(labels)
right = 0
for text, answer in QUERIES:
guess = keys[(L @ model.encode([text], normalize_embeddings=True)[0]).argmax()]
right += guess == answer
print(f"{guess:<9} {text}")
print(f"Correct with no training: {right}/{len(QUERIES)}")delivery where is my parcel, it has not arrived refund i need my money back please quality the pages are torn and the cover is bent delivery courier lost my package refund refund the payment to my card quality print is smudged and hard to read Correct with no training: 6/6
Also uses tickets.py
TICKETS = [
("my order arrived late and the box was crushed", "delivery"),
("the parcel never came, where is my delivery", "delivery"),
("courier left the package in the rain", "delivery"),
("tracking has not moved for a week", "delivery"),
("shipping took three weeks, far too slow", "delivery"),
("delivery driver was rude and threw the parcel", "delivery"),
("package is lost, nobody knows where it went", "delivery"),
("the courier says delivered but nothing arrived", "delivery"),
("i want my money back for this book", "refund"),
("please refund the second copy you charged me for", "refund"),
("you took payment twice, return my money", "refund"),
("how long does a refund take to reach my card", "refund"),
("cancel my order and give me a refund", "refund"),
("i returned the book but got no money back", "refund"),
("charged the wrong price, want the difference back", "refund"),
("refund has not appeared on my statement", "refund"),
("the book has missing pages in chapter four", "quality"),
("cover is torn and the spine is broken", "quality"),
("pages are printed upside down in the middle", "quality"),
("ink is smudged on every page", "quality"),
("the binding fell apart after one read", "quality"),
("this copy has a coffee stain on the cover", "quality"),
("wrong edition, and the print quality is awful", "quality"),
("several pages are stuck together badly", "quality"),
]
QUERIES = [
("where is my parcel, it has not arrived", "delivery"),
("i need my money back please", "refund"),
("the pages are torn and the cover is bent", "quality"),
("courier lost my package", "delivery"),
("refund the payment to my card", "refund"),
("print is smudged and hard to read", "quality"),
]Summary
Describing labels gets you started on day one. But embeddings have learned from us, and that includes our biases.
Key takeaways- Describe labels in sentences
- Pick the closest
- Add categories instantly