NLP · Lesson 70 of 106
Clustering Text
Why embeddings changed everything.
Your manager asks what customers are complaining about this month. You've got fifty thousand messages, and no labels. You could read them all. Or you could let the computer group them. This is also the clearest demonstration I know of why embeddings changed the field.
- What are customers complaining about?
- Fifty thousand messages, no labels
- Let the computer group them
Word counts struggle
We ignore the labels, and cluster our tickets into three groups using TF-IDF. Then we check how well those groups match the true topics. A score of one is a perfect match, and zero is random. Run it. Point zero five. Basically random. Look at the words in each group. They're a mixed up jumble. Short messages share very few words, so word counts can't see that they're about the same thing.
cluster_words.pypython
import numpy as np
from sklearn.cluster import KMeans
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics import adjusted_rand_score
from tickets import TICKETS
texts, y = zip(*TICKETS)
vec = TfidfVectorizer(stop_words="english")
X = vec.fit_transform(texts)
km = KMeans(3, n_init=20, random_state=0).fit(X)
names = np.array(vec.get_feature_names_out())
for c in km.cluster_centers_:
print(", ".join(names[np.argsort(-c)[:4]]))
print("Agreement with true labels:", round(adjusted_rand_score(y, km.labels_), 2))Output
refund, pages, order, copy money, book, want, returned delivery, parcel, came, rude Agreement with true labels: 0.05
Also uses tickets.py
TICKETS = [
("my order arrived late and the box was crushed", "delivery"),
("the parcel never came, where is my delivery", "delivery"),
("courier left the package in the rain", "delivery"),
("tracking has not moved for a week", "delivery"),
("shipping took three weeks, far too slow", "delivery"),
("delivery driver was rude and threw the parcel", "delivery"),
("package is lost, nobody knows where it went", "delivery"),
("the courier says delivered but nothing arrived", "delivery"),
("i want my money back for this book", "refund"),
("please refund the second copy you charged me for", "refund"),
("you took payment twice, return my money", "refund"),
("how long does a refund take to reach my card", "refund"),
("cancel my order and give me a refund", "refund"),
("i returned the book but got no money back", "refund"),
("charged the wrong price, want the difference back", "refund"),
("refund has not appeared on my statement", "refund"),
("the book has missing pages in chapter four", "quality"),
("cover is torn and the spine is broken", "quality"),
("pages are printed upside down in the middle", "quality"),
("ink is smudged on every page", "quality"),
("the binding fell apart after one read", "quality"),
("this copy has a coffee stain on the cover", "quality"),
("wrong edition, and the print quality is awful", "quality"),
("several pages are stuck together badly", "quality"),
]
QUERIES = [
("where is my parcel, it has not arrived", "delivery"),
("i need my money back please", "refund"),
("the pages are torn and the cover is bent", "quality"),
("courier lost my package", "delivery"),
("refund the payment to my card", "refund"),
("print is smudged and hard to read", "quality"),
]Embeddings nail it
Now the very same clustering method, but on sentence embeddings. Only one thing's changed, the way that we represent the text. Run it. One point zero. A perfect match with the true topics, with no labels and no training. From almost random to perfect, simply by using a better representation. That's why embeddings changed everything. And to name your clusters, you can send a few messages from each one to a language model.
cluster_embed.pypython
from sentence_transformers import SentenceTransformer
from sklearn.cluster import KMeans
from sklearn.metrics import adjusted_rand_score
from tickets import TICKETS
texts, y = zip(*TICKETS)
model = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")
X = model.encode(list(texts), normalize_embeddings=True)
labels = KMeans(3, n_init=20, random_state=0).fit_predict(X)
print("Agreement with true labels:", round(adjusted_rand_score(y, labels), 2))Output
Agreement with true labels: 1.0
Also uses tickets.py
TICKETS = [
("my order arrived late and the box was crushed", "delivery"),
("the parcel never came, where is my delivery", "delivery"),
("courier left the package in the rain", "delivery"),
("tracking has not moved for a week", "delivery"),
("shipping took three weeks, far too slow", "delivery"),
("delivery driver was rude and threw the parcel", "delivery"),
("package is lost, nobody knows where it went", "delivery"),
("the courier says delivered but nothing arrived", "delivery"),
("i want my money back for this book", "refund"),
("please refund the second copy you charged me for", "refund"),
("you took payment twice, return my money", "refund"),
("how long does a refund take to reach my card", "refund"),
("cancel my order and give me a refund", "refund"),
("i returned the book but got no money back", "refund"),
("charged the wrong price, want the difference back", "refund"),
("refund has not appeared on my statement", "refund"),
("the book has missing pages in chapter four", "quality"),
("cover is torn and the spine is broken", "quality"),
("pages are printed upside down in the middle", "quality"),
("ink is smudged on every page", "quality"),
("the binding fell apart after one read", "quality"),
("this copy has a coffee stain on the cover", "quality"),
("wrong edition, and the print quality is awful", "quality"),
("several pages are stuck together badly", "quality"),
]
QUERIES = [
("where is my parcel, it has not arrived", "delivery"),
("i need my money back please", "refund"),
("the pages are torn and the cover is bent", "quality"),
("courier lost my package", "delivery"),
("refund the payment to my card", "refund"),
("print is smudged and hard to read", "quality"),
]Summary
Better representations beat better algorithms. Next, can you sort messages with no training at all?
Key takeaways- Word counts fail on short text
- Embeddings capture meaning
- Representation matters most
Next up: Classifying with No Training — describe the labels instead