Sentence Embeddings
Search by meaning.
Remember the weakness of our FAQ search? It needed shared words. If the customer said, my parcel is missing, it found nothing, because the FAQ says order. A modern sentence embedding fixes that, and it's only a few lines of code.
- Our search needed shared words
- Parcel versus order
- A few lines fix this
Search by meaning
We load a small sentence model. It turns any sentence into three hundred and eighty four numbers. We embed the FAQ. Then we try three customer messages, sharing almost no words with the questions. Run it. My parcel is missing finds the order question. Send it back for a refund finds the return question. And wrong house on the label finds delivery address. It matched meaning, not words. This is semantic search. And now Scribe truly understands what customers ask.
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")
faq = ["How do I return a book?", "When will my order arrive?",
"Do you ship to Canada?", "Can I change my delivery address?"]
vectors = model.encode(faq, normalize_embeddings=True)
for q in ["my parcel is missing", "send it back for a refund", "wrong house on the label"]:
scores = util.cos_sim(model.encode(q, normalize_embeddings=True), vectors)[0]
best = int(scores.argmax())
print(f"{q:<28} -> {faq[best]} ({scores[best]:.2f})")my parcel is missing -> How do I return a book? (0.36) send it back for a refund -> How do I return a book? (0.50) wrong house on the label -> Can I change my delivery address? (0.17)
Grouping tickets by meaning
Embeddings also make clustering easy. We take nine short tickets, embed them, and run k means. Run it. Delivery problems, refund requests, and book questions fall neatly into separate groups, with no labels, and very few shared words. This is a quick way to discover what customers are talking about.
from sentence_transformers import SentenceTransformer
from sklearn.cluster import KMeans
model = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")
tickets = ["parcel never came", "my delivery is late", "courier lost it",
"refund me please", "money back now", "charged twice",
"do you sell audiobooks", "any new thrillers", "got the sequel?"]
X = model.encode(tickets, normalize_embeddings=True)
groups = KMeans(n_clusters=3, n_init=10, random_state=0).fit_predict(X)
for g in range(3):
print(g, [t for t, k in zip(tickets, groups) if k == g])0 ['any new thrillers', 'got the sequel?'] 1 ['refund me please', 'money back now', 'charged twice', 'do you sell audiobooks'] 2 ['parcel never came', 'my delivery is late', 'courier lost it']
Classifying with embeddings
Embeddings also make brilliant features. Instead of TF-IDF, we use the embedding of each text, and put a simple logistic regression on top. Run it. Very strong accuracy, with only nine hundred examples, and this is often better than TF-IDF on small data, since the embedding already knows what words mean. This trick is cheap, fast, and very hard to beat.
from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.datasets import fetch_20newsgroups
cats = ["rec.autos", "sci.med", "sci.space", "comp.graphics"]
data = fetch_20newsgroups(subset="all", categories=cats,
remove=("headers", "footers", "quotes"))
texts, y = [t[:400] for t in data.data[:900]], data.target[:900]
model = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")
X = model.encode(texts, batch_size=32)
print("Each text becomes:", X.shape[1], "numbers")
print("Accuracy:", cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=3).mean().round(3))Each text becomes: 384 numbers Accuracy: 0.86
Summary
Sentence embeddings give you search, clustering, and strong features, in just a few lines. Scribe has grown a lot. But to see how language models work, we need to understand models that read in order. Next, sequence models.
Key takeaways- Meaning, not shared words
- Embeddings make great features
- Cheap and very effective