TF-IDF

If a word appears ten times in one document, it's probably important to that document. But if it also appears in every other document, it's useless. The best words are common here, and rare elsewhere. That single insight, from nineteen seventy two, is still inside search engines today.

• Common here, rare elsewhere
• An idea from 1972
• Still in search engines

Scoring words

TF means term frequency, how often a word appears here. IDF means inverse document frequency, which is higher for rare words. We multiply them together. Here are three short documents. We look at the highest scoring words in the third. Run it. Wonderful, and gift score the highest, because they're unique to this document. The word the sits near the bottom, though it's in every document. That's exactly what we want.

from sklearn.feature_extraction.text import TfidfVectorizer

docs = ["the book arrived late",
        "the refund arrived late",
        "the book was a wonderful gift"]

vec = TfidfVectorizer()
X = vec.fit_transform(docs).toarray()
names = vec.get_feature_names_out()

top = sorted(zip(X[2], names), reverse=True)[:4]
for score, word in top:
    print(f"{word:<10} {score:.3f}")
print("Score of 'the':", round(X[2][list(names).index("the")], 3))

wonderful  0.505
was        0.505
gift       0.505
book       0.384
Score of 'the': 0.298

Searching the FAQ

Now Scribe can use this to find answers. Here's a small FAQ. We turn each question into a TF-IDF vector, and the customer's message too. Cosine similarity measures the angle between two vectors. Smaller angle, more similar. Run it. It matched the customer's question to the right FAQ, using the shared words order and arrive. That's a search engine, in ten lines. But notice its limit. If the customer said, my parcel is missing, there'd be no shared words, and it would fail. We'll fix this later with embeddings.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

faq = ["How do I return a book?",
       "When will my order arrive?",
       "Do you ship to Canada?",
       "Can I change my delivery address?"]

vec = TfidfVectorizer(stop_words="english")
X = vec.fit_transform(faq)

question = vec.transform(["my order has not arrived yet"])
scores = cosine_similarity(question, X)[0]

for s, q in sorted(zip(scores, faq), reverse=True)[:2]:
    print(f"{s:.2f}  {q}")

0.71  When will my order arrive?
0.00  How do I return a book?

Summary