Keywords and Extractive Summaries
Finding the key points.
Bright Books gets a forty page report from a supplier, and nobody has time to read it. Before reaching for an expensive model, ask whether a simpler method will do the job. Often, the key points can be found with no generation at all.
- A forty page report
- Nobody has time
- Simple methods first
Extracting keywords
TF-IDF gives us keywords for free. For any document, the top scoring words and pairs are what make it different from the rest. We print the top six for one document. Run it. In a few milliseconds, we get a fair list of what it's about, and it never invents anything. These keywords are great for tagging and for boosting search.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.datasets import fetch_20newsgroups
data = fetch_20newsgroups(subset="train", categories=["sci.space", "sci.med"],
remove=("headers", "footers", "quotes")).data
vec = TfidfVectorizer(stop_words="english", ngram_range=(1, 2), max_df=0.4, min_df=3)
X = vec.fit_transform(data)
words = vec.get_feature_names_out()
doc = 3
top = X[doc].toarray()[0].argsort()[-6:][::-1]
print("Document:", data[doc][:90].replace("\n", " "))
print("Keywords:", [words[i] for i in top])Document: It depends on the cause of the pneumonia. For treating bacterial pneumonia in young othe Keywords: ['pneumonia', 'erythromycin', 'pathogens', 'bacterial', 'antibiotic', 'depends']
TextRank summaries
TextRank applies Google's PageRank idea to sentences. Each sentence votes for the ones that it resembles. A sentence that's similar to many others is probably central. We build a similarity table, and run the same simple update, thirty times. Then we keep the top two, in their original order. Run it. We get a short summary, made of the document's real sentences. Since it only copies, it can't invent facts. For legal or medical text, that safety is very valuable.
import numpy as np
from sentence_transformers import SentenceTransformer
text = ("I ordered three books for my daughter. The site promised two days. "
"After a week nothing had arrived. I phoned and waited forty minutes. "
"The agent was polite. When the box finally came one book was torn. "
"Another book was the wrong title. I want a refund for both books.")
sentences = [s.strip() + "." for s in text.split(".") if s.strip()]
model = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")
v = model.encode(sentences, normalize_embeddings=True)
sim = v @ v.T
rank = np.ones(len(sentences)) / len(sentences)
walk = sim / sim.sum(1, keepdims=True)
for _ in range(30):
rank = 0.15 / len(sentences) + 0.85 * walk.T @ rank
top = sorted(rank.argsort()[-2:])
print(" ".join(sentences[i] for i in top))I ordered three books for my daughter. When the box finally came one book was torn.
Finding duplicate tickets
Here's a simple and valuable trick. When a delivery van breaks down, hundreds of customers write in, in different words. Embeddings can group them together. We compare all of the tickets, and we join anything above a similarity threshold. Run it. Each group is one real problem. The team can now fix one issue, instead of reading hundreds of messages.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")
tickets = ["My parcel has not arrived", "The package never came",
"Where is my delivery?", "Do you sell audiobooks?",
"Can I get this as an audiobook?"]
v = model.encode(tickets, normalize_embeddings=True)
sim = v @ v.T
seen = set()
for i in range(len(tickets)):
if i in seen:
continue
group = [j for j in range(len(tickets)) if sim[i, j] > 0.55]
seen.update(group)
print([tickets[j] for j in group])['My parcel has not arrived', 'Where is my delivery?'] ['The package never came'] ['Do you sell audiobooks?', 'Can I get this as an audiobook?']
Summary
Keywords, extractive summaries, and de-duplication, all cheap and safe. Text can give us more than summaries though. It can give us facts. Next, we'll turn documents into a map of how things connect.
Key takeaways- TF-IDF gives keywords
- TextRank copies key sentences
- Embeddings group duplicates