Bag of Words

Here's an idea that seems far too simple to work. Take a document. Throw away the order of its words, as if you'd tipped them all into a bag. Count what's there. It sounds crude, and yet it powered search engines and spam filters for decades, and you'll still use it today.

• Tip the words into a bag
• Count what's there
• It powered search for decades

Three documents as numbers

Count vectorizer does the work. It learns the vocabulary across all of the documents, and gives each word a column. Then each document becomes a row of counts. Run it. Look at the bottom row. The third document says great twice, so that column shows a two. Each document is now a list of numbers, so all of machine learning is open to us.

from sklearn.feature_extraction.text import CountVectorizer

docs = ["the book arrived late",
        "the late book was great",
        "great service and great price"]

vec = CountVectorizer()
X = vec.fit_transform(docs)

print(vec.get_feature_names_out())
print(X.toarray())

['and' 'arrived' 'book' 'great' 'late' 'price' 'service' 'the' 'was']
[[0 1 1 0 1 0 0 1 0]
 [0 0 1 1 1 0 0 1 1]
 [1 0 0 2 0 1 1 0 0]]

Keeping a little order

The obvious weakness is that order is gone. Not good and good are nearly identical in a bag. A simple fix is to count pairs of words as well, called bigrams. Run it. The pairs version now has a column for not good, which can carry a totally different meaning. The vocabulary grows quickly, but it's a cheap and effective trick.

from sklearn.feature_extraction.text import CountVectorizer

docs = ["not good at all", "good not bad"]

one = CountVectorizer(ngram_range=(1, 1)).fit(docs)
two = CountVectorizer(ngram_range=(1, 2)).fit(docs)

print("Single words:", list(one.get_feature_names_out()))
print("With pairs:  ", list(two.get_feature_names_out()))

Single words: ['all', 'at', 'bad', 'good', 'not']
With pairs:   ['all', 'at', 'at all', 'bad', 'good', 'good at', 'good not', 'not', 'not bad', 'not good']

Summary