Finding Phrases

New York is one idea, written as two words. If your system treats new and york separately, then it loses that meaning. The same goes for hot dog, machine learning, and customer service. How could a computer discover, all by itself, which words belong together?

• New York is one idea
• Two words, one meaning
• Discover them automatically

Pointwise mutual information

The trick is to compare what we see with what we'd expect by chance. If new and york appear together far more often than their separate counts would predict, then they're linked. That's called pointwise mutual information. Run it. New york was seen fifteen times, with a score of two and a half. Old book scores higher still, although it's rarer. The new appears five times, and scores lower, because the is everywhere, so seeing them together isn't surprising. Counting alone misleads, but PMI corrects for how common each word is. Run this on your support tickets, and you'll discover the phrases that your customers really use.

import math
from collections import Counter

text = ("new york is big . new york has many book shops . the old book was new . "
        "they went to new york . a good book in york . the new shop has an old book . ") * 5
w = text.split()
one, two = Counter(w), Counter(zip(w, w[1:]))
N = len(w)

def pmi(a, b):
    return math.log2((two[(a, b)] / N) / (one[a] / N * one[b] / N))

for a, b in [("new", "york"), ("old", "book"), ("the", "new")]:
    print(f"{a} {b}: seen {two[(a, b)]:>2}, PMI {pmi(a, b):.2f}")

new york: seen 15, PMI 2.51
old book: seen 10, PMI 3.25
the new: seen  5, PMI 1.93

Using phrases

Once you've found them, join those words into a single token. Add them to your search, so that a search for new york won't return old york. Feed them to your topic models. And note that a very rare pair can score highly by chance, so always set a minimum count. This idea also shows up in word embeddings, as they're closely related to these same statistics.

Summary