Zipf's Law

In the nineteen thirties, a linguist named George Zipf noticed something odd. Take any large text, in any language, and rank the words by frequency. The second most common word appears about half as often as the first. The third, about a third as often. It's a pattern that shows up nearly everywhere.

• A pattern in every language
• Second word: half as common
• Third word: a third as common

Testing the law

Let's test it, on real medical discussion posts. We count all of the words, then print their rank, and rank times count. If Zipf is right, that product should stay roughly constant. Run it. Across a thousand times the rank, the numbers stay in the same ballpark. And look at the last line. A huge share of all the different words appear only one time.

from collections import Counter
from sklearn.datasets import fetch_20newsgroups
import re

text = " ".join(fetch_20newsgroups(subset="train", categories=["sci.med"]).data)
words = re.findall(r"[a-z']+", text.lower())
freq = Counter(words).most_common()

print("Total words:", len(words), " Unique:", len(freq))
for rank in [1, 2, 10, 100, 1000]:
    w, n = freq[rank - 1]
    print(f"rank {rank:<5} {w:<10} count {n:<7} rank x count {rank * n}")

once = sum(1 for _, n in freq if n == 1)
print(f"Words seen only once: {once / len(freq):.0%}")

Total words: 180983  Unique: 14962
rank 1     the        count 6856    rank x count 6856
rank 2     of         count 4560    rank x count 9120
rank 10    it         count 1893    rank x count 18930
rank 100   disease    count 210     rank x count 21000
rank 1000  business   count 23      rank x count 23000
Words seen only once: 40%

Why this matters to you

This shapes everything in NLP. A few words make up most of your text, and they carry little meaning. A huge number of rare words make up your vocabulary, and each one appears too rarely to learn from. So any model will always meet words that it's never seen. That's called the long tail, and it's one reason why tokenisers break rare words into pieces.

Summary