Word Embeddings

Here's the problem that held NLP back for fifty years. To a computer, the words parcel and package have nothing in common. Every search, every classifier, had to treat them as totally separate things. Then, in twenty thirteen, a simple idea changed everything.

• Parcel and package: no connection
• A fifty-year problem
• 2013: a simple idea

Why counting fails

In a bag of words, each word is its own column. That's called one hot. Each word is a one in its own position, and zeros everywhere else. Now we measure how similar two words are. Run it. Parcel and package score zero. Parcel and banana also score zero. To this system, every pair of words is equally unrelated. There's no meaning in here at all.

import numpy as np

words = ["parcel", "package", "banana"]
vectors = np.eye(3)

for i, w in enumerate(words):
    print(f"{w:<8} {vectors[i].astype(int).tolist()}")

print("parcel vs package:", vectors[0] @ vectors[1])
print("parcel vs banana: ", vectors[0] @ vectors[2])

parcel   [1, 0, 0]
package  [0, 1, 0]
banana   [0, 0, 1]
parcel vs package: 0.0
parcel vs banana:  0.0

A map of words

What if each word was a point on a map, placed so that similar words sit close together? Parcel sits beside package. Book sits beside novel. Banana and apple are off somewhere else. Now every word gets a list of a few hundred numbers. These numbers are called an embedding, and they give the computer its first real sense of meaning.

You shall know a word by its company

So where do these numbers come from? In nineteen fifty seven, the linguist JR Firth said, you shall know a word by the company it keeps. Parcel and package appear next to the same kinds of words, like delivered, shipped, and arrived. So if we train a network to predict a word's neighbours, words used alike get similar vectors. Word two vec, in twenty thirteen, did exactly this.

Summary