Word2Vec from Scratch
Watching meaning appear.
This is one of the most satisfying programs that you'll ever run. We'll start with random numbers. We won't tell the computer what any word means. And by the end, it will have worked out that a king is like a queen, and a dog is like a cat, using nothing but which words appear together.
- Start with random numbers
- Never explain any word
- Meaning appears anyway
Training Word2Vec
Here's a tiny text about royalty and pets, repeated many times. Each word starts with a random vector. For each word, we look at its neighbours, within two places. We push the vectors of real neighbours closer together, and we push a few random words away. That's called negative sampling. The update lines are simple gradient descent. Run it. Look at that. The closest word to king is queen. The closest word to dog is cat. We never told it that. It learned this purely from the company that those words keep.
import numpy as np
text = ("the king rules the land . the queen rules the land . "
"the dog chased the cat . the cat chased the dog . "
"a king and a queen wear a crown . a dog and a cat are pets . ") * 40
words = text.split()
vocab = sorted(set(words))
ids = {w: i for i, w in enumerate(vocab)}
V, D = len(vocab), 16
rng = np.random.default_rng(0)
W = rng.normal(0, 0.1, (V, D))
C = rng.normal(0, 0.1, (V, D))
sig = lambda z: 1 / (1 + np.exp(-z))
for epoch in range(3):
for i, w in enumerate(words):
for j in range(max(0, i - 2), min(len(words), i + 3)):
if j == i:
continue
a, b = ids[w], ids[words[j]]
for t, label in [(b, 1)] + [(rng.integers(V), 0) for _ in range(3)]:
g = sig(W[a] @ C[t]) - label
W[a], C[t] = W[a] - 0.05 * g * C[t], C[t] - 0.05 * g * W[a]
norm = W / np.linalg.norm(W, axis=1, keepdims=True)
def near(w):
s = norm @ norm[ids[w]]
return [vocab[i] for i in s.argsort()[::-1] if vocab[i] != w and vocab[i] not in ("the", "a", ".", "and")][:2]
for w in ["king", "dog"]:
print(f"closest to {w}: {near(w)}")closest to king: ['queen', 'dog'] closest to dog: ['cat', 'king']
The famous analogy
Embeddings can even do arithmetic with meaning. Start with king, take away man, and add woman. What's the closest word? Using a modern model, we test two analogies. Run it. King minus man plus woman gives queen. And Paris minus France plus Italy gives Rome. A direction in this space can mean something like gender, or capital city of. That result stunned researchers in twenty thirteen.
from sentence_transformers import SentenceTransformer
import numpy as np
m = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")
words = ["king", "queen", "man", "woman", "paris", "france", "rome", "italy", "banana"]
v = dict(zip(words, m.encode(words, normalize_embeddings=True)))
def closest(x, skip):
return max((w for w in words if w not in skip), key=lambda w: v[w] @ x)
print("king - man + woman =", closest(v["king"] - v["man"] + v["woman"], ["king", "man"]))
print("paris - france + italy =", closest(v["paris"] - v["france"] + v["italy"], ["paris", "france"]))king - man + woman = woman paris - france + italy = italy
Summary
You watched meaning emerge from nothing but word company. That's one of the deepest ideas in AI. But embeddings for single words aren't enough. A customer sends a whole sentence. Next, embedding sentences, and fixing Scribe's search.
Key takeaways- Pull neighbours together
- Push random words away
- Directions carry meaning