Tokens and Embeddings

A computer sees the word cat as nothing more than an arbitrary code. Nothing in the letters says that it's closer to dog than to car. But what if each word could be a point on a map, where words with similar meanings sit close together? That's an embedding, and it's the foundation of all modern language A I.

• Cat is just a code
• Put meaning on a map
• The foundation of language AI

Text to numbers

First, we need tokens. The simplest tokeniser splits text into single characters, and gives each one a number. We build two lookup tables, one in each direction. Run it. Hello world becomes a list of numbers, and we can turn it back again. Modern models work on word pieces instead of characters, but the idea is identical.

text = "hello world"
chars = sorted(set(text))
to_id = {c: i for i, c in enumerate(chars)}
to_char = {i: c for c, i in to_id.items()}

ids = [to_id[c] for c in text]

print("Vocabulary:", chars)
print("Encoded:   ", ids)
print("Decoded:   ", "".join(to_char[i] for i in ids))

Vocabulary: [' ', 'd', 'e', 'h', 'l', 'o', 'r', 'w']
Encoded:    [3, 2, 4, 4, 5, 0, 7, 5, 6, 4, 1]
Decoded:    hello world

The embedding layer

An embedding layer is simply a big table, with one row of learnable numbers for each token. Here, a thousand tokens, with four numbers each. We look up three tokens. Run it. Note that token seven gets the same vector both times. At the start, these are random. During training, gradients nudge them, so that tokens used in similar ways drift closer together. Meaning emerges from usage, with nobody explaining what a word means.

import torch
from torch import nn

embed = nn.Embedding(num_embeddings=1000, embedding_dim=4)

ids = torch.tensor([7, 42, 7])
vectors = embed(ids)

print("Shape:", list(vectors.shape))
print("Word 7, first time: ", vectors[0].tolist())
print("Word 7, second time:", vectors[2].tolist())

Shape: [3, 4]
Word 7, first time:  [1.3307901620864868, -0.9900457859039307, 0.04178701713681221, -1.5162359476089478]
Word 7, second time: [1.3307901620864868, -0.9900457859039307, 0.04178701713681221, -1.5162359476089478]

A map of meaning

Here's what trained embeddings look like, squeezed down to two dimensions. Cat and dog sit close together. Car and bus form their own group. And the directions mean something too. The step from man to woman is roughly the same as the step from king to queen. Nobody programmed this. It came from reading text.

Summary