Embeddings
How meaning becomes numbers.
A token number, like four thousand and twelve, tells the model nothing about meaning. The word king, and the word queen, should feel close. The word banana should feel far away. Embeddings solve this problem, and they are one of the most beautiful ideas in AI.
- A bare number carries no meaning
- Similar meanings should be close
- Embeddings: meaning as a position
A map of meaning
An embedding turns every token into a list of numbers, which you can imagine as a position on a map. Words with similar meanings land close together. Here is a simplified two dimensional picture. Animals gather in one region. Fruits in another. Cities in a third. Real models use hundreds, or thousands of dimensions, which we cannot draw, but the idea is the same. Closeness on the map means closeness in meaning.
Meaning has directions
The most amazing part is that directions on the map carry meaning too. Start from the word king, subtract the idea of man, and add the idea of woman. You arrive very close to queen. The model was never taught this. The relationship between king and queen, is the same direction as between man and woman. The maths of meaning emerges from the patterns in the data.
What embeddings make possible
Embeddings are used everywhere. Search engines use them to find results by meaning, not just by matching words. Recommendation systems use them, to suggest films similar to ones you like. And they are the foundation of a technique called retrieval, which lets a chatbot look up your own documents. Later in the course, you will build exactly that.
Summary
Excellent. An embedding turns a token into a list of numbers, a position on a map of meaning. Similar things land close together, and even relationships become directions. Next, a problem. Word order matters. How does a model know that dog bites man, is not the same as man bites dog?
Key takeaways- Embedding = position on a map of meaning
- Close in space = close in meaning
- Directions can carry relationships