Position and Context

Look at these two sentences. The dog bit the man. The man bit the dog. Same words, completely different meaning. And the word bank, can mean a river, or a place for money. Meaning depends on position, and on context. In this lesson, you will see how models handle both.

• Same words, different order
• Same word, different meanings
• Models must track position and context

Order changes meaning

Embeddings alone would treat both sentences the same, because they contain the same words. So models add a second ingredient, called positional information. Each token receives a signal, that says, I am the first word, I am the second word, and so on. Now the model can tell who bit whom. The meaning of a sentence, is the words, and their order, together.

One word, many meanings

Context solves the second problem. Take the word bank. In, I sat on the river bank, it means land beside water. In, I deposited money at the bank, it is a building with a vault. The model must look at the surrounding words, to decide. This idea of looking around, to understand each word, is exactly what the transformer does, and it is our next topic.

The reading pipeline so far

Let's put together what we have learned. Raw text comes in. It is split into tokens. Each token becomes a number, a token I D. Each number becomes an embedding, a position on the map of meaning. Position signals are added, so order is known. And now the model has a rich set of numbers, ready for the real thinking, which happens in the transformer.

1. Raw text"The cat sat"
2. TokensThe | cat | sat
3. Token IDs464 | 3797 | 3332
4. Embeddingsa position in meaning-space
5. + Positionwho comes first, second, third

Summary