Attention

In twenty seventeen, a research paper with a bold title, Attention Is All You Need, changed artificial intelligence. The idea inside it, attention, is the reason today's models understand language so well. In this lesson, you will see how attention works, with no maths at all.

• The 2017 paper that changed AI
• Attention: look at what matters
• No maths needed

Which word does "it" mean?

Read this sentence. The animal did not cross the street, because it was too tired. What does the word it refer to? The animal, of course. You knew that instantly, because you paid attention to the right word. A model must do the same. For every word, attention decides how much to look at every other word. Here, when the model processes the word it, most of its attention flows to animal, a little to street, and almost none to the rest.

Change one word, change the focus

Now watch what happens, if we change the last word. The animal did not cross the street, because it was too wide. Now, it refers to the street. A good model shifts its attention, to street, with a high weight. The same word, it, pays attention to different words, depending on the context. That flexibility is what makes language models so powerful.

Queries, keys and values

How does the model compute attention? With a clever trick, using three roles. Every word creates a query, which asks, what am I looking for? Every word also creates a key, which says, here is what I contain. The model compares each query with every key, and the best matches get high attention scores. Finally, each word also carries a value, its actual content. The model blends the values of the words it attends to, weighted by the scores.

1. Querywhat am I looking for?
2. Keywhat do I contain?
3. Matchscore each query against each key
4. Blend valuesmix content by the scores

Many heads are better than one

One attention pattern is not enough. Language has many kinds of relationships at once. So models run many attention heads, in parallel. One head may track grammar, which verb belongs to which noun. Another may track meaning, which pronoun refers to which person. Another may track nearby words. Each head learns its own job, during training, and their results are combined.

Summary