Attention

In twenty seventeen, eight researchers at Google published a paper, with a cheeky title. Attention is all you need. They threw away recurrent networks entirely. No memory cell, no step by step reading. Every A I chatbot that you've ever used is built on what was in that paper. Today, you're going to understand it.

• 2017: eight researchers at Google
• Attention is all you need
• Behind every chatbot

What attention means

Read this sentence. The cat sat because it was tired. What does the word it refer to? You know instantly that it's the cat. Your mind looks back, and pays attention to the most relevant word. Attention gives a network that same power. When processing the word it, the model looks at all of the other words at once, gives each one a score, and blends in the most relevant. Here, cat gets the highest weight. So the meaning of it gets filled in by cat.

Attention in six lines

Attention is shockingly simple. Three words, each as a vector. Step one, compare every word with every other, using a dot product. Larger means more similar. Step two, softmax turns each row into weights that add up to one. Step three, each word becomes a weighted blend of all the words. Run it. Look at the row for cat. It gets most of its new meaning from words that resemble it. That's self attention, and it's all just a matrix product.

import torch

words = ["the", "cat", "sat"]
x = torch.tensor([[1.0, 0.0], [0.9, 0.3], [0.2, 1.0]])

scores = x @ x.T
weights = scores.softmax(dim=1)
blended = weights @ x

print("Scores:\n", scores.round(decimals=2))
print("Weights:\n", weights.round(decimals=2))
print("New 'cat':", blended[1].round(decimals=2).tolist())

Scores:
 tensor([[1.0000, 0.9000, 0.2000],
        [0.9000, 0.9000, 0.4800],
        [0.2000, 0.4800, 1.0400]])
Weights:
 tensor([[0.4200, 0.3800, 0.1900],
        [0.3800, 0.3800, 0.2500],
        [0.2200, 0.2900, 0.5000]])
New 'cat': [0.7599999904632568, 0.36000001430511475]

Summary