The Birth of Attention

Imagine translating a long paragraph, after reading it once, without being allowed to look back at it. It's impossible. A real translator keeps glancing at the original. In twenty fourteen, researchers gave networks that same ability. They called it attention.

• Translating without looking back?
• Translators keep glancing
• Networks learned to glance

Glancing back

At each step of writing, the decoder looks back at all of the source words. It gives each one a score, for how relevant it is right now. When it's writing the word libro, it pays the most attention to book. The result is a custom summary, for each new word. There's no bottleneck any more. The model can look at whatever it needs.

Attention in code

Attention is only a few lines. The decoder asks a question, as a vector. We compare it with each source word, using a dot product. Softmax turns those into weights. And the blended context goes to the decoder. Run it. Book gets by far the most attention, because it best matches the decoder's question. In our Deep Learning course, we built the whole transformer from this idea, and if you haven't seen that, it's a great companion to this course.

import torch

encoder = torch.tensor([[1.0, 0.1], [0.2, 1.0], [0.9, 0.2]])
words = ["the", "book", "arrived"]
decoder = torch.tensor([0.1, 1.0])

scores = encoder @ decoder
weights = scores.softmax(0)
context = weights @ encoder

for w, a in zip(words, weights):
    print(f"{w:<8} attention {a:.2f}")
print("Context vector:", context.round(decimals=2).tolist())

the      attention 0.23
book     attention 0.52
arrived  attention 0.25
Context vector: [0.5600000023841858, 0.5899999737739563]

Summary