Positions Matter
Telling attention about order.
Attention has a blind spot. It looks at all of the words at once, as a set. So to attention, dog bites man, and man bites dog, contain exactly the same information. We threw away the step by step reading, and with it, we lost word order. We have to put it back.
- Attention treats words as a set
- Dog bites man?
- We must add order back
Adding position
The fix is simple. We give each position its own learned vector, just like each word has one. Then we add the two together. Word five at place zero, and word five at place two, now have different vectors. Run it. They're not the same, even though it's the same word. The original transformer used sine waves for this, but learned positions work well, and they're easy to write.
import torch
from torch import nn
vocab, block, dim = 100, 16, 8
tok = nn.Embedding(vocab, dim)
pos = nn.Embedding(block, dim)
ids = torch.tensor([[5, 9, 5]])
places = torch.arange(3)
x = tok(ids) + pos(places)
print("Shape:", list(x.shape))
print("Same word, different place, same vector?",
torch.equal(x[0, 0], x[0, 2]))Shape: [1, 3, 8] Same word, different place, same vector? False
Summary
Word embedding, plus position embedding. Attention now knows what each word is, and where it is. We've got all of the parts. Let's assemble the real thing.
Key takeaways- Attention ignores order by itself
- Add a position vector
- Same word, different place, different vector