Sequence to Sequence
Teaching a machine to translate.
Bright Books wants customers in Spain. Translating isn't word by word. The output could be longer or shorter, and in a different order. In twenty fourteen, a neat idea cracked this. One network reads the whole sentence. A second one writes the translation. Together, they're called sequence to sequence.
- Translation isn't word by word
- Different length, different order
- One reads, one writes
A tiny translator
Here's a tiny translator, working on six word pairs, one letter at a time. The encoder reads the English word, and it boils it down to one memory vector. That vector is handed to the decoder, which writes the Spanish, one letter at a time. Training shows the decoder the right letters. Run it. It correctly translates the words that it was trained on. Real translators used this same idea, on millions of sentences.
import torch
from torch import nn
PAIRS = [("hello", "hola"), ("thanks", "gracias"), ("book", "libro"),
("house", "casa"), ("water", "agua"), ("friend", "amigo")]
chars = sorted(set("".join(a + b for a, b in PAIRS))) + ["<", ">"]
ix = {c: i for i, c in enumerate(chars)}
enc_e, dec_e = nn.Embedding(len(chars), 32), nn.Embedding(len(chars), 32)
enc, dec = nn.GRU(32, 64, batch_first=True), nn.GRU(32, 64, batch_first=True)
out = nn.Linear(64, len(chars))
params = [p for m in (enc_e, dec_e, enc, dec, out) for p in m.parameters()]
opt = torch.optim.Adam(params, lr=0.01)
T = lambda s: torch.tensor([[ix[c] for c in s]])
torch.manual_seed(0)
for step in range(400):
a, b = PAIRS[step % len(PAIRS)]
_, h = enc(enc_e(T(a)))
y, _ = dec(dec_e(T("<" + b)), h)
loss = nn.functional.cross_entropy(out(y)[0], T(b + ">")[0])
opt.zero_grad(); loss.backward(); opt.step()
def translate(word):
_, h = enc(enc_e(T(word)))
s, c = "", "<"
for _ in range(10):
y, h = dec(dec_e(T(c)), h)
c = chars[out(y)[0, -1].argmax()]
if c == ">":
break
s += c
return s
for w in ["hello", "water", "friend"]:
print(w, "->", translate(w))hello -> hola water -> agua friend -> amigo
Summary
An encoder reads, a decoder writes. But this has a serious flaw. The entire sentence has to squeeze through one small vector. For a long sentence, that's impossible. The fix was an idea called attention, and it changed the field forever.
Key takeaways- Encoder reads, decoder writes
- One vector holds the whole sentence
- Long sentences lose detail