Sequence Models

Embeddings know what words mean. But they don't know the order of a sentence. For that, we need a model that reads left to right, just as you do. For years, recurrent networks were the kings of language, and understanding them explains exactly why transformers were needed.

• Embeddings ignore order
• Read left to right
• Why transformers were needed

A network that reads

A recurrent network reads one word at a time. It keeps a memory vector, and updates it with every word. By the end of the sentence, the memory holds a summary. That's perfect for classification. A clever upgrade is to read in both directions, called bidirectional, since a word's meaning often depends on what comes after it.

A bidirectional LSTM

Here's a classic text model. Each token gets an embedding. An LSTM reads it in both directions. We join the final forward and backward memories into a summary. And a linear head gives three class scores. Run it. Each word gets one hundred and twenty eight numbers, sixty four for each direction. This design ruled text classification and NER for many years, until transformers came along.

import torch
from torch import nn

embed = nn.Embedding(1000, 32)
lstm = nn.LSTM(32, 64, batch_first=True, bidirectional=True)
head = nn.Linear(128, 3)

ids = torch.randint(0, 1000, (2, 12))
out, (h, c) = lstm(embed(ids))

summary = torch.cat([h[0], h[1]], dim=1)
print("Each word, both directions:", list(out.shape))
print("Sentence summary:", list(summary.shape))
print("Class scores:", list(head(summary).shape))

Each word, both directions: [2, 12, 128]
Sentence summary: [2, 128]
Class scores: [2, 3]

Summary