Sequences and Memory

Read this sentence. The movie was not bad. Now try reading those same words, in a random order. Bad, was, movie, the, not. The meaning has gone. Order carries meaning. And a network that looks at everything at once has no idea which word came first. Aria needs a memory.

• Order carries meaning
• Shuffled words mean nothing
• Aria needs a memory

A network with a memory

A recurrent neural network reads one word at a time. It keeps a small vector, called the hidden state, which is its memory. For each new word, it combines that word with its memory, and writes a new memory. Then it passes that along to the next step. By the end of the sentence, the memory holds a summary of everything so far. Notice that the very same network is reused at every step, so it can handle sentences of any length.

Shapes in an RNN

In PyTorch, it's one line. Our input has two sentences, five words each, and eight numbers for each word. The R N N returns two things. Output has the memory at every step. Hidden has only the final memory. Run it. Five steps, with sixteen numbers of memory each. For classifying a whole sentence, you'd use the final memory. To generate text, you'd use every step.

import torch
from torch import nn

rnn = nn.RNN(input_size=8, hidden_size=16, batch_first=True)

x = torch.rand(2, 5, 8)   # 2 sentences, 5 words, 8 numbers per word
output, hidden = rnn(x)

print("Input:         ", list(x.shape))
print("Each step:     ", list(output.shape))
print("Final memory:  ", list(hidden.shape))

Input:          [2, 5, 8]
Each step:      [2, 5, 16]
Final memory:   [1, 2, 16]

Summary