Autoencoders

Learning by compressing.

⏱ 2 min read▶ Video at 1:14:55🎓 Free

Watch this lesson in the Deep Learning courseWatch on YouTube ↗Full Deep Learning course

Up to now, every network has needed a teacher, with labelled answers. But most of the world's data has no labels. Here's a neat trick. Give a network the job of copying its input, but force everything through a tiny bottleneck. To succeed, it has to discover what really matters.

Most data has no labels
Copy through a tiny bottleneck
Discover what really matters

Squeeze, then rebuild

An autoencoder has two halves. The encoder squeezes an image down into a tiny code, just a few numbers. The decoder tries to rebuild the full image, from that code alone. The target is simply the input itself. Because the code is so small, the network can't memorise everything. It has to learn the essence, such as which digit this is, its slant, and its thickness. That small code is a compressed summary of meaning.

Compressing digits

Here's a real autoencoder. Seven hundred and eighty four pixels go in, and just eight numbers come out of the encoder. The decoder expands those again. The loss compares the output to the input, and notice that the labels aren't used at all. Run it. The top row shows the originals, and the bottom row shows what it rebuilt, from only eight numbers. They're blurry, but recognisable. It compressed each image by nearly one hundred times, and nobody told it what a digit was.

autoencoder.pypython

import torch
from torch import nn
from torchvision import datasets, transforms
import matplotlib.pyplot as plt

root = "data"
train = datasets.MNIST(root, train=True, transform=transforms.ToTensor())
test = datasets.MNIST(root, train=False).data[:6].float().unsqueeze(1) / 255

class AE(nn.Module):
    def __init__(self, code=8):
        super().__init__()
        self.enc = nn.Sequential(nn.Flatten(), nn.Linear(784, 128), nn.ReLU(),
                                 nn.Linear(128, code))
        self.dec = nn.Sequential(nn.Linear(code, 128), nn.ReLU(),
                                 nn.Linear(128, 784), nn.Sigmoid())
    def forward(self, x):
        return self.dec(self.enc(x)).view(-1, 1, 28, 28)

torch.manual_seed(0)
model = AE()
opt = torch.optim.Adam(model.parameters(), lr=2e-3)
dl = torch.utils.data.DataLoader(train, batch_size=256, shuffle=True)
for epoch in range(4):
    for x, _ in dl:
        loss = ((model(x) - x) ** 2).mean()
        opt.zero_grad(); loss.backward(); opt.step()

with torch.no_grad():
    out = model(test)
fig, ax = plt.subplots(2, 6, figsize=(9, 3.2))
for i in range(6):
    ax[0, i].imshow(test[i, 0], cmap="gray"); ax[1, i].imshow(out[i, 0], cmap="gray")
    ax[0, i].axis("off"); ax[1, i].axis("off")
plt.tight_layout()
plt.savefig("rebuilt.png", dpi=120)
print(f"784 numbers squeezed into 8. Final error {loss.item():.4f}")

Output

784 numbers squeezed into 8. Final error 0.0235

Summary

Bottlenecks force a network to find what matters, without any labels. But there's a more practical use for this idea. What if we taught a network to repair damage? That one idea is the seed of the image generators that amazed the world.

Key takeaways

Encoder squeezes, decoder rebuilds
No labels needed
The code captures the essence

Next up: Denoising — teaching a network to repair

❮ From Tiny to Giant Denoising ❯