GANs

In twenty fourteen, Ian Goodfellow was celebrating with friends at a bar in Montreal, when they began arguing about how to make a computer generate realistic pictures. He went home, and coded his idea that same night. It worked on the first try. He had invented generative adversarial networks.

• A bar in Montreal, 2014
• Coded the same night
• It worked first time

The forger and the detective

A GAN has two networks. The generator is a forger, turning random numbers into fake images. The discriminator is a detective, shown real and fake images, and it learns to tell them apart. They train against each other. As the detective improves, the forger has to improve, in order to fool it. Round after round, the fakes get better, and eventually the detective can't do any better than a coin flip.

Training a forger

Here's a real GAN. G turns sixty four random numbers into an image, and D gives a score for real or fake. In each step, two things happen. First, train the detective, showing it real images labelled real, and fakes labelled fake. Detach stops the gradient, so that this step doesn't change the forger. Second, train the forger. Its loss says, I want the detective to call this real. Run it. After just five epochs, it's forging digits, and it has never directly seen a single one of the training images. It only ever got feedback through the detective.

import torch
from torch import nn
from torchvision import datasets, transforms
import matplotlib.pyplot as plt

tf = transforms.Compose([transforms.ToTensor(), transforms.Normalize([0.5], [0.5])])
train = datasets.MNIST("data", train=True, transform=tf)
dl = torch.utils.data.DataLoader(train, batch_size=128, shuffle=True, drop_last=True)

torch.manual_seed(0)
G = nn.Sequential(nn.Linear(64, 256), nn.LeakyReLU(0.2), nn.Linear(256, 512),
                  nn.LeakyReLU(0.2), nn.Linear(512, 784), nn.Tanh())
D = nn.Sequential(nn.Flatten(), nn.Linear(784, 512), nn.LeakyReLU(0.2),
                  nn.Linear(512, 256), nn.LeakyReLU(0.2), nn.Linear(256, 1))
opt_g = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_d = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
bce = nn.BCEWithLogitsLoss()
one, zero = torch.ones(128, 1), torch.zeros(128, 1)

for epoch in range(5):
    for real, _ in dl:
        fake = G(torch.randn(128, 64)).view(-1, 1, 28, 28)
        loss_d = bce(D(real), one) + bce(D(fake.detach()), zero)
        opt_d.zero_grad(); loss_d.backward(); opt_d.step()

        loss_g = bce(D(fake), one)
        opt_g.zero_grad(); loss_g.backward(); opt_g.step()

with torch.no_grad():
    new = G(torch.randn(16, 64)).view(-1, 28, 28)
fig, ax = plt.subplots(2, 8, figsize=(9, 2.6))
for a, img in zip(ax.ravel(), new):
    a.imshow(img, cmap="gray"); a.axis("off")
plt.tight_layout()
plt.savefig("gan.png", dpi=120)
print("Forged by the generator after 5 epochs")

Forged by the generator after 5 epochs

The GAN era and its limits

GANs led to images so sharp that they fooled people, such as those faces of people who don't exist. But they're notoriously hard to train. The two networks must stay in balance. And they can suffer from mode collapse, when the forger finds one trick that works, and only produces that. These problems opened the door for a calmer approach.

Summary