Regularisation

Geoffrey Hinton was in a bank, when he noticed that the tellers kept changing. He asked why. They said, it stops us from working together to cheat the bank. That gave him an idea. What if we randomly removed neurons during training, so that they couldn't rely on each other?

• A story from a bank
• Tellers kept rotating
• Remove neurons at random

Dropout in action

Watch this network. At each training step, a random half of the hidden neurons are switched off. The rest must do the work, without leaning on any particular colleague. Every neuron has to learn something useful by itself. It's a bit like training a huge crowd of smaller networks, that all share weights. At test time, all of the neurons switch back on, and their answers blend together.

Fighting overfitting

Here's a big network, on only eight hundred images, a perfect recipe for overfitting. We train it three ways. With nothing to help. With dropout. And with dropout and weight decay, which gently shrinks all of the weights. Run it. With no help, the training score is nearly perfect, but the test score is much lower. Dropout closes that gap, and raises the test score. This is why these tools are in almost every network.

import torch
from torch import nn
from torchvision import datasets, transforms

root = "data"
tf = transforms.ToTensor()
train = datasets.FashionMNIST(root, train=True, transform=tf)
test = datasets.FashionMNIST(root, train=False, transform=tf)
small = torch.utils.data.Subset(train, range(800))
X_te = test.data.float().unsqueeze(1) / 255

def run(drop, decay):
    torch.manual_seed(0)
    model = nn.Sequential(nn.Flatten(), nn.Linear(784, 512), nn.ReLU(),
                          nn.Dropout(drop), nn.Linear(512, 512), nn.ReLU(),
                          nn.Dropout(drop), nn.Linear(512, 10))
    opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=decay)
    dl = torch.utils.data.DataLoader(small, batch_size=64, shuffle=True)
    for _ in range(40):
        model.train()
        for x, y in dl:
            loss = nn.functional.cross_entropy(model(x), y)
            opt.zero_grad(); loss.backward(); opt.step()
    model.eval()
    with torch.no_grad():
        tr = torch.stack([small[i][0] for i in range(800)])
        ty = torch.tensor([small[i][1] for i in range(800)])
        return ((model(tr).argmax(1) == ty).float().mean().item(),
                (model(X_te).argmax(1) == test.targets).float().mean().item())

for name, d, wd in [("No regularisation", 0.0, 0.0),
                    ("Dropout", 0.5, 0.0),
                    ("Dropout + decay", 0.5, 0.05)]:
    tr, te = run(d, wd)
    print(f"{name:<18} train {tr:.3f}  test {te:.3f}")

No regularisation  train 0.990  test 0.789
Dropout            train 0.979  test 0.796
Dropout + decay    train 0.984  test 0.800

The whole toolbox

You have many tools against overfitting. Get more data, which is always the best option. Use data augmentation, to create new variations of your images. Add dropout, and weight decay. Stop early. Use a smaller model. A good workflow is to first make your model large enough to overfit, proving that it can learn, and then add these tools one at a time until the gap closes.

Summary