Smarter Hyperparameter Search

You've got five settings to tune, and four options for each. A full grid means over a thousand runs. Nobody can afford that. It turns out that the obvious method, a tidy grid, is much worse than simply choosing at random. And there's a good reason why.

• Five settings, four options each
• Over a thousand runs
• Random beats a grid

Random search

We try six random learning rates. Notice that we sample on a log scale, using ten to the power of a random number. That's vital, as point zero zero one and point zero one are as different to a network as one and ten. Run it. Six runs, and now we know the shape of the problem. Rates around point zero one do best here, and tiny ones do far worse. Why does random beat a grid? Usually only one or two settings matter. A grid wastes runs by repeating the same few values. Random tries a new value of every setting, in every run.

import random, torch
from base import load, net, fit, score

X, y, Xt, yt = load(3000)
random.seed(0)
results = []
for trial in range(6):
    lr = 10 ** random.uniform(-4, -1.5)
    torch.manual_seed(trial)
    results.append((score(fit(net(), X, y, lr=lr, epochs=2), Xt, yt), lr))

for acc, lr in sorted(results, reverse=True):
    print(f"lr {lr:.4f}: accuracy {acc:.3f}")

lr 0.0129: accuracy 0.876
lr 0.0079: accuracy 0.874
lr 0.0019: accuracy 0.859
lr 0.0011: accuracy 0.848
lr 0.0010: accuracy 0.842
lr 0.0004: accuracy 0.785

import torch
from torchvision import datasets, transforms

def load(n=6000):
    t = transforms.ToTensor()
    a = datasets.MNIST("data", True, t, download=False)
    b = datasets.MNIST("data", False, t, download=False)
    return a.data[:n].float()[:, None] / 255, a.targets[:n], b.data[:2000].float()[:, None] / 255, b.targets[:2000]

def net():
    return torch.nn.Sequential(torch.nn.Flatten(), torch.nn.Linear(784, 128), torch.nn.ReLU(),
                               torch.nn.Linear(128, 10))

def fit(model, X, y, lr=1e-3, epochs=3, opt=None):
    opt = opt or torch.optim.Adam(model.parameters(), lr=lr)
    for _ in range(epochs):
        for i in torch.randperm(len(X)).split(64):
            loss = torch.nn.functional.cross_entropy(model(X[i]), y[i])
            opt.zero_grad(); loss.backward(); opt.step()
    return model

def score(model, X, y):
    return (model(X).argmax(1) == y).float().mean().item()

A tuning order

With limited time, tune in this order. First, the learning rate, as it matters most. Then the batch size and the model's size. Then regularisation, such as weight decay and dropout. Then the schedule. Leave everything else at its default. Tools like Optuna take this further, by concentrating on promising areas, and by stopping bad runs early.

Summary