Experiments and Tests

Three weeks ago, you got a great result. But which settings gave you that? Which data? Which version of the code? If you can't answer that, you don't really have a result. You've only got a memory.

• A great result, three weeks ago
• Which settings? Which data?
• You only have a memory

Recording every run

At the very least, each run should record its settings, its score, the code version, and the date. Here's the simplest possible version. Run it. Now you can always find your best run. Tools like Weights and Biases, MLflow, and TensorBoard do this automatically, and they draw live charts of your training. Use one from day one.

import json, time, random

random.seed(0)
runs = []
for lr in [1e-2, 1e-3, 3e-4]:
    run = {"lr": lr, "val_accuracy": round(0.86 + random.random() / 20, 3),
           "commit": "a1b2c3d", "time": "2026-11-02"}
    runs.append(run)

best = max(runs, key=lambda r: r["val_accuracy"])
print(json.dumps(best))
print("Runs recorded:", len(runs))

{"lr": 0.01, "val_accuracy": 0.902, "commit": "a1b2c3d", "time": "2026-11-02"}
Runs recorded: 3

Tests for models

Models need tests too, since bugs here are silent. These three tests run in seconds. Does the output have the right shape? Do gradients reach every layer? Are there any bad numbers? Run it. All pass. Add these to your project, and they'll catch many of the silent bugs we talked about, before you waste hours on a broken training run.

import torch
from torch import nn

model = nn.Sequential(nn.Linear(4, 8), nn.ReLU(), nn.Linear(8, 3))

def test_output_shape():
    assert model(torch.rand(2, 4)).shape == (2, 3)

def test_gradients_flow():
    model(torch.rand(2, 4)).sum().backward()
    assert all(p.grad is not None for p in model.parameters())

def test_no_bad_numbers():
    assert torch.isfinite(model(torch.rand(2, 4))).all()

for t in [test_output_shape, test_gradients_flow, test_no_bad_numbers]:
    t()
    print("passed", t.__name__)

passed test_output_shape
passed test_gradients_flow
passed test_no_bad_numbers

Summary