Checking Gradients by Hand

Everything in deep learning rests on one thing. The gradient. If it's wrong, your model fails quietly, and it won't give you any error message at all. So how can you know that a gradient is right? There's a beautifully simple test, and it goes all the way back to school maths. Nudge the input, and see what happens to the output.

• Everything rests on the gradient
• Wrong gradients fail quietly
• Nudge it, and watch

Autograd against arithmetic

Here's a small function of two weights. Autograd gives us the slope, using backward. Then we work it out a completely different way. We nudge one weight up by a tiny amount, then down, and we see how much the output changes. That change, divided by the size of the nudge, is the slope. No calculus involved at all. Run it. Both ways give eight and sixteen. Exactly the same. That's a gradient check. If you ever write a custom layer, or a custom loss, do this once. It takes five minutes, and it can save you a week. Notice that we use sixty four bit numbers here, because tiny nudges need high precision.

import torch

def f(w):
    return ((w * w).sum() - 3) ** 2

w = torch.tensor([1.0, 2.0], requires_grad=True, dtype=torch.float64)
f(w).backward()

numeric = torch.zeros(2, dtype=torch.float64)
h = 1e-6
for i in range(2):
    step = torch.zeros(2, dtype=torch.float64); step[i] = h
    numeric[i] = (f(w.detach() + step) - f(w.detach() - step)) / (2 * h)

print("Autograd:", w.grad.tolist())
print("By hand: ", [round(v, 4) for v in numeric.tolist()])

Autograd: [8.0, 16.0]
By hand:  [8.0, 16.0]

When to check

You don't need this for standard layers, as PyTorch's are well tested. Use it when you've written something new. A custom loss, a custom layer, or anything which calls detach, because that's a common way of accidentally cutting off the gradient. PyTorch even has a built in tool for this, called grad check. Check on small inputs, so that it stays fast.

Summary