Backpropagation

Imagine a company misses its target. The chief executive asks each manager, how much of this was your fault? Each manager asks their own team. The blame flows down through the company, until every person knows how much to change. That's backpropagation, and it's arguably the most important algorithm of the last forty years.

• A company misses its target
• Blame flows down the layers
• Everyone learns what to change

Forward, then backward

In the forward pass, data flows from left to right, and the network makes a guess. We compare it to the truth. Then comes the backward pass. Starting from the end, the error flows back through each layer. At each connection, the chain rule works out how much that weight contributed. In a single backward sweep, every weight gets its own gradient. That's why it's so efficient. The cost of one sweep is about the same as one forward pass, even with billions of weights.

The chain rule by hand

Let's do one by hand, so that it's never mysterious again. A tiny network with two weights. W one feeds h, h feeds p, and p feeds the loss. The chain rule says, to get the effect of w one on the loss, multiply the local slopes along the path. Each one is simple. Run it. Our hand calculations match autograd exactly. Now you know what loss dot backward does. It just multiplies small slopes, along every path in the graph.

import torch

x, target = torch.tensor(2.0), torch.tensor(1.0)
w1, w2 = torch.tensor(0.5), torch.tensor(-0.3)

h = w1 * x
p = w2 * h
loss = (p - target) ** 2

dL_dp = 2 * (p - target)
dp_dw2 = h
dp_dh = w2
dh_dw1 = x

print("dL/dw2 by hand:", (dL_dp * dp_dw2).item())
print("dL/dw1 by hand:", (dL_dp * dp_dh * dh_dw1).item())

w1a = w1.clone().requires_grad_()
w2a = w2.clone().requires_grad_()
((w2a * (w1a * x) - target) ** 2).backward()
print("Autograd:", w2a.grad.item(), w1a.grad.item())

dL/dw2 by hand: -2.5999999046325684
dL/dw1 by hand: 1.5600000619888306
Autograd: -2.5999999046325684 1.5600000619888306

When the blame fades away

That multiplying has a dark side. If each slope is smaller than one, then multiplying many of them gives a number close to zero. The early layers receive almost no signal. This is the vanishing gradient problem, and it blocked deep networks for years. We build twelve layer networks with two activations, and measure the gradient at the first, middle, and last layers. Run it. With sigmoid, the first layer gets a tiny fraction of the signal at the end. With ReLU, the gradient survives. That's a big reason why ReLU changed everything.

import torch
from torch import nn

def gradient_sizes(activation):
    torch.manual_seed(0)
    layers = []
    for _ in range(12):
        layers += [nn.Linear(32, 32), activation()]
    net = nn.Sequential(*layers, nn.Linear(32, 1))
    net(torch.rand(64, 32)).mean().backward()
    return [net[i].weight.grad.abs().mean().item() for i in (0, 10, 22)]

for name, act in [("Sigmoid", nn.Sigmoid), ("ReLU", nn.ReLU)]:
    first, middle, last = gradient_sizes(act)
    print(f"{name:<8} first {first:.2e}  middle {middle:.2e}  last {last:.2e}")

Sigmoid  first 1.80e-12  middle 4.47e-08  last 1.24e-02
ReLU     first 3.75e-07  middle 4.52e-06  last 2.08e-03

Summary