Autograd

In our Machine Learning course, we worked out gradients by hand, for a model with two weights. Modern networks have billions of weights. Nobody can do that by hand. So in a sense, the greatest invention in deep learning isn't any network at all. It's a tool that computes every slope for you, automatically.

• Two weights by hand is fine
• Billions is impossible
• Automatic slopes

One line of magic

Requires grad tells PyTorch to watch this number. We compute a loss, in plain code. Then we call backward. That's it. PyTorch now knows the slope of the loss with respect to w, and it stores it in w dot grad. Run it. The slope is minus sixteen. And when we work it out by hand, using the chain rule, we get the same answer. You'll never need to do that by hand again.

import torch

w = torch.tensor(3.0, requires_grad=True)

loss = (w * 2 - 10) ** 2
loss.backward()

print("Loss:", loss.item())
print("Slope at w = 3:", w.grad.item())
print("By hand: 2 * (2w - 10) * 2 =", 2 * (2 * 3 - 10) * 2)

Loss: 16.0
Slope at w = 3: -16.0
By hand: 2 * (2w - 10) * 2 = -16

Gradient descent with autograd

Now we can put it to work, in a real training loop. We compute the loss, call backward, then step the weight downhill. No grad tells PyTorch not to track the update itself. And zero_clears the gradient, because otherwise, each new gradient would pile on top of the last. Forgetting this is a classic bug. Run it. W marches towards five, the answer, and the loss falls to nearly zero.

import torch

w = torch.tensor(0.0, requires_grad=True)

for step in range(30):
    loss = (w * 2 - 10) ** 2
    loss.backward()
    with torch.no_grad():
        w -= 0.05 * w.grad
    w.grad.zero_()
    if step % 7 == 0:
        print(f"step {step:>2}  w = {w.item():.3f}  loss = {loss.item():.3f}")

step  0  w = 2.000  loss = 100.000
step  7  w = 4.916  loss = 0.078
step 14  w = 4.998  loss = 0.000
step 21  w = 5.000  loss = 0.000
step 28  w = 5.000  loss = 0.000

The computation graph

How does it work? Each time you do an operation, PyTorch quietly records it, building a graph of everything that happened. Here, x goes through a multiply, an add, and a power. We can walk that graph backward, and print each step. Run it. There's the record, from the end back to the start. Backward simply travels along this graph, applying the chain rule at each step. That's backpropagation, and we're going to see it in detail soon.

import torch

x = torch.tensor(2.0, requires_grad=True)
a = x * 3
b = a + 1
c = b ** 2

print("The graph, from the end back:")
fn = c.grad_fn
while fn is not None:
    print(" ", type(fn).__name__)
    fn = fn.next_functions[0][0] if fn.next_functions else None

c.backward()
print("dc/dx =", x.grad.item())

The graph, from the end back:
  PowBackward0
  AddBackward0
  MulBackward0
  AccumulateGrad
dc/dx = 42.0

Summary