AI Hints
Statistics & Math for ML · Lesson 101 of 138

Backpropagation Idea

The chain rule in networks.

⏱ 2 min read▶ Video at 1:59:48🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A neural network can have millions of weights. To train it, we need the gradient of the error with respect to every one of them. Backpropagation computes them all efficiently, using the chain rule, from the output backwards.

Backpropagation Idea
From the video: Backpropagation Idea
  • Millions of weights
  • A gradient for each
  • Backpropagation

One tiny neuron

Here's a neuron with one weight. It computes W times X, passes it through a sigmoid, and compares the output with the target. Backpropagation multiplies the local derivatives, from the loss back to W. Then we check it with a tiny nudge. Run it. The chain rule and the nudge agree.

backprop.pypython
import numpy as np

sigmoid = lambda z: 1 / (1 + np.exp(-z))
x, target, w = 2.0, 1.0, 0.3

def loss(w):
    return (sigmoid(w * x) - target) ** 2

out = sigmoid(w * x)
grad = 2 * (out - target) * out * (1 - out) * x
print("backprop:", round(grad, 5))
print("nudge:   ", round((loss(w + 1e-6) - loss(w)) / 1e-6, 5))
Output
backprop: -0.32427
nudge:    -0.32427

Forward, then backward

Training a network repeats four steps. The forward pass computes the prediction, layer by layer. The loss measures the error. The backward pass sends gradients back through each layer, with the chain rule. And the optimizer updates every weight a little, using gradient descent.

  1. Forwardcompute the prediction
  2. Lossmeasure the error
  3. Backwardchain rule, layer by layer
  4. Updategradient descent step
Forward, then backward
From the video: Forward, then backward

Summary

Backpropagation is the chain rule, applied from the output back through every layer. Libraries like PyTorch do it automatically. Next, a quiz on calculus.

Key takeaways
  • Chain rule backwards
  • Check with a nudge
  • Forward, loss, backward, update
Next up: Calculus Quiz — test yourself