The Chain Rule
Functions inside functions.
Neural networks are functions inside functions inside functions. Each layer feeds into the next. To find how the final error depends on an early weight, we need the chain rule.

- Functions inside functions
- Layers feed into layers
- The chain rule
Multiply the links
If Y depends on U, and U depends on X, then the rate at which Y changes with X is the two rates multiplied. How fast Y changes with U, times how fast U changes with X. Like gears. If gear A turns gear B twice as fast, and B turns C three times as fast, then A turns C six times as fast.
| key | value |
|---|---|
| y = f(u), u = g(x) | a chain |
| dy/dx | dy/du × du/dx |
| Gears | 2× then 3× = 6× |
| Networks | a long chain |
Checking the chain rule
Take Y equals three X plus one, all squared. The outside function is squaring, and its derivative is two U. The inside is three X plus one, with derivative three. So the chain rule gives two times three X plus one, times three. Run it. Sim Pi agrees, and so does a tiny numerical nudge.
import sympy as sp
x = sp.symbols("x")
y = (3 * x + 1) ** 2
print("SymPy: ", sp.factor(sp.diff(y, x)))
print("chain rule at x=2:", 2 * (3 * 2 + 1) * 3)
f = sp.lambdify(x, y)
print("nudge at x=2: ", round((f(2 + 1e-6) - f(2)) / 1e-6, 2))SymPy: 6*(3*x + 1) chain rule at x=2: 42 nudge at x=2: 42.0
Summary
The chain rule multiplies the rates along a chain of functions. It's how neural networks trace errors back through their layers. Next, functions with more than one input.
Key takeaways- dy/dx = dy/du × du/dx
- Like gears
- Check by nudging