AI Hints
Home›Statistics & Math for ML Tutorial›Chain Rule and Gradients
Statistics & Math for ML · Lesson 94 of 138

The Chain Rule

Functions inside functions.

⏱ 2 min read▶ Video at 1:51:36🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Neural networks are functions inside functions inside functions. Each layer feeds into the next. To find how the final error depends on an early weight, we need the chain rule.

The Chain Rule
From the video: The Chain Rule
  • Functions inside functions
  • Layers feed into layers
  • The chain rule

If Y depends on U, and U depends on X, then the rate at which Y changes with X is the two rates multiplied. How fast Y changes with U, times how fast U changes with X. Like gears. If gear A turns gear B twice as fast, and B turns C three times as fast, then A turns C six times as fast.

keyvalue
y = f(u), u = g(x)a chain
dy/dxdy/du × du/dx
Gears2× then 3× = 6×
Networksa long chain

Checking the chain rule

Take Y equals three X plus one, all squared. The outside function is squaring, and its derivative is two U. The inside is three X plus one, with derivative three. So the chain rule gives two times three X plus one, times three. Run it. Sim Pi agrees, and so does a tiny numerical nudge.

chain.pypython
import sympy as sp

x = sp.symbols("x")
y = (3 * x + 1) ** 2
print("SymPy:      ", sp.factor(sp.diff(y, x)))
print("chain rule at x=2:", 2 * (3 * 2 + 1) * 3)

f = sp.lambdify(x, y)
print("nudge at x=2:     ", round((f(2 + 1e-6) - f(2)) / 1e-6, 2))
Output
SymPy:       6*(3*x + 1)
chain rule at x=2: 42
nudge at x=2:      42.0

Summary

The chain rule multiplies the rates along a chain of functions. It's how neural networks trace errors back through their layers. Next, functions with more than one input.

Key takeaways
  • dy/dx = dy/du × du/dx
  • Like gears
  • Check by nudging
Next up: Partial Derivatives — one variable at a time