AI Hints
Home›Statistics & Math for ML Tutorial›Chain Rule and Gradients
Statistics & Math for ML · Lesson 95 of 138

Partial Derivatives

One variable at a time.

⏱ 2 min read▶ Video at 1:52:52🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A model's error doesn't depend on one weight. It depends on all of them at once. To see how the error changes with one weight, we hold all the others still, and take the derivative. That's a partial derivative.

Partial Derivatives
From the video: Partial Derivatives
  • Error depends on many weights
  • Hold the others still
  • A partial derivative

Nudging one input at a time

Here's a function of two inputs, X squared plus three XY. To get the partial derivative with respect to X, we nudge only X. For Y, we nudge only Y. Run it. At the point two, one, the slopes are seven in the X direction, and six in the Y direction.

partials.pypython
def f(x, y):
    return x ** 2 + 3 * x * y

h = 1e-6
x, y = 2.0, 1.0
dx = (f(x + h, y) - f(x, y)) / h
dy = (f(x, y + h) - f(x, y)) / h
print(f"df/dx = {dx:.2f}, df/dy = {dy:.2f}")
Output
df/dx = 7.00, df/dy = 6.00

By the rules

With the rules, treat the other variable as a constant. For X squared plus three XY, the partial with respect to X is two X plus three Y. At two, one, that's seven. The partial with respect to Y is three X, which is six. The same answers as our nudges.

keyvalue
∂f/∂x2x + 3y = 7
∂f/∂y3x = 6
Ruleothers are constants
Symbol∂, a curly d

Summary

A partial derivative measures change in one input, holding the others fixed. Next, putting all the partials together into a gradient.

Key takeaways
  • Nudge one input
  • Others are constants
  • ∂f/∂x
Next up: Gradients — the uphill direction