Partial Derivatives
One variable at a time.
A model's error doesn't depend on one weight. It depends on all of them at once. To see how the error changes with one weight, we hold all the others still, and take the derivative. That's a partial derivative.

- Error depends on many weights
- Hold the others still
- A partial derivative
Nudging one input at a time
Here's a function of two inputs, X squared plus three XY. To get the partial derivative with respect to X, we nudge only X. For Y, we nudge only Y. Run it. At the point two, one, the slopes are seven in the X direction, and six in the Y direction.
def f(x, y):
return x ** 2 + 3 * x * y
h = 1e-6
x, y = 2.0, 1.0
dx = (f(x + h, y) - f(x, y)) / h
dy = (f(x, y + h) - f(x, y)) / h
print(f"df/dx = {dx:.2f}, df/dy = {dy:.2f}")df/dx = 7.00, df/dy = 6.00
By the rules
With the rules, treat the other variable as a constant. For X squared plus three XY, the partial with respect to X is two X plus three Y. At two, one, that's seven. The partial with respect to Y is three X, which is six. The same answers as our nudges.
| key | value |
|---|---|
| ∂f/∂x | 2x + 3y = 7 |
| ∂f/∂y | 3x = 6 |
| Rule | others are constants |
| Symbol | ∂, a curly d |
Summary
A partial derivative measures change in one input, holding the others fixed. Next, putting all the partials together into a gradient.
Key takeaways- Nudge one input
- Others are constants
- ∂f/∂x