Gradients
The uphill direction.
Put all the partial derivatives into one vector, and you get the gradient. It has a wonderful property. The gradient points in the direction where the function rises fastest. Uphill.

- All partials in one vector
- The gradient
- Points uphill
Arrows on a hill
Here's a bowl shaped function, X squared plus two Y squared. We draw its contour lines, like a map of a valley. Then we draw the gradient at several points as arrows. Run it. Every arrow points away from the bottom of the valley, straight uphill.
import numpy as np
import matplotlib.pyplot as plt
g = np.linspace(-3, 3, 100)
X, Y = np.meshgrid(g, g)
plt.contour(X, Y, X**2 + 2 * Y**2, levels=12, cmap="cool")
px, py = np.meshgrid(np.linspace(-2.5, 2.5, 6), np.linspace(-2.5, 2.5, 6))
plt.quiver(px, py, 2 * px, 4 * py, color="#fbbf24")
plt.gca().set_aspect("equal"); plt.title("Gradient arrows point uphill")
plt.savefig("gradient.png")Go the other way
If the gradient points uphill, then minus the gradient points downhill. And a model's error is a landscape, where each position is a set of weights. To reduce the error, step in the direction of minus the gradient. That simple idea, repeated millions of times, is how neural networks learn.
| key | value |
|---|---|
| ∇f | vector of partials |
| Direction | steepest uphill |
| −∇f | steepest downhill |
| Length | how steep |
Summary
The gradient is the vector of partial derivatives. It points uphill, so minus the gradient points downhill. Next, gradient descent.
Key takeaways- ∇f = [∂f/∂x, ∂f/∂y]
- Points uphill
- −∇f goes down