AI Hints
Home›Statistics & Math for ML Tutorial›Chain Rule and Gradients
Statistics & Math for ML · Lesson 96 of 138

Gradients

The uphill direction.

⏱ 2 min read▶ Video at 1:53:57🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Put all the partial derivatives into one vector, and you get the gradient. It has a wonderful property. The gradient points in the direction where the function rises fastest. Uphill.

Gradients
From the video: Gradients
  • All partials in one vector
  • The gradient
  • Points uphill

Arrows on a hill

Here's a bowl shaped function, X squared plus two Y squared. We draw its contour lines, like a map of a valley. Then we draw the gradient at several points as arrows. Run it. Every arrow points away from the bottom of the valley, straight uphill.

gradient.pypython
import numpy as np
import matplotlib.pyplot as plt

g = np.linspace(-3, 3, 100)
X, Y = np.meshgrid(g, g)
plt.contour(X, Y, X**2 + 2 * Y**2, levels=12, cmap="cool")

px, py = np.meshgrid(np.linspace(-2.5, 2.5, 6), np.linspace(-2.5, 2.5, 6))
plt.quiver(px, py, 2 * px, 4 * py, color="#fbbf24")
plt.gca().set_aspect("equal"); plt.title("Gradient arrows point uphill")
plt.savefig("gradient.png")

Go the other way

If the gradient points uphill, then minus the gradient points downhill. And a model's error is a landscape, where each position is a set of weights. To reduce the error, step in the direction of minus the gradient. That simple idea, repeated millions of times, is how neural networks learn.

keyvalue
∇fvector of partials
Directionsteepest uphill
−∇fsteepest downhill
Lengthhow steep

Summary

The gradient is the vector of partial derivatives. It points uphill, so minus the gradient points downhill. Next, gradient descent.

Key takeaways
  • ∇f = [∂f/∂x, ∂f/∂y]
  • Points uphill
  • −∇f goes down
Next up: Gradient Descent — walking downhill