AI Hints
Statistics & Math for ML · Lesson 97 of 138

Gradient Descent

Walking downhill.

⏱ 2 min read▶ Video at 1:55:01🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Imagine you're on a foggy mountain, and you want to reach the valley. You can't see far, but you can feel which way the ground slopes under your feet. So you take a small step downhill, and repeat. That's gradient descent.

Gradient Descent
From the video: Gradient Descent
  • A foggy mountain
  • Feel the slope
  • Step downhill, repeat

Finding the bottom

Let's find the lowest point of X minus three, squared. Its derivative is two times X minus three. We start at zero, and each step moves X by minus the learning rate, times the derivative. Run it. X walks towards three, the bottom of the valley, in smaller and smaller steps.

descent.pypython
def f(x):
    return (x - 3) ** 2

def slope(x):
    return 2 * (x - 3)

x, lr = 0.0, 0.2
for step in range(8):
    x = x - lr * slope(x)
    print(f"step {step + 1}: x = {x:.3f}, f(x) = {f(x):.4f}")
Output
step 1: x = 1.200, f(x) = 3.2400
step 2: x = 1.920, f(x) = 1.1664
step 3: x = 2.352, f(x) = 0.4199
step 4: x = 2.611, f(x) = 0.1512
step 5: x = 2.767, f(x) = 0.0544
step 6: x = 2.860, f(x) = 0.0196
step 7: x = 2.916, f(x) = 0.0071
step 8: x = 2.950, f(x) = 0.0025

The update rule

Every gradient descent step uses the same rule. New value equals old value, minus the learning rate, times the gradient. When the slope is steep, the step is big. As we get close to the bottom, the slope flattens, and the steps get smaller. At the bottom, the slope is zero, and we stop moving.

keyvalue
Rulex ← x − lr × gradient
Steepbig steps
Flatsmall steps
Bottomgradient = 0

Summary

Gradient descent repeatedly steps in the direction of minus the gradient. It's how almost every machine learning model learns. Next, choosing the step size.

Key takeaways
  • x ← x − lr × slope
  • Steps shrink near the bottom
  • Stop when flat
Next up: The Learning Rate — step size