Gradient Descent
Walking downhill.
Imagine you're on a foggy mountain, and you want to reach the valley. You can't see far, but you can feel which way the ground slopes under your feet. So you take a small step downhill, and repeat. That's gradient descent.

- A foggy mountain
- Feel the slope
- Step downhill, repeat
Finding the bottom
Let's find the lowest point of X minus three, squared. Its derivative is two times X minus three. We start at zero, and each step moves X by minus the learning rate, times the derivative. Run it. X walks towards three, the bottom of the valley, in smaller and smaller steps.
def f(x):
return (x - 3) ** 2
def slope(x):
return 2 * (x - 3)
x, lr = 0.0, 0.2
for step in range(8):
x = x - lr * slope(x)
print(f"step {step + 1}: x = {x:.3f}, f(x) = {f(x):.4f}")step 1: x = 1.200, f(x) = 3.2400 step 2: x = 1.920, f(x) = 1.1664 step 3: x = 2.352, f(x) = 0.4199 step 4: x = 2.611, f(x) = 0.1512 step 5: x = 2.767, f(x) = 0.0544 step 6: x = 2.860, f(x) = 0.0196 step 7: x = 2.916, f(x) = 0.0071 step 8: x = 2.950, f(x) = 0.0025
The update rule
Every gradient descent step uses the same rule. New value equals old value, minus the learning rate, times the gradient. When the slope is steep, the step is big. As we get close to the bottom, the slope flattens, and the steps get smaller. At the bottom, the slope is zero, and we stop moving.
| key | value |
|---|---|
| Rule | x ← x − lr × gradient |
| Steep | big steps |
| Flat | small steps |
| Bottom | gradient = 0 |
Summary
Gradient descent repeatedly steps in the direction of minus the gradient. It's how almost every machine learning model learns. Next, choosing the step size.
Key takeaways- x ← x − lr × slope
- Steps shrink near the bottom
- Stop when flat