Local Minimums
Valleys and saddles.
A simple bowl has one bottom. But many loss landscapes have lots of valleys. Gradient descent rolls into whichever valley is nearest, which might not be the deepest one. That's a local minimum.

- Many valleys
- Gradient descent rolls into the nearest
- Maybe not the deepest
Starting point matters
This curve has two valleys. We run gradient descent from three different starting points. Run it. Starting on the left finds the deep valley, near minus one point three. Starting on the right gets stuck in the shallower valley, near one point one.
def f(x):
return x**4 - 3 * x**2 + x
def slope(x):
return 4 * x**3 - 6 * x + 1
for start in [-2.0, 0.5, 2.0]:
x = start
for _ in range(200):
x -= 0.01 * slope(x)
print(f"start {start:5.1f} -> x = {x:.3f}, f = {f(x):.3f}")start -2.0 -> x = -1.301, f = -3.514 start 0.5 -> x = 1.131, f = -1.070 start 2.0 -> x = 1.131, f = -1.070
In deep learning
Neural network losses have countless valleys, and saddle points, which slope down in some directions and up in others. Surprisingly, most of the valleys are almost equally good in practice. The noise in stochastic gradient descent, momentum, and random restarts all help models escape bad spots.
| key | value |
|---|---|
| Global minimum | the lowest point |
| Local minimum | a low point |
| Saddle | down one way, up another |
| Escape | noise, momentum, restarts |
Summary
Gradient descent finds the nearest valley, which may be a local minimum. The starting point matters. Next, momentum, which helps roll through.
Key takeaways- Many valleys
- Start point matters
- Noise helps escape