AI Hints
Home›Statistics & Math for ML Tutorial›Going Deeper: Optimization
Statistics & Math for ML · Lesson 131 of 138

Local Minimums

Valleys and saddles.

⏱ 2 min read▶ Video at 2:36:43🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

A simple bowl has one bottom. But many loss landscapes have lots of valleys. Gradient descent rolls into whichever valley is nearest, which might not be the deepest one. That's a local minimum.

Local Minimums
From the video: Local Minimums
  • Many valleys
  • Gradient descent rolls into the nearest
  • Maybe not the deepest

Starting point matters

This curve has two valleys. We run gradient descent from three different starting points. Run it. Starting on the left finds the deep valley, near minus one point three. Starting on the right gets stuck in the shallower valley, near one point one.

local_min.pypython
def f(x):
    return x**4 - 3 * x**2 + x

def slope(x):
    return 4 * x**3 - 6 * x + 1

for start in [-2.0, 0.5, 2.0]:
    x = start
    for _ in range(200):
        x -= 0.01 * slope(x)
    print(f"start {start:5.1f} -> x = {x:.3f}, f = {f(x):.3f}")
Output
start  -2.0 -> x = -1.301, f = -3.514
start   0.5 -> x = 1.131, f = -1.070
start   2.0 -> x = 1.131, f = -1.070

In deep learning

Neural network losses have countless valleys, and saddle points, which slope down in some directions and up in others. Surprisingly, most of the valleys are almost equally good in practice. The noise in stochastic gradient descent, momentum, and random restarts all help models escape bad spots.

keyvalue
Global minimumthe lowest point
Local minimuma low point
Saddledown one way, up another
Escapenoise, momentum, restarts

Summary

Gradient descent finds the nearest valley, which may be a local minimum. The starting point matters. Next, momentum, which helps roll through.

Key takeaways
  • Many valleys
  • Start point matters
  • Noise helps escape
Next up: Momentum — a rolling ball