AI Hints
Home›Statistics & Math for ML Tutorial›Going Deeper: Optimization
Statistics & Math for ML · Lesson 130 of 138

Second Derivatives

Curvature and minimums.

⏱ 2 min read▶ Video at 2:35:25🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

The first derivative tells you the slope. The second derivative tells you how the slope is changing. That's curvature. Is the curve bending upwards like a bowl, or downwards like a hill?

Second Derivatives
From the video: Second Derivatives
  • First derivative: slope
  • Second derivative: change in slope
  • Bowl or hill?

Bowl or hill?

At a flat point, the first derivative is zero. The second derivative tells us what kind of flat point it is. We use Sim Pi on X cubed minus three X, which has two flat points. Run it. At minus one, the second derivative is negative, so it's a hilltop. At one, it's positive, so it's the bottom of a valley.

second.pypython
import sympy as sp

x = sp.symbols("x")
f = x**3 - 3 * x

flat = sp.solve(sp.diff(f, x), x)
for p in flat:
    curve = sp.diff(f, x, 2).subs(x, p)
    print(f"x = {p}: second derivative = {curve}")
Output
x = -1: second derivative = -6
x = 1: second derivative = 6

Curvature in machine learning

A loss that curves sharply needs small steps, or gradient descent overshoots. A gently curving loss can take bigger steps. Some optimizers use second derivative information, called the Hessian, to choose better steps. And a positive second derivative everywhere means the loss is convex, a single bowl with one bottom.

keyvalue
f''(x) > 0bowl, a minimum
f''(x) < 0hill, a maximum
Hessiansecond derivatives in many dimensions
Convexone bowl, one bottom

Summary

The second derivative measures curvature. Positive is a bowl, negative is a hill. Next, the difference between the lowest point, and just a low point.

Key takeaways
  • f'' > 0: minimum
  • f'' < 0: maximum
  • Convex: one bowl
Next up: Local Minimums — valleys and saddles