The Learning Rate
Step size.
The learning rate controls how big each step is. It's the most important setting in training a model. Too small, and learning takes forever. Too big, and you overshoot the valley, and bounce right out of it.

- How big is each step?
- Too small: forever
- Too big: overshoot
Three learning rates
We run gradient descent on the same valley, with three learning rates, and plot X after each step. Run it. Zero point zero two crawls slowly. Zero point three gets to three quickly. And one point zero five overshoots further each time, and flies away.
import matplotlib.pyplot as plt
def path(lr, steps=25):
x, xs = 0.0, [0.0]
for _ in range(steps):
x = x - lr * 2 * (x - 3)
xs.append(x)
return xs
for lr in [0.02, 0.3, 1.05]:
plt.plot(path(lr), "o-", ms=4, label=f"lr = {lr}")
plt.axhline(3, color="#ffffff", ls="--"); plt.ylim(-4, 10); plt.legend()
plt.title("x after each step"); plt.savefig("lr.png")Choosing it in practice
Common starting points are zero point zero one, or zero point zero zero one. If the loss explodes, lower it. If the loss falls very slowly, raise it. Many modern optimizers, like Adam, adapt the step size for each weight automatically, and many training runs slowly shrink the learning rate over time.
| key | value |
|---|---|
| Too small | slow progress |
| Too big | bounces or explodes |
| Start | 0.01 or 0.001 |
| Adam | adapts per weight |
Summary
The learning rate sets the step size. Too small is slow, and too big diverges. Next, using gradient descent to learn a line.
Key takeaways- Small: slow
- Big: diverges
- Watch the loss