Stochastic Gradient Descent
Small batches.
With a million training examples, computing the exact gradient on all of them, for every single step, is very slow. The trick is to estimate the gradient from a small random batch of examples instead. That's stochastic gradient descent.

- A million examples
- Every step is slow
- Use a small random batch
Mini-batches
We create a hundred thousand points on the line Y equals three X plus two, with noise. Each step uses a random batch of just thirty two points. One pass through the data, called an epoch, is over three thousand quick steps. Run it. After one epoch, the weights are very close to three and two.
import numpy as np
rng = np.random.default_rng(0)
x = rng.uniform(0, 1, 100_000)
y = 3 * x + 2 + rng.normal(0, 0.1, 100_000)
w, b, lr = 0.0, 0.0, 0.1
for idx in rng.permutation(len(x)).reshape(-1, 32):
err = (w * x[idx] + b) - y[idx]
w -= lr * 2 * (err * x[idx]).mean()
b -= lr * 2 * err.mean()
print(f"after one epoch: w = {w:.2f}, b = {b:.2f}")after one epoch: w = 3.00, b = 2.00
Noisy, but fast
Each batch gives a noisy estimate of the gradient, so the path wobbles. But the steps are so cheap that it gets there much faster overall. The noise can even help, shaking the model out of bad spots. Batch sizes of thirty two to a few hundred are common. Every big neural network is trained this way.
| key | value |
|---|---|
| Batch | a small random subset |
| Epoch | one pass through the data |
| Noise | wobbly but fast |
| Typical batch | 32 to 512 |
Summary
Stochastic gradient descent estimates the gradient from small random batches. It's noisy, fast, and how big models are trained. Next, backpropagation.
Key takeaways- Random mini-batches
- Epoch = one pass
- Fast and noisy