Gradient Descent for Regression
Learning a line.
Earlier, we found the best line through the study data with a formula. Now let's find it the way a neural network would. Start with a random line, and improve it, step by step, with gradient descent.

- We used a formula before
- Now learn it step by step
- Like a neural network
Learning m and b
The loss is the mean squared error. Its gradients for M and B come from the chain rule. Each step, we predict, measure the errors, and nudge M and B downhill. Run it. After two thousand steps, gradient descent reaches the same line as polyfit.
import numpy as np
from study import hours, scores
m, b, lr = 0.0, 0.0, 0.02
for step in range(2000):
error = (m * hours + b) - scores
m -= lr * 2 * (error * hours).mean()
b -= lr * 2 * error.mean()
print(f"gradient descent: m = {m:.2f}, b = {b:.2f}")
print("polyfit: m = %.2f, b = %.2f" % tuple(np.polyfit(hours, scores, 1)))gradient descent: m = 5.73, b = 36.86 polyfit: m = 5.73, b = 36.86
Also uses study.py
import numpy as np
hours = np.array([1, 2, 3, 4, 5, 6, 7, 8])
scores = np.array([42, 50, 53, 61, 64, 72, 74, 85])Why bother?
For one line, the formula is easier. But gradient descent works for any model, as long as you can compute the gradient of the loss. Neural networks with billions of weights have no neat formula. They're trained with exactly this loop. Predict, measure the error, compute gradients, and step.
| key | value |
|---|---|
| Loss | mean squared error |
| Gradients | from the chain rule |
| Loop | predict, error, step |
| Scales | to billions of weights |
Summary
Gradient descent learned the same line as the formula, one step at a time. The same loop trains neural networks. Next, speeding it up with small batches.
Key takeaways- Predict
- Measure the error
- Step downhill