Slopes and Derivatives

When we did gradient descent, we used a formula for the slope, and I asked you to trust it. Let's now see where that comes from. Don't worry, there's no heavy calculus. You only need one single idea.

• Remember the slope formula?
• Where does it come from?
• Only one idea needed

What a derivative means

A derivative answers one question. If I nudge this number a tiny bit, how much does the output change? If the answer is large and positive, the loss rises quickly, so we should move the other way. If it's near zero, we're near the bottom. For many weights, we get one slope for each, and together they form the gradient. In neural networks, an algorithm called backpropagation computes all of these automatically.

Measuring a slope

Let's check it. Our loss is w minus three, squared, with its lowest point at three. The formula for the slope is two times w minus three. But we can also just measure it, by nudging w a tiny amount, and seeing how the loss changes. Run it. The two answers are nearly identical. So a derivative isn't mysterious. It's just rise over run, on a tiny scale. Then the loop steps downhill, and w moves toward three, with the loss shrinking at every step.

def loss(w):
    return (w - 3) ** 2

def slope(w):
    return 2 * (w - 3)

h = 0.0001
w = 7.0

print("Formula slope: ", slope(w))
print("Measured slope:", round((loss(w + h) - loss(w)) / h, 3))

for step in range(5):
    w = w - 0.3 * slope(w)
    print(f"step {step + 1}: w = {w:.3f}, loss = {loss(w):.3f}")

Formula slope:  8.0
Measured slope: 8.0
step 1: w = 4.600, loss = 2.560
step 2: w = 3.640, loss = 0.410
step 3: w = 3.256, loss = 0.066
step 4: w = 3.102, loss = 0.010
step 5: w = 3.041, loss = 0.002

Summary