AI Hints
Home›Statistics & Math for ML Tutorial›Loss and Logistic Regression
Statistics & Math for ML · Lesson 103 of 138

Loss Functions

Measuring mistakes.

⏱ 2 min read▶ Video at 2:02:36🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Before a model can improve, it needs a single number that says how wrong it is. That's the loss function. Training means changing the weights to make the loss as small as possible. The choice of loss changes what the model cares about.

Loss Functions
From the video: Loss Functions
  • One number: how wrong?
  • Training shrinks it
  • The choice matters

MSE and MAE

Mean squared error squares each error, so big mistakes count much more. Mean absolute error just takes the size of each error. Here, one prediction is badly wrong. Run it. The one big error pushes the squared error up to twenty point four, but the absolute error is only two point four.

losses.pypython
import numpy as np

actual = np.array([10, 12, 11, 13, 12])
predicted = np.array([11, 12, 10, 13, 22])
err = predicted - actual

print("errors:", err)
print("MSE:", (err ** 2).mean())
print("MAE:", np.abs(err).mean())
Output
errors: [ 1  0 -1  0 10]
MSE: 20.4
MAE: 2.4

Common losses

For predicting numbers, mean squared error is the usual choice. Mean absolute error is better when you don't want outliers to dominate. Huber loss is a mix of the two. And for classification, we use cross entropy, which we'll meet in a few lessons.

LossTaskNote
MSEregressionpunishes big errors
MAEregressionrobust to outliers
Huberregressiona mix of both
Cross-entropyclassificationfor probabilities

Summary

A loss function turns mistakes into one number to minimize. Squared error punishes big mistakes, and absolute error is more forgiving. Next, the sigmoid.

Key takeaways
  • MSE squares errors
  • MAE is robust
  • Training minimizes loss
Next up: The Sigmoid — squash into 0 to 1