AI Hints
Home›Statistics & Math for ML Tutorial›Loss and Logistic Regression
Statistics & Math for ML · Lesson 105 of 138

Logistic Regression

Probabilities from a line.

⏱ 2 min read▶ Video at 2:04:47🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

Will a student pass, given the hours they studied? Logistic regression takes a line, W times hours plus B, and passes it through the sigmoid. The result is the probability of passing.

Logistic Regression
From the video: Logistic Regression
  • Pass or fail?
  • A line, then a sigmoid
  • A probability

Fitting and predicting

Here are sixteen students, with their hours, and whether they passed. Scikit learn's logistic regression finds the best W and B. Predict proba gives probabilities for fail and pass. Run it. One hour of study gives a low chance of passing. Eight hours gives a very high chance.

logistic.pypython
from sklearn.linear_model import LogisticRegression
from exam import hours, passed

model = LogisticRegression().fit(hours.reshape(-1, 1), passed)

for h in [1, 4, 8]:
    p = model.predict_proba([[h]])[0, 1]
    print(f"{h} hours: P(pass) = {p:.2f}")
Output
1 hours: P(pass) = 0.05
4 hours: P(pass) = 0.44
8 hours: P(pass) = 0.97
Also uses exam.py
import numpy as np

hours = np.array([0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 5.5, 6, 6.5, 7, 8, 9])
passed = np.array([0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1])

How it learns

Logistic regression is trained with gradient descent, using the cross entropy loss, not squared error. The output is a probability, and we choose a threshold, usually a half, to turn it into a yes or no. It's simple, fast, and easy to explain, so it's often the first classifier people try.

keyvalue
Modelσ(w · x + b)
OutputP(class = 1)
Thresholdusually 0.5
Losscross-entropy

Summary

Logistic regression is a line passed through a sigmoid. It outputs probabilities, and it's a great first classifier. Next, softmax, for more than two classes.

Key takeaways
  • σ(w·x + b)
  • predict_proba
  • Threshold at 0.5
Next up: Softmax — many classes