AI Hints
Statistics & Math for ML · Lesson 120 of 138

Practice: A Model from Scratch

Logistic regression by hand.

⏱ 2 min read▶ Video at 2:23:27🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

For the last project, let's build logistic regression ourselves, using only NumPy. Dot products, the sigmoid, the cross entropy gradient, and gradient descent. Every piece comes from this course.

Practice: A Model from Scratch
From the video: Practice: A Model from Scratch
  • Logistic regression by hand
  • Only NumPy
  • Every piece from this course

Training loop

We make a dataset with four features, and standardize it. The model is the sigmoid of X times W, plus B. The gradient of the cross entropy loss is X transpose, times the errors, divided by N. We take five hundred gradient descent steps. Run it. Our hand built model matches scikit learn's accuracy.

scratch.pypython
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression

X, y = make_classification(n_samples=1000, n_features=4, random_state=1)
X = (X - X.mean(0)) / X.std(0)

w, b = np.zeros(4), 0.0
for _ in range(500):
    p = 1 / (1 + np.exp(-(X @ w + b)))
    w -= 0.5 * X.T @ (p - y) / len(y)
    b -= 0.5 * (p - y).mean()

print("ours:   ", ((X @ w + b > 0) == y).mean())
print("sklearn:", LogisticRegression().fit(X, y).score(X, y))
Output
ours:    0.848
sklearn: 0.85

What you just used

That short loop used almost every big idea in the course. Standardizing from the spread section. Dot products and matrix multiplication from linear algebra. The sigmoid and cross entropy from machine learning maths. And gradient descent from calculus. Neural networks are this same loop, with more layers.

keyvalue
Standardizez-scores
X @ wmatrix multiplication
Sigmoid + cross-entropyprobabilities and loss
Updategradient descent

Summary

You built a classifier from scratch, with nothing but the maths from this course. Next, a bonus section that goes a little deeper, starting with log transforms.

Key takeaways
  • Standardize
  • Sigmoid of X @ w
  • Gradient descent
Next up: Log Transforms — taming skewed data