AI Hints
Home›Statistics & Math for ML Tutorial›Going Deeper: Geometry
Statistics & Math for ML · Lesson 127 of 138

Distances for ML

K nearest neighbours.

⏱ 2 min read▶ Video at 2:32:00🎓 Free
Watch this lesson in the Statistics & Math for ML courseWatch on YouTube ↗Full Statistics & Math for ML course

The simplest machine learning model of all uses nothing but distance. To classify a new point, find the K training points closest to it, and let them vote. It's called K nearest neighbours.

Distances for ML
From the video: Distances for ML
  • Nothing but distance
  • Find the k closest
  • Let them vote

K nearest neighbours by hand

We compute the distance from the new point to every training point, with one NP dot lin alg dot norm call. Arg sort gives the positions of the closest. Then Counter picks the most common label among the nearest three. Run it. Two of the three nearest are red, so the new point is red.

knn.pypython
import numpy as np
from collections import Counter

X = np.array([[1, 1], [2, 1], [1, 2], [6, 6], [7, 6], [6, 7], [3, 3]])
labels = ["red", "red", "red", "blue", "blue", "blue", "blue"]
new = np.array([2.5, 2.5])

dist = np.linalg.norm(X - new, axis=1)
nearest = np.argsort(dist)[:3]
print("nearest:", [labels[i] for i in nearest])
print("vote:", Counter(labels[i] for i in nearest).most_common(1)[0][0])
Output
nearest: ['blue', 'red', 'red']
vote: red

Scale matters

Because K nearest neighbours uses distance, features on big scales dominate. Income in dollars would swamp age in years. So always standardize first. And in very high dimensions, every point ends up roughly the same distance from every other, which makes distances less useful. That's called the curse of dimensionality.

keyvalue
Modelvote of the k closest
Distanceusually Euclidean
Standardizeor big features dominate
High dimensionsdistances lose meaning

Summary

K nearest neighbours classifies by the votes of the closest training points. Standardize first, because it's all about distance. Next, projections.

Key takeaways
  • np.linalg.norm(X − new, axis=1)
  • argsort for the nearest
  • Standardize first
Next up: Projections — shadows of vectors