Distances for ML
K nearest neighbours.
The simplest machine learning model of all uses nothing but distance. To classify a new point, find the K training points closest to it, and let them vote. It's called K nearest neighbours.

- Nothing but distance
- Find the k closest
- Let them vote
K nearest neighbours by hand
We compute the distance from the new point to every training point, with one NP dot lin alg dot norm call. Arg sort gives the positions of the closest. Then Counter picks the most common label among the nearest three. Run it. Two of the three nearest are red, so the new point is red.
import numpy as np
from collections import Counter
X = np.array([[1, 1], [2, 1], [1, 2], [6, 6], [7, 6], [6, 7], [3, 3]])
labels = ["red", "red", "red", "blue", "blue", "blue", "blue"]
new = np.array([2.5, 2.5])
dist = np.linalg.norm(X - new, axis=1)
nearest = np.argsort(dist)[:3]
print("nearest:", [labels[i] for i in nearest])
print("vote:", Counter(labels[i] for i in nearest).most_common(1)[0][0])nearest: ['blue', 'red', 'red'] vote: red
Scale matters
Because K nearest neighbours uses distance, features on big scales dominate. Income in dollars would swamp age in years. So always standardize first. And in very high dimensions, every point ends up roughly the same distance from every other, which makes distances less useful. That's called the curse of dimensionality.
| key | value |
|---|---|
| Model | vote of the k closest |
| Distance | usually Euclidean |
| Standardize | or big features dominate |
| High dimensions | distances lose meaning |
Summary
K nearest neighbours classifies by the votes of the closest training points. Standardize first, because it's all about distance. Next, projections.
Key takeaways- np.linalg.norm(X − new, axis=1)
- argsort for the nearest
- Standardize first