How Many Groups?

Adding more groups always makes inertia smaller. With one group per point, inertia would be zero, and completely useless. So we cannot simply pick the lowest score. We need a smarter way to decide. There are two popular ones, and neither is perfect.

• More groups always lowers inertia
• Lowest score is meaningless
• Two smarter methods

The elbow and the silhouette

We try k from two to seven, and print two scores. First, the elbow method. Inertia falls quickly at first, then it flattens. The bend is a good choice. Look at the column. It drops sharply up to four, then barely changes. Second, the silhouette score. For each point, it compares how close that point is to its own group, against the next nearest group. Closer to one is better. It peaks at k equals four, which agrees with the hidden truth. In real data, these signals are usually much less clear.

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs
from sklearn.metrics import silhouette_score

X, _ = make_blobs(n_samples=600, centers=4, cluster_std=0.9, random_state=7)

for k in range(2, 8):
    km = KMeans(n_clusters=k, n_init=10, random_state=0).fit(X)
    sil = silhouette_score(X, km.labels_)
    print(f"k = {k}  inertia {km.inertia_:7.0f}  silhouette {sil:.3f}")

k = 2  inertia   18163  silhouette 0.594
k = 3  inertia    5018  silhouette 0.773
k = 4  inertia     913  silhouette 0.830
k = 5  inertia     826  silhouette 0.696
k = 6  inertia     746  silhouette 0.571
k = 7  inertia     663  silhouette 0.467

Real neighbourhood groups

Now for your boss's request. We take four facts about each district, clip the outliers, scale them, and find four groups. Scaling is essential, because k means is built on distance. Then comes the important part. We describe each group by its averages. Read the table across. Each row is a different type of neighbourhood. One is wealthy, with large homes. Another is older and crowded. Clustering gives you numbers, but people give them meaning. Your marketing team can now name these groups, and talk to each one differently.

import pandas as pd
from sklearn.cluster import KMeans
from sklearn.datasets import fetch_california_housing
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

df = fetch_california_housing(as_frame=True).frame
cols = ["MedInc", "HouseAge", "AveRooms", "AveOccup"]
X = df[cols].clip(upper=df[cols].quantile(0.99), axis=1)

model = make_pipeline(StandardScaler(),
                      KMeans(n_clusters=4, n_init=10, random_state=0))
df["group"] = model.fit_predict(X)

summary = df.groupby("group")[cols + ["MedHouseVal"]].mean().round(1)
summary["size"] = df["group"].value_counts()
print(summary.to_string())

       MedInc  HouseAge  AveRooms  AveOccup  MedHouseVal  size
group
0         3.5      40.8       5.0       2.6          2.2  6277
1         3.2      18.4       5.0       2.6          1.7  6088
2         6.3      22.2       7.5       3.0          3.0  4481
3         2.7      32.5       4.4       4.8          1.4  3794

Where k-means struggles

K means has limits. It assumes round groups of similar size. It cannot find a long curved shape. Every point must join a group, even real oddities. And outliers can drag a centre away. When your data breaks these assumptions, you need a different algorithm, and we have two good ones coming up.

Summary