Recommenders with Embeddings

Most of what you watch, buy, and listen to was chosen by a recommender. The core idea is amazingly simple. Give each person and each item a short list of numbers. If the numbers line up, then that person will probably like that item.

• What you watch and buy
• Chosen by a recommender
• Numbers that line up

Matrix factorisation

We've got sixty users and forty items, and we can only see thirty percent of the ratings. The model gives each user four numbers, and each item four numbers. A predicted rating is just their dot product. We train only on ratings that we can see. Run it. Look at the last two lines. On ratings the model has never seen, its error is only point two three, against two for guessing. It has filled in the blanks. It worked out the tastes of users, and the qualities of items, with no descriptions at all. It's only ever seen ratings.

import torch

torch.manual_seed(0)
n_users, n_items = 60, 40
true_u, true_i = torch.randn(n_users, 4), torch.randn(n_items, 4)
rating = true_u @ true_i.T + 0.1 * torch.randn(n_users, n_items)
mask = torch.rand(n_users, n_items) < 0.3

class MF(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.u, self.i = torch.nn.Embedding(n_users, 4), torch.nn.Embedding(n_items, 4)
    def forward(self):
        return self.u.weight @ self.i.weight.T

m = MF()
opt = torch.optim.Adam(m.parameters(), lr=.02, weight_decay=1e-3)
for step in range(800):
    loss = ((m() - rating)[mask] ** 2).mean()
    opt.zero_grad(); loss.backward(); opt.step()

print(f"Seen ratings error: {((m() - rating)[mask] ** 2).mean().sqrt():.2f}")
print(f"Unseen ratings error: {((m() - rating)[~mask] ** 2).mean().sqrt():.2f}")
print(f"Guessing error: {rating.std():.2f}")

Seen ratings error: 0.10
Unseen ratings error: 0.23
Guessing error: 2.01

Learning by comparison

The same idea powers contrastive learning, which trains embeddings without any labels. We take pairs of things that should match, such as two crops of one photo. The goal is for each item to be nearest to its own partner. That turns into simple cross entropy, on a table of similarities. Run it. With true pairs, the loss is near zero. When we mix them up, it jumps to about seven and a half. Train to bring that number down, and you get the embeddings that are behind image search, and models like C L I P.

import torch
from torch.nn.functional import normalize, cross_entropy

torch.manual_seed(0)
a = normalize(torch.randn(8, 16), dim=1)
b = normalize(a + 0.1 * torch.randn(8, 16), dim=1)
sim = a @ b.T / 0.1
print("Loss with matching pairs:", round(cross_entropy(sim, torch.arange(8)).item(), 3))

shuffled = b[torch.randperm(8)]
print("Loss with mixed pairs:   ", round(cross_entropy(a @ shuffled.T / 0.1, torch.arange(8)).item(), 3))

Loss with matching pairs: 0.004
Loss with mixed pairs:    7.504

From demo to production

Real recommenders work in two steps. First, a fast search finds a few hundred candidates, out of millions, using embeddings. Then a heavier model ranks them. Watch out for feedback loops, where you only learn about what you've already shown, and popular items take over. And start simple. Popular items are a tough baseline to beat.

Summary