Learning Without Labels

A child doesn't need a million labelled photos to learn what a dog is. They see the world, and they notice patterns, with almost no labels at all. Self supervised learning tries to do the same. It's how language models learn too. They create their own labels from the data.

• Children need few labels
• They notice patterns
• Models can create their own labels

Making up your own task

The trick is to invent a task from the data itself. Hide the next word, and predict it. That's how G P T learns. Hide patches of an image, and rebuild them. That's a masked autoencoder. Or show two cropped views of one photo, and teach the network that they're the same, and that they differ from all other photos. That's contrastive learning.

Contrastive learning

Here's the core idea in a few lines. We compare every item in one group with every item in the other. The right answer for item one is the matching item, and so on down the diagonal. So it's just cross entropy. We test it with two views of the same things, and then with unrelated things. Run it. Matching views give a low loss. Unrelated ones give a high loss. Training pulls matching views together, and pushes the others apart. No labels are required.

import torch
from torch import nn

torch.manual_seed(0)

def contrastive(a, b, temperature=0.1):
    a = nn.functional.normalize(a, dim=1)
    b = nn.functional.normalize(b, dim=1)
    sim = a @ b.T / temperature
    labels = torch.arange(len(a))
    return nn.functional.cross_entropy(sim, labels)

base = torch.rand(8, 16)
view1 = base + 0.02 * torch.randn(8, 16)
view2 = base + 0.02 * torch.randn(8, 16)
stranger = torch.rand(8, 16)

print("Two views of the same things:", round(contrastive(view1, view2).item(), 3))
print("Unrelated things:           ", round(contrastive(view1, stranger).item(), 3))

Two views of the same things: 0.571
Unrelated things:            2.232

Summary