CLIP: Images Meet Words
One space for pictures and text.
Until twenty twenty one, if you wanted a model to recognise a new kind of thing, you had to collect examples and train it. Then OpenAI released CLIP. Suddenly, you could classify images into categories that you just typed out, with no training at all.
- New class meant new training
- CLIP changed that
- Type your categories
How CLIP learns
CLIP has two encoders, one for images, and one for text. It learned from hundreds of millions of images with captions. The training goal is simple. Pull matching images and captions close together, and push others apart. So images and words end up in one shared space. A photo of a dog lands close to the words, a photo of a dog.
Classifying with no training
We type four labels, in plain English. And we pass in two photos. Nobody trained this model on these specific labels. Run it. It picks the right label for each one. That's called zero shot classification. AppleEye could sort by typing, a bruised apple, or a fresh apple, and getting useful results on day one.
import torch
from PIL import Image
from skimage import data
from transformers import CLIPModel, CLIPProcessor
name = "openai/clip-vit-base-patch32"
model = CLIPModel.from_pretrained(name)
proc = CLIPProcessor.from_pretrained(name)
labels = ["a photo of a cat", "a photo of a dog", "a cup of coffee", "a rocket launch"]
images = [Image.fromarray(data.chelsea()), Image.fromarray(data.coffee())]
inputs = proc(text=labels, images=images, return_tensors="pt", padding=True)
with torch.no_grad():
chance = model(**inputs).logits_per_image.softmax(1)
for name, row in zip(["cat photo", "coffee photo"], chance):
print(f"{name:<13} {labels[row.argmax()]:<20} {row.max():.2f}")cat photo a photo of a cat 0.99 coffee photo a cup of coffee 1.00
Searching photos with words
Since text and images share one space, you can search photos using a sentence. We embed four photos once. Then we try three queries, none of which use the exact words for what's in the pictures. Run it. Something to drink finds the coffee. Space travel finds the rocket. A sleepy pet finds the cat. That's how photo search works on your phone.
import torch
from PIL import Image
from skimage import data
from transformers import CLIPModel, CLIPProcessor
name = "openai/clip-vit-base-patch32"
model = CLIPModel.from_pretrained(name)
proc = CLIPProcessor.from_pretrained(name)
photos = {"chelsea": data.chelsea(), "coffee": data.coffee(),
"rocket": data.rocket(), "astronaut": data.astronaut()}
inputs = proc(images=[Image.fromarray(v) for v in photos.values()], return_tensors="pt")
with torch.no_grad():
i = model.get_image_features(**inputs)
i = (i.pooler_output if hasattr(i, "pooler_output") else i)
i = i / i.norm(dim=-1, keepdim=True)
for q in ["something to drink", "space travel", "a sleepy pet"]:
t = model.get_text_features(**proc(text=[q], return_tensors="pt", padding=True))
t = (t.pooler_output if hasattr(t, "pooler_output") else t)
t = t / t.norm(dim=-1, keepdim=True)
print(f"{q:<20} -> {list(photos)[int((i @ t.T).argmax())]}")something to drink -> coffee space travel -> rocket a sleepy pet -> chelsea
Summary
CLIP puts pictures and words on one map, so that you can classify and search with plain language. Today's multimodal language models go further. You can just ask them questions about an image.
Key takeaways- Two encoders, one space
- Zero-shot classification
- Search images with words