Designing Visual Search
A second worked answer.
Here's another classic. Design a system where someone can photograph a product, and find it in a shop of ten million items. It's the sort of feature that you've seen in shopping apps. Let's work it through.
- Photograph a product
- Find it among ten million
- A classic question
The pipeline
First, detect the main object, and crop it, as phone photos are messy. Then embed the crop with a network. Search a vector index for the nearest neighbours. Rerank the top hundred, with a stronger model. Then show the results. Each step is simple, and you've built nearly all of them.
- Detect and cropfind the product
- Embeda vector
- Vector searchten million items
- Reranktop hundred
- Showresults
The hard parts
The hard part is the gap between domains. Catalogue photos are clean studio shots, and customer photos are dark and blurry. So you train with pairs of photos, one from each, using contrastive learning. Then there's scale. Ten million vectors need an approximate index, which trades a little accuracy for a lot of speed. New products need embedding the same day.
Measuring success
Offline, measure how often the right product is in the top ten. Online, measure clicks and purchases. Test on real customer photos, rather than on catalogue images. Log searches with no clicks, as those are your failures. And plan for a language model option too, for describing what's in a photo in words.
Summary
Detect, embed, search, rerank. You've got every piece. Next, how to read research, so that you never fall behind.
Key takeaways- Cut the problem into stages
- Bridge the domain gap
- Measure on real photos