Choosing an Architecture

There are thousands of published architectures, with a new one each week. Newcomers often spend weeks reading about them, and get nowhere. Here's the truth that experienced engineers know. For most projects, the choice barely matters, and a good default will take you ninety percent of the way.

• Thousands of architectures
• A new one each week
• A good default gets you far

Sensible defaults

Here are sensible defaults. For classification, use a small ConvNeXt or EfficientNet, with transfer learning. For detection, use a recent YOLO. For segmentation, use U-Net for medical images, and a YOLO segmentation model for general use. For features, use DINO, which is very strong. And to go from text to image understanding, use CLIP or a multimodal model.

CNNs and transformers

Are transformers better than convolutional networks? It depends. Convolutions build in useful assumptions, so they learn well from smaller datasets, and they run efficiently on devices. Transformers learn more from huge datasets, and see the whole image at once. With a pretrained start, both work well. In a small project, the data will matter much more.

How to choose in practice

First, begin with the smallest model that could possibly work. Measure it against your speed limit. Fix your data problems. Only then try a bigger model. Most of the improvement in real projects comes from better data, not from better architecture. And keep your code modular, so that you can swap models in an afternoon.

Summary