Training at Scale
Many machines, safely.
Some models train in seconds, on a laptop. Others need days, on dozens of graphics cards. At that scale, new problems appear. How do you split the work? What if a machine fails on day three? How do you avoid a giant bill? In this lesson, we cover the main ideas, without needing a giant cluster.
- Split the work
- Survive failures
- Control the bill
Ways to split training
There are three main ways to split training. Data parallelism, every machine holds a full copy of the model, and trains on a different slice of the data, then they share what they learned. It is the most common. Model parallelism, the model itself is too big for one machine, so its layers are spread across several. And pipeline parallelism, which cuts the model into stages, and streams batches through them, like an assembly line. Libraries like Py Torch distributed, Ray, and Deep Speed handle the details.
Checkpoints save the day
The most important habit for long training jobs is checkpointing. Every few minutes, the job saves its full state, the weights, the optimiser, and the position in the data, to durable storage. If a machine fails, or a spot instance is taken away, the job restarts from the last checkpoint, instead of from zero. Together with an orchestrator, that restarts failed jobs automatically, checkpoints turn a fragile, multi day run, into a resilient one.
- Trainfor a few minutes
- Save checkpointweights, optimiser, data position
- Failure happensmachine lost
- Resumefrom the last checkpoint
Controlling the cost
Training bills can be large, so manage them. Use spot, or preemptible, instances, which cost much less, and rely on checkpoints to survive interruptions. Right size the machines, because idle graphics cards still cost money. Use mixed precision, which trains faster with less memory. Stop failing experiments early. Start with a small sample of the data, to debug quickly, before the full run. And track cost per experiment in your experiment tracker, next to the metrics, so the price of every result is visible.
| Lever | Effect |
|---|---|
| Spot instances plus checkpoints | large discounts, safe from interruptions |
| Right-sized machines | no paying for idle GPUs |
| Mixed precision | faster, less memory |
| Early stopping, small samples | fewer wasted runs |
| Cost in the tracker | visible price per result |
Summary
Wonderful. Split training with data, model, or pipeline parallelism. Checkpoint often, so failures cost minutes, not days. Use spot instances, mixed precision, and early stopping, and track cost with every experiment. Next, the platforms that bundle many of these ideas, for teams.
Key takeaways- Data, model, pipeline parallelism
- Checkpoint regularly
- Spot + checkpoints + tracked cost