Hard Deep Learning Questions

Here are questions that good candidates often stumble on. Try each one yourself, and then listen. Say your answer out loud, as that's the true test.

• Good candidates stumble
• Try each one
• Say it out loud

Your loss goes to NaN. What do you do?

Stay calm, and work through it. Is the learning rate too high? Lower it. Is there a division by zero, or a log of zero, in your loss? Add a small number. Are there bad values in your data? Check for infinities. Clip your gradients. And use anomaly detection in PyTorch, which will tell you exactly which operation produced the first bad value.

Training is great, validation is poor. Why?

That's overfitting, so go through the options. More data, or stronger augmentation. Regularisation, such as weight decay and dropout. A smaller model. Early stopping. Then check the split itself. If validation is much worse from the very start, your two sets may come from different distributions, and that's a data problem, rather than overfitting.

Why not just use a huge batch size?

Large batches are fast per epoch. But each update's less noisy, and that noise can help training to find better solutions. Very large batches need a higher learning rate, and they may generalise a little worse. And memory limits you too. So try a moderate batch, and scale the learning rate along with it.

Your model works in a notebook, but fails in production. Why?

Start by comparing the inputs. Is the preprocessing identical? Check resizing, normalisation, and colour order. Is the model in evaluation mode? Are the weights the same version? Has the data changed since training? Most production failures come from a mismatch in the pipeline, rather than from the model itself.

Accuracy is stuck at random chance. Where do you look?

This usually means that nothing's flowing. First, try to overfit a single batch. If that fails, you've got a bug, rather than a tuning problem. Check that the labels line up with the inputs, because a bad shuffle will destroy learning. Check that the gradients aren't zero. And make sure that you're calling zero grad, backward, and step, in the right order.

Why is validation loss lower than training loss?

It feels impossible, but it's often harmless. Dropout is switched on during training, and off during validation, so the training network's working with a handicap. Augmentation makes training examples harder. And training loss is averaged over a whole epoch, while the model was still improving. If the gap's huge, though, check for a leak between your two sets.

Why do transformers use a warmup?

At the very start, the weights are random, and the gradients can be wild. Adam's estimates are also poor, as they've seen very little data. A big learning rate at that moment can throw the model somewhere bad. So we start small, and we rise slowly for a while. Then we decay. It costs almost nothing, and it makes training far more stable.

Batch norm or layer norm?

Batch norm uses the statistics of the batch. So it works beautifully for images with large batches. But it behaves differently in training and in evaluation, and it struggles with tiny batches. Layer norm looks at one example at a time, so batch size doesn't matter, and that suits sequences. That's why convolution networks often use batch norm, and transformers use layer norm.

Your GPU sits at thirty percent. What is wrong?

The GPU's waiting for work, so the bottleneck is usually your data. Time each step before you change anything. Are you decoding images on one process? Add workers. Is your disk slow? Cache the data. Are you doing heavy augmentation? Make it cheaper. Also check that batches are moved efficiently. Never buy a bigger GPU until you've looked at this.

How do you choose the size of a model?

Start with something small which trains quickly, and beat a baseline. If it can't overfit, it's too small, so make it bigger. If it overfits badly, add regularisation, or data, before you shrink it. Stop growing when the gain no longer pays for the cost in speed and money. Remember that the budget in production decides the winner, rather than the leaderboard.

Summary