Interview Questions

Interviewers want to hear that you understand why things work, not just that you've called some library. Here are the questions that come up again and again, each with a strong answer. Use these as a pattern, then practise them out loud in your own words.

• They want the why
• Common questions
• Practise out loud

Explain backpropagation

A strong answer goes like this. Backpropagation computes the gradient of the loss with respect to every weight, by applying the chain rule from the output back to the input. It's efficient because each step reuses results, so a full pass costs about the same as a forward pass. The optimiser then uses those gradients to update the weights. Mention the vanishing gradient problem, and how ReLU and skip connections help with it.

Why are transformers better than RNNs?

Compare them directly. R N Ns process one step at a time, so they can't run in parallel, and they struggle with long range links. Transformers look at all positions at once with attention, so they run in parallel on G P Us, and any word can directly reach any other. The cost is more memory, and quadratic time in sequence length.

Training loss is low, validation is high

Say this in order. It's overfitting. First, I'd check for data leakage, and verify my validation set. Then I'd get more data, or add augmentation. Next, I'd add regularisation, such as dropout and weight decay, or use early stopping, or try a smaller model. If I've got little data, I'd move to transfer learning. Showing an order of attack is what impresses people.

How do you pick a batch size?

A balanced answer. Larger batches make better use of the G P U, and give smoother gradients. Smaller ones add a bit of helpful noise, and use less memory. I'd start with what fits in memory, often thirty two to two hundred and fifty six. If I change the batch size a lot, then I also rethink the learning rate. And if memory's tight, gradient accumulation helps.

Summary