Interview Questions, Part Two

Senior interviews often have a design round. They'll say something like, design a system to detect fraud. There's no single right answer. They're watching how you think. And you already know the framework, because you've used it all through this course.

• Design a fraud system
• No single right answer
• You know the framework

A framework for design questions

Follow six steps out loud. Clarify the goal, and ask about the costs of each mistake. Define the data, and how labels are obtained. Choose a baseline, and a metric. Propose features and a model, and start simple. Plan how to test it, offline and then in a live experiment. And finally, cover deployment, monitoring, and retraining. Talking through these steps calmly will put you far ahead of most candidates.

1. Clarifygoal and costs
2. Datafeatures and labels
3. Baselineand the metric
4. Modelstart simple
5. Shiptest and monitor

Example: design a fraud detector

Let's practise. The goal is to block fraud, without annoying good customers, so we ask about the cost of each error. Labels come from chargebacks, which arrive weeks later, so training data is delayed. Fraud is rare, so we use precision recall. Features include spending compared to the customer's normal behaviour. We start with boosting. We split by time, because criminals adapt. And we monitor for drift, and send borderline cases to human reviewers.

Questions to ask them

At the end, you'll be asked if you have any questions. Don't say no. Ask, how do you measure whether a model is working in production? Who owns a model after launch? How long does it take to get a new model live? What's the most valuable model that you've retired, and why? These show that you understand the real job, beyond the algorithms.

Summary