Review: Vision in Production

Most courses stop at accuracy. You've gone further. This round tests the knowledge that separates someone who's trained a model from someone who's shipped one.

• Most courses stop at accuracy
• You went further
• Trained versus shipped

Recap: production

Domain shift means that the world differs from your training data, so test each condition separately. Augment for the changes that you expect. Set thresholds using the cost of each mistake. Split data by object, or you'll leak. Confidence can be high while accuracy is low. And monitor the inputs, because you won't have labels.

Question 1

Question 1. A model drops from 98 to 53 percent in dim light. What's this called? A, Overfitting the labels. B, Domain shift. C, Underfitting. D, Aliasing. Pause and choose. The answer is B. The world changed, not the code.

Question 2

Question 2. What's the cheapest fix for that? A, A new architecture. B, Brightness augmentation. C, More layers. D, A larger batch. Pause and choose. The answer is B. Ours went from 53 to 97 percent.

Question 3

Question 3. How should you set a decision threshold? A, Always 0.5. B, From the cost of each mistake. C, Randomly. D, By accuracy alone. Pause and choose. The answer is B. Our cost fell by over 80 percent.

Question 4

Question 4. A video dataset gets 99 percent with a random split. What do you suspect? A, A great model. B, Frames leaking between sets. C, Too few epochs. D, Wrong labels. Pause and choose. The answer is B. Split by video.

Question 5

Question 5. How can you detect a defect that you've never seen? A, More labels. B, Learn what normal looks like. C, Lower accuracy. D, Bigger images. Pause and choose. The answer is B. That's anomaly detection.

Question 6

Question 6. Our model was 98 percent sure, and right 59 percent of the time. What does that show? A, It is calibrated. B, Confidence can't be trusted on new data. C, It needs more colour. D, It is perfect. Pause and choose. The answer is B. Add input checks.

Summary