Review: NLP in the Real World

Knowing the tools is one thing. Using them well is another. Here are the judgment calls.

• Knowing the tools
• Using them well
• Judgment calls

Recap: judgment

Use the cheapest tool that works, in layers. Read fifty mistakes before you change a model. Treat prompts as code, with tests. Test fairness with swapped names. Test each language separately. And design for failure, with a clear route to a person.

Question 1

Question 1. What's the first thing to try for most tickets? A, A large model. B, Rules and small models. C, Fine-tuning. D, A new dataset. Pause and choose. The answer is B. Pass only the hard cases up.

Question 2

Question 2. Accuracy is 79 percent. What's the best next step? A, A bigger model. B, Read the mistakes. C, More epochs. D, Change the seed. Pause and choose. The answer is B. Ours pointed to missing tracking examples.

Question 3

Question 3. Two prompts both score four out of four. What should you do? A, Pick the longer. B, Keep the shorter, or add harder tests. C, Use both. D, Delete the tests. Pause and choose. The answer is B. Changes must prove themselves.

Question 4

Question 4. Word counts call 'not great' positive. Why? A, A bug. B, They miss how words relate. C, Bad data. D, Wrong language. Pause and choose. The answer is B. That's what attention fixed.

Question 5

Question 5. How do you test a model for bias? A, Ask it. B, Swap names and terms, and compare outputs. C, Check its speed. D, Read its code. Pause and choose. The answer is B. Outputs shouldn't change unfairly.

Question 6

Question 6. What should a stuck bot do? A, Keep guessing. B, Hand over with the full history. C, End the chat. D, Apologise repeatedly. Pause and choose. The answer is B. Never make people repeat themselves.

Summary