Running an NLP Project

You've got the skills. Now somebody asks you to lead a real project. Where do you start? Not with a model. Here's a plan which works, whether it's for a two person startup, or for a large bank.

• Now you lead a project
• Don't start with a model
• A plan that works

The project steps

Start by reading two hundred real examples. By hand. You'll learn more from that than from anything else. Then define success as a number. Build the simplest baseline. Write your test set. Only then try better models. Launch in shadow mode, measure it, and improve from real failures.

1. Read 200examples by hand
2. Define successa number
3. Simple baselinefirst
4. Test setbefore models
5. Shadow launchthen measure

The first week

Here's a real first week. Monday, read tickets, and talk to the people who do this job. Tuesday, write down categories and success measures. Wednesday, a keyword baseline. Thursday, a simple classifier, and a small test set. Friday, show results, including all of the mistakes. In five days, you'll know more than most teams do after three months.

What to avoid

Avoid the common traps. Don't begin with the fanciest model. Don't skip the test set. Don't measure only accuracy. Don't launch to everyone on day one. And don't forget about maintenance, because language changes, new products arrive, and models get replaced. An NLP system's never finished.

Summary