Review: Lessons from Failure

Each of these questions comes from a failure that you've studied. If you can answer them, you'll be able to avoid repeating them.

• Real failures
• Answer them
• Don't repeat them

Recap: failures

Your agent's words are your company's words. Prices and promises belong in code. Instructions aren't controls. Separate production credentials. Break the lethal trifecta. Buy what's common, build what's special. And invest in evaluation and judgment, because they outlast each model.

Question 1

Question 1. A chatbot promised a discount that doesn't exist. Who's responsible? A, The model vendor. B, The company. C, The customer. D, Nobody. Pause and choose. The answer is B. A tribunal agreed.

Question 2

Question 2. How do you stop a bot agreeing to sell a car for a dollar? A, Politely ask it not to. B, Keep prices and offers in code. C, Use a bigger model. D, Add emojis. Pause and choose. The answer is B. It can't make binding offers.

Question 3

Question 3. An agent wiped a production database. What's the best prevention? A, A stronger prompt. B, No production credentials for the agent. C, A bigger context. D, More temperature. Pause and choose. The answer is B. Instructions aren't controls.

Question 4

Question 4. When should you buy, rather than build? A, For your unique process. B, For common tasks. C, For sensitive data. D, Never. Pause and choose. The answer is B. Thousands of people improve those.

Question 5

Question 5. How do you test a vendor's agent? A, Watch their demo. B, Run it on your own test set. C, Read their blog. D, Ask a friend. Pause and choose. The answer is B. A demo always works.

Question 6

Question 6. Which skill lasts through model changes? A, A prompt trick. B, Evaluation and judgment. C, A framework's syntax. D, A vendor's menu. Pause and choose. The answer is B. Models change, judgment doesn't.

Summary