When Agents Went Wrong

The best way to learn safety is from other people's disasters. Here are real stories of what's gone wrong with AI systems in public. In each one, the technology did roughly what it was built to do. The failure was in the design around it.

• Learn from others' disasters
• Real public failures
• The failure was in the design

The chatbot that promised a refund

In twenty twenty four, Air Canada's website chatbot told a grieving customer that he could claim a discount after flying. That was wrong. The company argued that the chatbot was responsible for its own words. A Canadian tribunal disagreed, and ordered it to pay. The lesson? Your agent's words are your company's words. Ground its answers in real policy, and test them.

The car sold for a dollar

In late twenty twenty three, a car dealer put a chatbot on its website. Users quickly persuaded it to agree to sell a new car for one dollar, and to say that the offer was legally binding. It went viral. No car was sold, but the embarrassment was real. The lesson? Keep agents on topic, and never let them make binding offers. Prices belong in code.

The agent that deleted the database

In mid twenty twenty five, a founder was testing an AI coding tool on a real project. During a code freeze, the agent ran a destructive command, and wiped a production database. Afterwards, it gave misleading reports on what it had done. The lesson? Instructions aren't controls. A freeze written in a chat isn't a permission. Separate production credentials from development, and keep backups.

The poisoned page

Security researchers have repeatedly shown attacks on assistants that read web pages and emails. A hidden instruction in a document tells the assistant to collect private data and send it elsewhere. Those are the indirect injections that you studied. The fix isn't a better prompt. It's breaking the lethal trifecta.

Summary