Rolling Out Safely

You don't launch a rocket by filling it with passengers, and hoping that it works. You test it empty, then with cargo, and then with a small crew. Agents deserve the same care. Each stage shows you something the last one couldn't.

• You don't launch with passengers
• Empty, then cargo, then crew
• Each stage teaches you something

The rollout stages

Stage one is shadow mode. The agent runs on real tickets, but nobody sees the replies. You compare them with what humans did. Stage two, it drafts, and humans approve. Stage three, it handles easy tickets by itself, for a small percentage of customers. Stage four, it grows slowly. Stage five, full launch, with people handling the exceptions.

• Shadownobody sees it
• Draftshumans approve
• Small percenteasy tickets
• Growwatch the metrics
• Full launchhumans for exceptions

A stable percentage rollout

How do you give a feature to just five percent of customers? Hash each customer's name into a bucket from zero to ninety nine. If it's below your percentage, they're in. Run it. The counts are close to five, twenty five, and a hundred percent. And the same customer always gets the same answer, so nobody flips between versions.

import hashlib

def in_rollout(customer, percent):
    bucket = int(hashlib.sha256(customer.encode()).hexdigest(), 16) % 100
    return bucket < percent

customers = [f"customer-{n}" for n in range(1000)]
for percent in [5, 25, 100]:
    count = sum(in_rollout(c, percent) for c in customers)
    print(f"{percent:>3}% rollout reaches {count} customers")
print("Same customer, same answer:", in_rollout("priya", 25), in_rollout("priya", 25))

  5% rollout reaches 56 customers
 25% rollout reaches 245 customers
100% rollout reaches 1000 customers
Same customer, same answer: True True

Every release needs a way back

Keep prompts and tool versions under version control, just like code. Have one setting that returns everyone to the old version in seconds. Define in advance what triggers a rollback, such as completion falling by five percent. And keep a person available. Automation handles the volume, and people handle the surprises.

Summary