When Tools Fail

At two in the morning, the courier's service goes down. A normal program would crash. But an agent has something better than a crash. It can read the error, understand it, and decide what to do next. If you let it.

• The courier service goes down
• A program would crash
• An agent can read the error

Turn failures into information

This function fails, because the service can't be reached. So the safe version catches the failure, and it returns a message rather than crashing. Look at that message. It doesn't just say, error. It says what happened, and what to do next. Run it. The model receives that as its observation. So it can tell the customer that it'll follow up, instead of the whole agent dying.

import requests

def fragile(city):
    return requests.get(f"http://localhost:1/{city}", timeout=1).json()

def safe(city):
    try:
        return fragile(city)
    except requests.RequestException:
        return {"error": "The courier service is down. Try again later, "
                         "or tell the customer you will follow up."}

print(safe("Oslo"))

{'error': 'The courier service is down. Try again later, or tell the customer you will follow up.'}

Let the model correct itself

Good errors let the model fix its own mistakes. Here, a model passes a lower case ID. The tool replies, IDs start with a capital letter. On the second try, it gets it right. Run it. We could have written code to handle that case. But with a clear message, the model corrects itself, and we don't need any special case.

from scripted import Scripted

def get_order(order_id):
    if not order_id[0].isupper():
        return {"error": "IDs start with a capital letter, such as A17."}
    return {"status": "shipped"}

model = Scripted([{"order_id": "a17"}, {"order_id": "A17"}])

for step in (1, 2):
    args = model.ask([])
    result = get_order(**args)
    print(f"Step {step}: {args} -> {result}")
    if "error" not in result:
        break

Step 1: {'order_id': 'a17'} -> {'error': 'IDs start with a capital letter, such as A17.'}
Step 2: {'order_id': 'A17'} -> {'status': 'shipped'}

ORDERS = {
    "A17": {"item": "Carbon wheelset", "price": 640, "status": "delivered", "days_ago": 12},
    "B42": {"item": "Chain", "price": 28, "status": "shipped", "days_ago": 2},
    "C09": {"item": "Helmet", "price": 95, "status": "delivered", "days_ago": 45},
}
STOCK = {"Carbon wheelset": 3, "Chain": 14, "Helmet": 0}
REFUNDS = []

def get_order(order_id: str) -> dict:
    """Look up one order by its ID, such as A17."""
    return ORDERS.get(order_id, {"error": f"No order {order_id}"})

def check_stock(item: str) -> dict:
    """Return how many of an item are in stock."""
    return {"item": item, "in_stock": STOCK.get(item, 0)}

def refund(order_id: str, amount: float) -> dict:
    """Refund money to the customer for an order."""
    REFUNDS.append((order_id, amount))
    return {"refunded": amount, "order_id": order_id}

class Scripted:
    """A stand-in model that replays a fixed list of replies."""
    def __init__(self, replies):
        self.replies = list(replies)
        self.calls = 0

    def ask(self, messages):
        self.calls += 1
        return self.replies.pop(0)

Retry with backoff

Many failures are temporary, so retrying works. But don't retry straight away, or you'll hammer a service that's already struggling. Exponential backoff waits a little longer after each failure. Those small random extras stop every client from retrying at the same moment. Run it. It failed a few times, waited longer each time, and then it worked. And it does have a limit, so it won't try forever.

import random, time

def flaky():
    if random.random() < 0.6:
        raise TimeoutError("service timed out")
    return "ok"

def with_retries(fn, tries=5):
    for n in range(tries):
        try:
            return fn()
        except TimeoutError:
            wait = 0.05 * 2 ** n + random.random() * 0.05
            print(f"Try {n + 1} failed, waiting {wait:.2f}s")
            time.sleep(wait)
    return {"error": "still failing after retries"}

random.seed(4)
print(with_retries(flaky))

Try 1 failed, waiting 0.06s
Try 2 failed, waiting 0.11s
Try 3 failed, waiting 0.22s
ok

Never refund twice

Here's a danger that's unique to retries. What if a refund works, but the reply gets lost? The agent thinks that it failed, so it tries again, and the customer gets paid twice. The fix is called idempotency. Every action carries a unique key, and a repeated key does nothing new. Run it. Three requests, and one payment. Every tool that changes something in the real world needs this.

SEEN = {}
PAID = []

def refund(order_id, amount, key):
    if key in SEEN:
        return SEEN[key]
    PAID.append(amount)
    SEEN[key] = {"refunded": amount}
    return SEEN[key]

for attempt in range(3):
    refund("A17", 40, key="ticket-881-refund")

print("Times requested: 3")
print("Times paid:", len(PAID))

Times requested: 3
Times paid: 1

Summary