Agent Metrics

The dashboard says that ninety nine percent of requests succeeded. Everyone's happy. But success just meant, no crash. Half of the customers still needed a person. For an agent, a normal server metric tells you almost nothing.

• 99% success on the dashboard
• Success just meant no crash
• Half still needed a person

Metrics by route

Here's some sample run data. We group by route, and look at time, cost, and how often it solved the problem. Look at the worst time as well as the median. Run it. Tracking is quick and cheap, and it always gets resolved. Refunds are slow, costly, and they only get resolved half of the time. One bad run took twenty one seconds, and cost a lot. Averages hide this. Always look at the worst cases.

import pandas as pd

runs = pd.DataFrame({
    "route": ["tracking"] * 4 + ["refund"] * 4,
    "seconds": [2.1, 1.8, 2.4, 2.0, 9.5, 7.2, 21.0, 8.1],
    "dollars": [0.002, 0.002, 0.003, 0.002, 0.02, 0.015, 0.09, 0.02],
    "resolved": [1, 1, 1, 1, 1, 0, 0, 1],
})

print(runs.groupby("route").agg(runs=("seconds", "size"),
                                median_s=("seconds", "median"),
                                worst_s=("seconds", "max"),
                                dollars=("dollars", "sum"),
                                resolved=("resolved", "mean")).round(3).to_string())

          runs  median_s  worst_s  dollars  resolved
route
refund       4      8.80     21.0    0.145       0.5
tracking     4      2.05      2.4    0.009       1.0

The numbers that matter

Watch these. Task completion, meaning that the problem was truly solved. Escalation rate, meaning how often a person was needed. Cost per solved task, rather than per run. Step count, as a rise means confusion. Slowest runs. And how often people edit or reject the agent's work.

A/B testing prompts

When you've got a new prompt, the honest test is on real traffic. Give half the users the old prompt, and half the new one, then compare how often problems get solved. Run it. Eighty seven percent against around eighty two. Our Data Science course goes deeper into how to read these results properly. Offline evals tell you that it's safe. Online tests tell you if it's better.

import random

random.seed(3)
old = [random.random() < 0.82 for _ in range(1000)]
new = [random.random() < 0.87 for _ in range(1000)]

print(f"Old prompt resolved: {sum(old) / len(old):.1%}")
print(f"New prompt resolved: {sum(new) / len(new):.1%}")

Old prompt resolved: 82.0%
New prompt resolved: 86.8%

Summary