Tracing

A customer complains that Juno gave a strange answer yesterday. What happened? Which tools did it call? What did the model see? Without a trace, you'll never know. An agent that you can't observe is an agent that you can't fix.

• A strange answer yesterday
• What did it see?
• Unobservable means unfixable

What a trace looks like

A trace is a timeline of a single run. The whole run sits at the top. Beneath it are spans, one for each model call, and for each tool. You can see how long each took, what went in, and what came out. Debugging becomes reading, rather than guessing.

A tiny tracer

Here's the idea, in a few lines. A span wraps a function call, and records its name and how long it took. Run it. There's our timeline, in JSON. Look at where the time goes. It's nearly all in model calls. In real systems you'd use a tool such as LangSmith, Langfuse, or anything built on Open Telemetry, which is becoming the standard for agent traces.

import json, time, uuid

class Tracer:
    def __init__(self, name):
        self.run, self.spans, self.name = uuid.uuid4().hex[:6], [], name

    def span(self, name, fn, *args):
        start = time.time()
        result = fn(*args)
        self.spans.append({"name": name, "ms": round((time.time() - start) * 1000)})
        return result

def slow_model(prompt):
    time.sleep(0.15)
    return "call get_order"

def get_order(order_id):
    time.sleep(0.05)
    return {"status": "shipped"}

t = Tracer("support")
t.span("model", slow_model, "Where is B42?")
t.span("get_order", get_order, "B42")
t.span("model", slow_model, "Write the reply")
print(json.dumps({"run": "demo", "spans": t.spans}, indent=1))

{
 "run": "demo",
 "spans": [
  {
   "name": "model",
   "ms": 152
  },
  {
   "name": "get_order",
   "ms": 52
  },
  {
   "name": "model",
   "ms": 154
  }
 ]
}

What to record

For each run, record the user's request, the prompt version, the model, each tool call with its arguments and result, tokens, cost, and time. And include a run ID, which you can give to support staff. But be careful. Traces hold private data, so redact them, and limit who can see them.

Summary