Voice Agents

In a chat window, waiting for three seconds is fine. On a phone call, three seconds of silence feels like the line's gone dead. People will say hello, hello, and hang up. Voice agents are the same agents that you've built, but with a hard limit of about one second.

• Three seconds is fine in chat
• On a call, it feels dead
• A limit of about one second

A latency budget

Here's where the time goes. Turning speech into text. The model producing its first word. A tool lookup. Turning text back into sound. And the network. Add those up. Run it. Twelve hundred milliseconds before the caller hears anything, and that feels slow. Now you know where to work. Stream everything, rather than waiting for complete sentences. Use a small, fast model. And say something immediately, like, let me check that, while the tool runs. People don't mind waiting, if they know that you're still there.

budget = {"Speech to text": 250, "Model first token": 400, "Tool lookup": 300,
          "Text to speech first sound": 150, "Network": 100}

total = sum(budget.values())
for step, ms in budget.items():
    print(f"{step:<28} {ms:>4} ms")
print(f"Before the caller hears anything: {total} ms")
print("Feels natural" if total < 1000 else "Feels slow")

Speech to text                250 ms
Model first token             400 ms
Tool lookup                   300 ms
Text to speech first sound    150 ms
Network                       100 ms
Before the caller hears anything: 1200 ms
Feels slow

What makes voice different

Voice needs its own design. People interrupt, so you've got to stop speaking as soon as they start. Keep replies very short, as nobody can skim audio. You can't show a list, so ask one question at a time. Background noise and accents cause errors, so confirm numbers by reading them back. And always allow a transfer to a person.

Summary