Faster and Cheaper

Juno works, and customers love it. Then the bill arrives, and it's three times what anyone expected. Don't worry. The same tricks work almost everywhere, and together they can cut costs dramatically.

• Customers love it
• The bill is three times too big
• The same tricks work everywhere

Caching answers

Many questions repeat. Why pay twice? A cache remembers results. We run the same lookup three times. Run it. The first call is slow. After that, it's almost instant, and it's free. Only cache what's safe, though. A policy is fine. An order status changes, so it'd need a short lifetime.

import functools, time

def expensive_lookup(question):
    time.sleep(0.2)
    return "Returns are free within thirty days."

@functools.lru_cache(maxsize=1000)
def cached_lookup(question):
    return expensive_lookup(question)

for _ in range(3):
    start = time.time()
    cached_lookup("What is your returns policy?")
    print(f"{(time.time() - start) * 1000:.0f} ms")

210 ms
0 ms
0 ms

Prompt caching

Agents resend the same system prompt and tools at each step. Model providers offer prompt caching, which charges far less for text that hasn't changed. These are example rates, so check real ones. Run it. A saving of seventy percent on a six step run. There's one key rule. Put stable content first, and things that change last, because a cache only matches from the start.

price, cached_price = 0.30, 0.03
stable, varying, calls = 6000, 400, 6

no_cache = calls * (stable + varying) / 1e6 * price
first = (stable + varying) / 1e6 * price
rest = (calls - 1) * (stable * cached_price + varying * price) / 1e6
with_cache = first + rest

print(f"No cache:   ${no_cache:.5f}")
print(f"With cache: ${with_cache:.5f}")
print(f"Saving:     {1 - with_cache / no_cache:.0%}")

No cache:   $0.01152
With cache: $0.00342
Saving:     70%

Routing by difficulty

Most tickets are easy, so they don't need your strongest model. A simple rule sends hard or risky tickets to a stronger model, and everything else to a cheap one. Run it. The tracking question goes cheap, and the injury goes strong. If most of your traffic is easy, this cuts your bill a lot, and hardly touches quality.

from llm import ask

CHEAP, STRONG = "cheap model", "strong model"

def choose_model(ticket):
    hard_words = ["legal", "injury", "warranty", "chargeback"]
    long_ticket = len(ticket.split()) > 40
    return STRONG if long_ticket or any(w in ticket.lower() for w in hard_words) else CHEAP

for ticket in ["Where is B42?", "I crashed and I'm injured. Is this a warranty case?"]:
    print(f"{choose_model(ticket):<13} {ticket}")

cheap model   Where is B42?
strong model  I crashed and I'm injured. Is this a warranty case?

def ask(*a,**k):
    return ""

Summary