Cost and Latency

Language model apps have a unique cost structure. Every call is billed by tokens, both the ones you send, and the ones you receive. A small design decision, like a longer prompt, can multiply a monthly bill. Speed matters too. In this lesson, you will estimate costs, and learn the main levers to pull them down.

• Billed by the token
• Small choices multiply
• Speed and cost levers

Estimating a monthly bill

Here is a cost model, with example prices. A function turns requests, and token counts, into dollars. The baseline is one million requests a month, with a long prompt of twenty five hundred tokens. We see the bill. Then we shorten the prompt, by tightening the instructions, and sending only the relevant passages, which cuts the bill substantially. Next, a cache answers repeated questions without calling the model at all, forty percent fewer calls. And finally, easy questions go to a much cheaper model, seventy percent of the calls. Four small decisions, together, reduce the bill from thirteen and a half thousand, to under two thousand dollars.

PRICE_IN, PRICE_OUT = 3.00, 15.00     # example dollars per million tokens

def monthly_cost(requests, in_tokens, out_tokens):
    per_call = in_tokens * PRICE_IN / 1e6 + out_tokens * PRICE_OUT / 1e6
    return requests * per_call

print("baseline        $", round(monthly_cost(1_000_000, 2500, 400), 2))
print("shorter prompt  $", round(monthly_cost(1_000_000, 900, 400), 2))
print("+ 40% cache hits $", round(monthly_cost(600_000, 900, 400), 2))
cached = monthly_cost(600_000, 900, 400)
print("+ cheap model on 70% of calls $", round(cached * (0.30 + 0.70 * 0.10), 2))

baseline        $ 13500.0
shorter prompt  $ 8700.0
+ 40% cache hits $ 5220.0
+ cheap model on 70% of calls $ 1931.4

Levers for cost and speed

Here are the main levers. Shorten prompts, and send only the relevant context. Cache answers for repeated, or similar, questions. Route by difficulty, with small cheap models for easy cases, and large ones for hard cases. Limit the length of answers. Stream the answer, so users see words immediately, which improves perceived speed. Batch work that is not urgent. And watch the bill, per feature, and per customer, so that surprises are small. Always measure quality after each change, because a cheaper answer that is wrong, is expensive.

What to monitor in production

Monitor these things for every L L M feature. Quality scores, from your judges, on a sample of real conversations. Cost per request, and per user. Latency, at the median and the slow tail. Error rates and timeouts. Refusals, and guardrail triggers. And user feedback, like thumbs up and down, which is gold for improving the golden set. Store the prompts, the retrieved passages, the model version, and the answers, with privacy controls, so that any complaint can be investigated.

Summary