NLP at Scale
Speed and cost.
Scribe works beautifully on a hundred tickets. Bright Books now has two hundred thousand a day. Costs that seemed tiny become a real bill, and waits that seemed short become complaints. Three simple ideas solve most of this.
- A hundred tickets is easy
- Two hundred thousand isn't
- Three simple ideas
Batching
Processing text in batches is far more efficient than one at a time, because the maths runs in parallel. We time both. Run it. Each item in a batch takes a fraction of the time. It's free, and it's usually a one line change.
import time
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2", device="cpu")
texts = ["where is my parcel"] * 256
start = time.time()
for t in texts[:32]:
model.encode(t)
one = (time.time() - start) / 32
start = time.time()
model.encode(texts, batch_size=32)
many = (time.time() - start) / 256
print(f"One at a time: {one * 1000:.1f} ms each")
print(f"In batches: {many * 1000:.1f} ms each")One at a time: 7.3 ms each In batches: 0.4 ms each
Caching
Customers ask the same questions, again and again. So store each answer, and reuse it. We normalise the text, make a key, and check the cache first. Sixteen questions are sent, in two spellings. Run it. Only one real model call. Everything else came straight from memory. In real support systems, this can remove a large share of the bill.
import hashlib
import time
cache = {}
def slow_model(text):
time.sleep(0.05)
return f"Answer to: {text}"
def cached(text):
key = hashlib.sha256(text.strip().lower().encode()).hexdigest()
if key not in cache:
cache[key] = slow_model(text)
return cache[key]
messages = ["What are your hours?"] * 8 + ["what are your hours? "] * 8
start = time.time()
for m in messages:
cached(m)
print(f"16 questions in {time.time() - start:.2f}s, model calls: {len(cache)}")16 questions in 0.05s, model calls: 1
Routing by difficulty
Here's the biggest saving of all. Don't use a big model for everything. Simple tickets go to a template or a regex. Medium ones go to a small model. And only hard ones go to the large model. We estimate a day of two thousand tickets. Run it. Routing cuts the bill by a huge factor, compared with sending everything to the large model. Most messages are easy. Use the cheapest tool that does the job.
def route(ticket):
if len(ticket.split()) < 12 and "refund" in ticket.lower():
return "regex + template", 0.0
if len(ticket.split()) < 30:
return "small model", 0.0002
return "large model", 0.003
tickets = ["refund please"] * 1200 + ["short question " * 5] * 600 + ["long story " * 40] * 200
total = sum(route(t)[1] for t in tickets)
print(f"Routed cost per day: ${total:.2f}")
print(f"All large model: ${len(tickets) * 0.003:.2f}")Routed cost per day: $0.72 All large model: $6.00
Summary
Batch, cache, and route. Those three ideas cut most NLP bills dramatically. Scribe is fast and affordable. But we skipped some of the deepest ideas in language. Next, we go back and build a language model the old way, from pure counting.
Key takeaways- Batching is free speed
- Caching removes repeat work
- Route easy tickets to cheap tools