Structured Output

A paragraph is great for a person, but useless to a database. If Scribe is going to route tickets, it has to return exact fields, every single time. Good news. You can force a language model to reply in exactly the shape that you need.

• Paragraphs don't fit databases
• Exact fields, every time
• Force the shape

Forcing a schema

We define the shape that we want, as a Pydantic class. A category, an urgency number, an order number, and a summary. We pass that to the model as a response schema. Run it. Look at that. Perfect JSON. It read a messy message, and filled in each field, with the order number, and a high urgency because of the birthday. This one trick turns a language model into a powerful extraction engine, with no training at all. A single call does the job of several of our earlier models.

import json
from google import genai
from google.genai import types
from pydantic import BaseModel

class Ticket(BaseModel):
    category: str
    urgency: int
    order_number: str
    summary: str

client = genai.Client()
reply = client.models.generate_content(
    model="gemini-flash-latest",
    config=types.GenerateContentConfig(response_mime_type="application/json",
                                       response_schema=Ticket),
    contents="Classify this. Urgency 1 to 5. Message: Order #48213 still has "
             "not come and it was a birthday gift for tomorrow!")

print(json.dumps(json.loads(reply.text), indent=1))

{
 "category": "Delivery Delay",
 "urgency": 5,
 "order_number": "48213",
 "summary": "Order not received and urgently needed for a birthday tomorrow."
}

What will this cost?

Language models charge by the token, so you need to count them. Here we use a tokeniser to see how text splits. Run it. Hello is short, but unbelievably takes several tokens. Next, a simple estimate. Two thousand tickets a day, at four hundred tokens each, gives a daily cost. Always do this sum before launch. It decides whether to use a big model, a small one, or the free classic tools from earlier in the course.

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

for text in ["Hello!", "unbelievably", "Bright Books ships worldwide."]:
    ids = enc.encode(text)
    print(f"{len(ids):>2} tokens  {[enc.decode([i]) for i in ids]}")

tickets_per_day, tokens_each, price_per_million = 2000, 400, 0.30
daily = tickets_per_day * tokens_each / 1_000_000 * price_per_million
print(f"Estimated cost per day: ${daily:.2f}")

 2 tokens  ['Hello', '!']
 3 tokens  ['un', 'belie', 'vably']
 5 tokens  ['Bright', ' Books', ' ships', ' worldwide', '.']
Estimated cost per day: $0.24

Stopping hallucinations

Language models will invent answers if they're allowed to. A simple rule helps a lot. Tell it to use only the text that you provide, and give it an exact way to say, I don't know. Here, the policy says nothing about gift wrapping. Run it. It refuses to make something up. That's a tiny example of the single most important pattern in modern NLP. Give the model the facts. Next, we'll build that properly.

from google import genai

client = genai.Client()

question = ("Using only the policy, answer the question. If the policy does "
            "not say, reply exactly: NOT IN POLICY.\n\n"
            "Policy: Returns are accepted within 30 days if unread.\n"
            "Question: Do you offer free gift wrapping?")

print(client.models.generate_content(model="gemini-flash-latest",
                                     contents=question).text.strip())

NOT IN POLICY.

Summary