Evaluator and Optimizer

No good writer publishes a first draft. They write, then an editor marks it up, and they rewrite it. Models can do the same, and it works surprisingly well. One writes, another judges against clear rules, and the feedback goes back around.

• No first drafts
• A writer and an editor
• Feedback goes around

Writer and reviewer

We've got a safety related ticket, and clear rules for the reply. The writer drafts something. The reviewer checks it against those rules, using a schema, so that it returns yes or no, plus some feedback. If it isn't good enough, we rewrite it using that feedback. And there are at most three rounds. Run it. You can watch the draft being judged, and improved. And the final reply follows the rules. Clear rules are the key. If you can't state what good means, a loop can't help you.

from pydantic import BaseModel
from llm import ask

class Review(BaseModel):
    good_enough: bool
    feedback: str

RULES = """Under 40 words. Apologise once. Give one clear next step. No promises of refunds."""
ticket = "My new chain snapped after one ride and I nearly crashed."

draft = ask(f"Write a support reply to: {ticket}")
for round_ in range(1, 4):
    review = ask(f"Check this reply against the rules.\nRules: {RULES}\nReply: {draft}", schema=Review)
    print(f"Round {round_}: {len(draft.split())} words, good enough = {review.good_enough}")
    if review.good_enough:
        break
    draft = ask(f"Rewrite this reply.\nRules: {RULES}\nFeedback: {review.feedback}\nReply: {draft}")
print(draft)

Round 1: 263 words, good enough = False
Round 2: 37 words, good enough = True
Hi [Customer Name],

I am deeply sorry to hear your chain snapped mid-ride. We take your safety very seriously.

To help us resolve this for you immediately, please reply with your order number.

Best regards,
[Your Name]

from google import genai
from google.genai import types

client = genai.Client()

def ask(prompt, system=None, schema=None):
    config = types.GenerateContentConfig(temperature=0.2, system_instruction=system)
    if schema:
        config.response_mime_type = "application/json"
        config.response_schema = schema
    r = client.models.generate_content(model="gemini-flash-latest",
                                       config=config, contents=prompt)
    return r.parsed if schema else r.text

When this pattern works

Use it when you've got clear criteria, and when feedback really does improve the result. Writing, translation, and code that must pass tests are all perfect. For code, the evaluator doesn't even need to be a model. Your test suite gives honest, exact feedback. And always limit the rounds, because reviewers can be fussy forever.

Summary