Summarising the Past

When you tell a friend about a long film, you don't repeat every line. You give the key points. When an agent's conversation gets long, it can do the same thing. Older turns become a short summary, and all of the important facts survive.

• You don't repeat a whole film
• You give the key points
• Older turns become a summary

Summarising a conversation

Here's a chat with several useful details, including a name, an order, and a local bike shop. We ask for a very short summary, and we tell it what to keep. That instruction matters. Run it. The key facts all survived, in less than half of the tokens. People call this compaction, and long running coding agents use it to work for hours.

from llm import ask

history = """Customer: Hi, I'm Priya, I race on weekends.
Juno: Hi Priya! How can I help?
Customer: Order A17 wobbles at speed. I'm 62 kg and ride carbon rims only.
Juno: Sorry to hear that. Was it professionally fitted?
Customer: Yes, by my local shop, Spoke House, last Tuesday.
Juno: Thanks. I'll check the warranty."""

summary = ask("Summarise this chat in under 30 words. Keep names, order IDs, and facts:\n" + history)

print(summary)
print(f"{len(history) // 4} tokens became {len(summary) // 4}")

Priya reported that order A17 wobbles at speed after being fitted by Spoke House last Tuesday. Juno is checking the warranty.
76 tokens became 31

from google import genai
from google.genai import types

client = genai.Client()

def ask(prompt, system=None, schema=None):
    config = types.GenerateContentConfig(temperature=0.2, system_instruction=system)
    if schema:
        config.response_mime_type = "application/json"
        config.response_schema = schema
    r = client.models.generate_content(model="gemini-flash-latest",
                                       config=config, contents=prompt)
    return r.parsed if schema else r.text

Summary plus recent turns

The best recipe combines both. The system prompt stays at the top. Then comes a summary of everything old. Then the last few messages, word for word. Run it. There's the context that the model will really see. It's small, and it's got both the long term story and the exact recent words. This small function is the heart of context engineering.

def build_context(summary, recent, system):
    parts = [system]
    if summary:
        parts.append(f"Earlier in this conversation: {summary}")
    parts.extend(recent)
    return "\n".join(parts)

print(build_context("Priya's wheelset A17 wobbles. Fitted by Spoke House.",
                    ["Customer: Any news?", "Juno: Checking now."],
                    "You are Juno."))

You are Juno.
Earlier in this conversation: Priya's wheelset A17 wobbles. Fitted by Spoke House.
Customer: Any news?
Juno: Checking now.

What to watch out for

Summaries can lose details, and they can quietly introduce mistakes. So tell the model what to keep. Summarise when you reach about seventy percent of the context limit, rather than at the very end. And keep the original messages in storage, so you can always go back to them. Never treat a summary as the truth.

Summary