Conversation Memory
How chatbots remember.
Here's a secret. Chatbots don't remember anything. Remember that from earlier? Each call starts from nothing. The illusion of memory is created by quietly sending the whole conversation again, every single time.
- Chatbots remember nothing
- Each call starts fresh
- Memory is an illusion
A conversation
The chat object keeps the history for us. In the first message, we say our name. In the second, we ask what it knows. Run it. It remembers our name and our favourite genre. But look at the last line. There's the stored list of messages, and all of it gets sent again each time. So long chats cost more and more.
from google import genai
from google.genai import types
client = genai.Client()
chat = client.chats.create(model="gemini-flash-latest", config=types.GenerateContentConfig(
system_instruction="You are Scribe. Reply in one short sentence."))
print(chat.send_message("Hi, my name is Sam and I love thrillers.").text.strip())
print(chat.send_message("What's my name and favourite genre?").text.strip())
print("Messages stored:", len(chat.get_history()))Welcome, Sam, I stand ready to chronicle your next great thriller. Your name is Sam, and your favourite genre is thrillers. Messages stored: 4
Trimming the history
History can't grow forever, because of the context window and the cost. The simple fix is to keep only the most recent turns that fit within a budget. We walk backward through the conversation. Run it. Older turns get dropped, and the latest ones stay. A better way is to summarise the old turns into a few sentences. For long term memory, store key facts about a customer in a database, and retrieve them, just like RAG.
history = [("user", "hi " * 300), ("model", "hello " * 300),
("user", "my order is late " * 100), ("model", "sorry " * 200),
("user", "what can you do?")]
def trim(history, limit=500):
kept, used = [], 0
for role, text in reversed(history):
cost = len(text.split())
if used + cost > limit:
break
kept.append((role, text))
used += cost
return list(reversed(kept)), used
kept, used = trim(history)
print("Turns before:", len(history))
print("Turns kept: ", len(kept), " tokens about:", used)Turns before: 5 Turns kept: 2 tokens about: 204
Summary
Memory means resending the history, trimming it, and storing long term facts. Now, a very practical headache. Real documents aren't neat text. They're messy PDFs. Next, reading documents properly.
Key takeaways- History is resent every time
- Trim or summarise old turns
- Long-term facts go in a database