Chunking

Most RAG systems that fail, fail here. Cut your documents too small, and each piece loses its meaning. Too large, and the search gets vague, and you waste tokens. And if you cut straight through a sentence, the answer gets split in half. How you cut your text is everything.

• Most RAG failures start here
• Too small loses meaning
• Too large gets vague

Two ways to cut

Here are two strategies. Fixed size chunking takes a set number of characters, with an overlap, so that nothing is lost at a boundary. By sentence respects the natural breaks. Run it. Look at the fixed chunks. They cut through the middle of words and sentences. The sentence version keeps each fact whole. In practice, chunking by paragraphs and headings works best for most documents, keeping each piece about one idea.

text = ("Books can be returned within 30 days if unread. "
        "Refunds reach your card in 5 working days. "
        "Gift cards cannot be refunded. "
        "Damaged books are replaced free of charge. "
        "Contact us within 14 days of delivery.")

def chunk(text, size=70, overlap=20):
    out, start = [], 0
    while start < len(text):
        out.append(text[start:start + size])
        start += size - overlap
    return out

def by_sentence(text):
    return [s.strip() + "." for s in text.split(".") if s.strip()]

print("Fixed size:")
for c in chunk(text)[:3]:
    print("  ", repr(c))
print("By sentence:")
for s in by_sentence(text)[:3]:
    print("  ", repr(s))

Fixed size:
   'Books can be returned within 30 days if unread. Refunds reach your car'
   'funds reach your card in 5 working days. Gift cards cannot be refunded'
   's cannot be refunded. Damaged books are replaced free of charge. Conta'
By sentence:
   'Books can be returned within 30 days if unread.'
   'Refunds reach your card in 5 working days.'
   'Gift cards cannot be refunded.'

Chunking rules of thumb

Here are some rules of thumb. Aim for a few hundred tokens per chunk. Add a small overlap. Respect structure such as headings and tables. Attach metadata, like the file name and date, so that you can filter and cite. And test different sizes, because the best choice depends on your documents and your questions.

Summary