Splitting Sentences

Split a text at every full stop. That's easy, isn't it? Then Dr. Khan arrives, and pays twelve dollars fifty for Volume two, from the UK depot. Suddenly your chunker has cut sentences in half, and your RAG system starts retrieving nonsense. Small details at this level ripple through everything.

• Split at every full stop?
• Dr. Khan arrives
• Small details ripple outward

Handling abbreviations

The naive split breaks on every full stop followed by a space. Run it. Seven pieces. Dr has been cut off from Khan, and Vol from two. The better rule uses look behind patterns. Split after sentence punctuation, but not if it follows Dr or Vol, and only if a capital letter comes next. Now we get four clean sentences. Notice that UK depot stayed together too, because a lowercase word follows it. A short list of abbreviations goes a long way. Libraries such as spaCy learn this from data, and for real work, you should use those.

import re

text = "Dr. Khan paid $12.50 for Vol. 2. It came late! Was it the U.K. depot? Nobody knows."

naive = re.split(r"[.!?]\s+", text)
print(len(naive), naive[:3])

smart = re.split(r"(?<=[.!?])(?<!Dr\.)(?<!Vol\.)\s+(?=[A-Z])", text)
print(len(smart))
for s in smart:
    print("-", s)

7 ['Dr', 'Khan paid $12.50 for Vol', '2']
4
- Dr. Khan paid $12.50 for Vol. 2.
- It came late!
- Was it the U.K. depot?
- Nobody knows.

Summary