Parsing Real Documents

Your RAG system is only as good as the text that you feed it. And real company knowledge lives in PDFs, scans, slides, and tables. Pull the text out badly, and every later step fails, no matter how clever your model is.

• RAG depends on clean text
• Knowledge lives in PDFs
• Garbage in, garbage out

Cleaning extracted text

Here's what text extracted from a PDF often looks like. Page headers repeat, and sentences break across lines. We remove the header lines with a pattern, and join broken lines, while keeping the blank lines and the table. Run it. The sentence is whole again, and the headers have gone. A small cleanup like this can raise RAG quality noticeably.

import re

page = """Bright Books Ltd          Page 3 of 12
Returns Policy

Books can be returned within
30 days if unread.

| Item      | Window  |
| Paperback | 30 days |
| Gift card | None    |

Bright Books Ltd          Page 3 of 12"""

lines = [l for l in page.split("\n") if not re.search(r"Page \d+ of \d+", l)]
text = "\n".join(lines)
text = re.sub(r"(?<![.\n|])\n(?![\n|])", " ", text)

print(text)

Returns Policy

Books can be returned within 30 days if unread.

| Item      | Window  |
| Paperback | 30 days |
| Gift card | None    |

Tools for real documents

Don't write this all by hand. PyMuPDF and pdfplumber read digital PDFs. Tools such as Docling and Unstructured understand layouts, headings, and tables. For scans, you need OCR, covered in our Computer Vision course. And modern vision language models can read a page image directly, including its tables and charts. Keep headings with each chunk, because they carry so much meaning.

Summary