Day 1:Week 9, Day 1: Structure-Aware Document Parsing — Markdown Hierarchy, Tables & Cleaning
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 9, Day 1: Structure-Aware Document Parsing — Markdown Hierarchy, Tables & Cleaning
1. "Garbage In, Hallucination Out": Why Naive Text Splitting Breaks RAG
Suppose an enterprise SRE runbook contains:
# Payments Service Runbook
## 1. Stripe Webhook Timeout
Run `kubectl rollout restart deploy/stripe-worker`.
## 2. Database Connection Pool Exhausted
Run `kubectl rollout restart deploy/pgbouncer`.
If you naively strip headers or split right between the heading ## 2. Database Connection Pool Exhausted and its command Run kubectl rollout restart deploy/pgbouncer, you get an orphan chunk containing only:
"Run kubectl rollout restart deploy/pgbouncer."
When a user asks "What command do I run when the database connection pool is exhausted?", that orphan chunk has zero mention of "database" or "connection pool"!
The Solution: Breadcrumb Header Path Inheritance
A production parser tracks the active heading stack (H1 -> H2 -> H3) as it scans a document and attaches the full breadcrumb_path ("Payments Service Runbook > 2. Database Connection Pool Exhausted") to every section and table!
2. Preserving Markdown Tables Intact
Never split a Markdown table mid-row without repeating the table header row (| Metric | Threshold | Action |)! If row 15 of a table is separated from its column headers, the numbers in row 15 become meaningless to both the embedding model and the LLM.
In Simple English: Why Naive Copy-Paste Breaks AI Search (Garbage In -> Garbage Out)
Imagine ripping a 10-page recipe book across random horizontal scissor cuts — half the ingredients list lands on Strip #1 and the oven temperature lands alone on Strip #2! That is what happens when naive scripts slice Markdown or PDF documents every 500 characters blindly:
- Table headers disappear on Chunk #2, leaving rows of orphaned numbers (
| 42 | 190 |) that the AI cannot understand! - Markdown code blocks get cut in half, breaking Python syntax!
- Structure-Aware Parsing tracks
# -> ## -> ###headings and preserves entire Markdown tables together so every chunk retains its exact topic title ([Database Failover > Step 2: Promote Replica])!