Week 9 • Day 1Week 9

Day 1:Week 9, Day 1: Structure-Aware Document Parsing — Markdown Hierarchy, Tables & Cleaning

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Week 9, Day 1: Structure-Aware Document Parsing — Markdown Hierarchy, Tables & Cleaning


1. "Garbage In, Hallucination Out": Why Naive Text Splitting Breaks RAG

Suppose an enterprise SRE runbook contains:

markdown
# Payments Service Runbook ## 1. Stripe Webhook Timeout Run `kubectl rollout restart deploy/stripe-worker`. ## 2. Database Connection Pool Exhausted Run `kubectl rollout restart deploy/pgbouncer`.

If you naively strip headers or split right between the heading ## 2. Database Connection Pool Exhausted and its command Run kubectl rollout restart deploy/pgbouncer, you get an orphan chunk containing only:

"Run kubectl rollout restart deploy/pgbouncer."

When a user asks "What command do I run when the database connection pool is exhausted?", that orphan chunk has zero mention of "database" or "connection pool"!

The Solution: Breadcrumb Header Path Inheritance

A production parser tracks the active heading stack (H1 -> H2 -> H3) as it scans a document and attaches the full breadcrumb_path ("Payments Service Runbook > 2. Database Connection Pool Exhausted") to every section and table!


2. Preserving Markdown Tables Intact

Never split a Markdown table mid-row without repeating the table header row (| Metric | Threshold | Action |)! If row 15 of a table is separated from its column headers, the numbers in row 15 become meaningless to both the embedding model and the LLM.


In Simple English: Why Naive Copy-Paste Breaks AI Search (Garbage In -> Garbage Out)

Imagine ripping a 10-page recipe book across random horizontal scissor cuts — half the ingredients list lands on Strip #1 and the oven temperature lands alone on Strip #2! That is what happens when naive scripts slice Markdown or PDF documents every 500 characters blindly:

  • Table headers disappear on Chunk #2, leaving rows of orphaned numbers (| 42 | 190 |) that the AI cannot understand!
  • Markdown code blocks get cut in half, breaking Python syntax!
  • Structure-Aware Parsing tracks # -> ## -> ### headings and preserves entire Markdown tables together so every chunk retains its exact topic title ([Database Failover > Step 2: Promote Replica])!