DataArena
Chunking Strategies & Boundary Optimization
RAG, Vector Search & Embeddings • RAG Architecture9 min read

Chunking Strategies & Boundary Optimization

Fixed-size, sentence splitting, semantic chunking, and the mandatory 15–20% sliding window overlap.

Practice in Prompt LabTest in AI Interview

Core Conceptual Brief

Garbage in, garbage out: RAG retrieval fails if text chunks bisect ideas or lose contextual references. Master chunk sizing, header inheritance, and sliding window overlaps.

Theoretical & Empirical Architecture

Embedding models have maximum token limits (e.g. 512, 1024, or 8192 tokens), but even within those limits, embedding a massive 20-page document into a single vector compresses too many disparate concepts, diluting specific facts.

Chunking breaks long texts into discrete, semantically cohesive units. The way you chunk determines whether relevant information can be discovered during search.

Under the Hood: Key Engineering Mechanics

1The Mandatory 15–20% Sliding Window Overlap

If you split text at exactly 500 characters, critical relationships (like a cause in sentence A and effect in sentence B) are frequently severed across chunk boundaries. Maintaining a 15–20% overlap (e.g. 500 tokens with 100 token overlap) guarantees that boundary sentences appear together in at least one retrieved chunk.

2Document Hierarchy & Header Prepending

A raw chunk that says "The refund period is 14 days" lacks meaning if it was pulled from §3.2 of the Enterprise Hardware Warranty. When creating chunks from markdown or HTML, prepend the document title and parent headers to every chunk: [Doc: Hardware Warranty > Section: Enterprise Returns] The refund period is 14 days.

3Semantic & Parent-Document Chunking

Modern advanced RAG uses small chunks (200 tokens) for vector search indexing to maximize precise semantic matches, but returns the larger parent chunk (1000 tokens) or surrounding window to the LLM for generation, providing full surrounding context.

Recursive Character Splitter with Overlap & Header Metadata

def chunk_document_with_overlap(
    text: str, 
    chunk_size: int = 500, 
    overlap: int = 100, 
    doc_header: str = ""
) -> list[dict]:
    step = chunk_size - overlap
    chunks = []
    
    for i in range(0, len(text), step):
        chunk_text = text[i : i + chunk_size]
        # Prepend hierarchical context so embedding vector has high semantic density
        enriched_content = f"{doc_header}\n{chunk_text}" if doc_header else chunk_text
        
        chunks.append({
            "chunk_id": len(chunks),
            "start_char": i,
            "end_char": i + len(chunk_text),
            "content": enriched_content
        })
        if i + chunk_size >= len(text):
            break
            
    return chunks

doc = "Enterprise SLA Agreement. Chapter 4: Uptime Guarantees... [Long text]"
chunks = chunk_document_with_overlap(doc, chunk_size=300, overlap=60, doc_header="[SLA: Chapter 4]")
print(f"Generated {len(chunks)} chunks with 20% overlap.")

Production Guardrails & Takeaways

  • Always implement a 15% to 20% chunk overlap to prevent cutting sentences across boundaries.
  • Prepend parent document titles and section breadcrumbs to each chunk text before embedding.
  • Use small chunks for vector similarity search, but feed the expanded parent context to the LLM.

Real Staff AI Systems Interview Question

Q: What is the "lost in the middle" problem in RAG, and how does chunk size selection impact it?

Staff Answer:

Research (Liu et al., 2023) shows LLMs attend effectively to information at the very beginning and very end of their input context, but frequently overlook facts placed in the middle. Feeding 20 large, noisy 1000-token chunks increases context bloat and pushes critical facts into the neglected middle. Optimal chunking (200-400 tokens with precise reranking) keeps context compact and relevant.

Chunking Strategies & Boundary Optimization | AI Systems Engineering Guide · DataArena