Chunking: Splitting Your Documents Intelligently

Objectives

• Understand why chunking is critical for RAG quality
• Master different splitting strategies
• Choose the right chunk size and overlap

Why split documents?

Embedding models have a token limit: 8,191 for OpenAI. This limit is the first reason to split documents, but it’s not the most important one. Even a document well under the ceiling produces poor results when encoded as a single block because the resulting vector averages all the content and loses details. Take a 40-page HR manual covering leave policies, remote work, expense reports, and health insurance: its single embedding doesn’t truly resemble any of these four topics, and a search for “how many leave days” will rank it below a much shorter but far more targeted document.

Chunking solves this problem by breaking the document into coherent pieces, each capturing a specific idea or paragraph. This is the step where a RAG pipeline gains or loses the most quality—even before you choose an embedding model.

Strategy 1: Fixed-size chunking

The simplest method counts words and cuts at regular intervals. It completely ignores text structure, making it brutal, but it takes three lines of code and serves as an essential baseline: any more sophisticated strategy must prove it performs better than this.

def chunker_fixed_size(
    text: str,
    size: int = 500,
    overlap: int = 100
) -> list[str]:
    """Splits text into fixed-size chunks with overlap."""
    words = text.split()
    chunks = []

    for i in range(0, len(words), size - overlap):
        chunk = " ".join(words[i:i + size])
        if chunk.strip():
            chunks.append(chunk)

    return chunks

long_text = "A very long document... " * 500
chunks = chunker_fixed_size(long_text, size=300, overlap=50)
print(f"{len(chunks)} chunks generated")

The overlap parameter deserves attention. Without overlap, a cut will eventually land in the middle of an explanation: the sentence “the notice period is three months” ends up in one chunk, and “except during the probationary period” in the next. Neither answers the question correctly, and the second is even misleading. By overlapping chunks by 10–20% of their size, you guarantee that each idea exists intact at least once somewhere in the index. This is a good starting point; adjust it later based on your document density.

Strategy 2: Separator-based chunking

Instead of cutting word by word, you can respect the text’s natural boundaries. The idea is to try the strongest separator first—the double line break that separates two paragraphs—then move down to progressively weaker separators as long as the resulting piece exceeds the maximum size. A blog post will thus be split paragraph by paragraph, and only a monstrous paragraph will be fragmented by sentences.

import re

def chunker_separators(
    text: str,
    separators: list[str] = ["\n\n", "\n", ". ", " "],
    max_size: int = 1000
) -> list[str]:
    """Splits text while respecting natural separators."""
    chunks = []
    current_chunk = ""

    # Split by the strongest separator first
    segments = [text]
    for sep in separators:
        new_segments = []
        for segment in segments:
            parts = segment.split(sep)
            for part in parts:
                if len(part.split()) > max_size:
                    new_segments.append(part)
                else:
                    if (current_chunk and
                        len((current_chunk + sep + part).split()) > max_size):
                        chunks.append(current_chunk.strip())
                        current_chunk = part
                    else:
                        current_chunk = (
                            current_chunk + sep + part
                            if current_chunk else part
                        )
        segments = new_segments

    if current_chunk.strip():
        chunks.append(current_chunk.strip())

    return chunks

Strategy 3: Markdown section chunking

When your corpus consists of structured documentation, headings already do the work for you: the author has already delimited the units of meaning. Splitting at # and ## heading levels produces chunks that correspond exactly to sections, and the function below keeps the heading alongside the text, which we’ll use immediately after for enrichment.

def chunker_markdown(text: str, max_level: int = 2) -> list[dict]:
    """Splits a Markdown document by sections."""
    chunks = []
    current_section = {"title": "", "content": "", "level": 0}

    for line in text.split("\n"):
        # Detect headings
        match = re.match(r"^(#{1,6})\s+(.+)$", line)
        if match:
            level = len(match.group(1))
            title = match.group(2)

            if level <= max_level and current_section["content"].strip():
                chunks.append({
                    "title": current_section["title"],
                    "text": current_section["content"].strip(),
                    "level": current_section["level"]
                })

            if level <= max_level:
                current_section = {
                    "title": title,
                    "content": "",
                    "level": level
                }
            else:
                current_section["content"] += line + "\n"
        else:
            current_section["content"] += line + "\n"

    if current_section["content"].strip():
        chunks.append({
            "title": current_section["title"],
            "text": current_section["content"].strip(),
            "level": current_section["level"]
        })

    return chunks

Strategy 4: Semantic chunking

For text without explicit structure—a meeting summary, a call transcript—neither size nor separators know where the topic changes. Embeddings do. You encode each sentence, compare each sentence to the previous one, and cut where similarity collapses: that’s exactly where the speaker switched from budget to recruitment. Note that text-embedding-3-small is sufficient here since you’re not looking for absolute precision but a relative break.

from openai import OpenAI
import numpy as np

client = OpenAI()

def chunker_semantic(
    text: str,
    similarity_threshold: float = 0.75,
    min_sentence_length: int = 20
) -> list[str]:
    """Groups sentences by semantic similarity."""
    # Split into sentences
    sentences = re.split(r"(?<=[.!?])\s+", text)
    sentences = [s for s in sentences if len(s) > min_sentence_length]

    if len(sentences) <= 1:
        return [text]

    # Embeddings for each sentence
    resp = client.embeddings.create(
        input=sentences,
        model="text-embedding-3-small"  # small is sufficient here
    )
    embs = np.array([d.embedding for d in sorted(resp.data, key=lambda x: x.index)])

    # Find break points
    chunks = []
    chunk_sentences = [sentences[0]]

    for i in range(1, len(sentences)):
        sim = float(np.dot(embs[i], embs[i - 1]))
        if sim < similarity_threshold:
            # Semantic break detected
            chunks.append(" ".join(chunk_sentences))
            chunk_sentences = [sentences[i]]
        else:
            chunk_sentences.append(sentences[i])

    if chunk_sentences:
        chunks.append(" ".join(chunk_sentences))

    return chunks

Enriching chunks with context

An isolated chunk often loses the information that made it interpretable. “The duration is three years” means nothing if the reader—or the embedding model—doesn’t know it’s from the manufacturer warranty chapter. Prefixing each chunk with the document title and section title restores this context directly into the vector itself, and a search for “warranty duration” will then find the passage.

def enrich_chunk(chunk: str, doc_title: str, section: str = "") -> str:
    """Adds context to the chunk for better embedding."""
    prefix = f"Document: {doc_title}"
    if section:
        prefix += f" | Section: {section}"
    return f"{prefix}\n\n{chunk}"

What chunk size should you choose?

The choice involves a trade-off between precision and context: a short chunk targets finely but is amputated, while a long chunk understands everything but dilutes the signal.

There’s no universal value: the ideal size depends on your documents and queries. The only reliable method is to index the same corpus with two or three sizes, then measure using the metrics from lesson 10 on a set of real questions. A support knowledge base made of short answers tolerates 200 tokens; a legal corpus will require 800.

Summary

• Chunking is the most impactful step in the RAG pipeline
• Use fixed size as a baseline, separators to respect structure
• Semantic chunking automatically detects topic changes
• Overlap and contextual enrichment improve quality
• Test different sizes and measure with search metrics