Retrieval: Chunking and Ranking

Retrieval: Chunking and Ranking

File Search relies on a complete retrieval pipeline: your documents are split into chunks, vectorized, indexed, and then ranked by relevance when a query arrives. Understanding this pipeline lets you optimize response quality.

The Retrieval Pipeline

When you upload a file to a vector store, OpenAI automatically executes:

1. Parsing: Extracts raw text from the source format
2. Chunking: Splits text into controlled-size segments
3. Embedding: Vectorizes each chunk
4. Indexing: Stores vectors in the vector index

At query time:

1. Query Embedding: The question is vectorized
2. Vector Search: Identifies the closest chunks
3. Ranking: Re-ranks results by relevance
4. Injection: Injects the best chunks into the model’s context

Configuring the Chunking Strategy

Chunking is the setting that drives everything else, and the trade-off is straightforward. Short chunks yield precise search results but provide extracts stripped of context—you find the right sentence but lose what it refers to. Long chunks preserve context but bury relevant information in irrelevant text, which weakens the similarity score.

There is no universal value: the right setting depends on your document structure, exactly as summarized in the table below.

from openai import OpenAI

client = OpenAI()

# Default strategy: auto
vector_store = client.vector_stores.create(
    name="legal-base",
    chunking_strategy={"type": "auto"}
)

# Custom strategy: smaller chunks for higher precision
vector_store = client.vector_stores.create(
    name="support-faq",
    chunking_strategy={
        "type": "static",
        "static": {
            "max_chunk_size_tokens": 400,
            "chunk_overlap_tokens": 100
        }
    }
)

Choosing the Right Chunk Size

Impact of Overlap

Overlap solves a specific problem: information straddling a chunk boundary would otherwise be cut in half, meaning neither half would match the query. By overlapping chunks, you guarantee that every sentence appears intact in at least one segment. The cost is redundancy—an overlap of 300 tokens on 800-token chunks stores nearly 40% duplicate text.

# Original text (simplified)
# "Clause 5.1 states the supplier must deliver within 30 days.
#  In case of delay, penalties of 1% per day apply.
#  Clause 5.2 specifies penalties are capped at 15%."

# Without overlap: Clause 5.1 and penalties might end up in different chunks
# With overlap: the link between clause and penalties is preserved

Relevance Threshold and Ranking

Ranking determines which chunks are retained. You can configure a filter:

response = client.responses.create(
    model="gpt-5.6-terra",
    input="What is the return policy?",
    tools=[{
        "type": "file_search",
        "vector_store_ids": [vector_store.id],
        "max_num_results": 10,
        "ranking_options": {
            "ranker": "auto",
            "score_threshold": 0.5
        }
    }]
)

The score_threshold filters out results below a relevance threshold. A value of 0.0 returns everything, while 1.0 would be too restrictive. In practice, 0.3 to 0.6 yields good results.

Optimizing Retrieval Quality

Preparing Documents

This is the most cost-effective lever in this entire lesson, and the only one that costs nothing in settings: a well-structured document indexes well. Use explicit headings, an introductory sentence for each section, and avoid tables converted into text mush. Conversely, no chunking setting will save a PDF whose extraction produces entangled columns.

# Bad: Scanned PDF with approximate OCR
# -> Incoherent chunks, poor-quality embeddings

# Good: Structured document with headings and sections
# -> Chunks aligned with logical sections

# Pattern: Pre-process before upload
def prepare_document(raw_text: str) -> str:
    """Cleans and structures text before indexing."""
    # Remove repetitive headers/footers
    lines = raw_text.split("\n")
    lines = [l for l in lines if not is_header_footer(l)]

    # Normalize spaces
    text = "\n".join(lines)
    text = re.sub(r"\n{3,}", "\n\n", text)

    return text

Enriching with Metadata

Metadata allows pre-filtering before vector search, which improves precision:

# Upload with rich metadata
documents = [
    {"file": "hr-policy-2026.pdf", "department": "hr", "type": "policy", "year": 2026},
    {"file": "supplier-contract-a.pdf", "department": "procurement", "type": "contract", "year": 2026},
    {"file": "it-security-guide.pdf", "department": "it", "type": "guide", "year": 2025},
]

for doc in documents:
    with open(doc["file"], "rb") as f:
        file_obj = client.files.create(file=f, purpose="assistants")

    client.vector_stores.files.create(
        vector_store_id=vector_store.id,
        file_id=file_obj.id,
        attributes={
            "department": doc["department"],
            "document_type": doc["type"],
            "year": doc["year"]
        }
    )

Measuring Retrieval Quality

Without measurement, any chunking setting is just intuition. A set of twenty questions where you know the expected answer and the containing document is enough to objectify the process: change a parameter, re-run the test set, and compare. This is little work compared to hours spent debating chunk size without data.

def evaluate_retrieval(test_questions: list[dict], vs_id: str) -> dict:
    """Evaluates retrieval quality on a test set.

    Each element in test_questions contains:
    - question: the question to ask
    - expected_answer: keywords or phrases expected in the response
    """
    results = {"total": 0, "relevant": 0, "irrelevant": 0}

    for test in test_questions:
        response = client.responses.create(
            model="gpt-5.6-terra",
            input=test["question"],
            tools=[{
                "type": "file_search",
                "vector_store_ids": [vs_id],
                "max_num_results": 5
            }]
        )

        answer = response.output_text.lower()
        found_keywords = sum(
            1 for word in test["expected_answer"]
            if word.lower() in answer
        )

        results["total"] += 1
        if found_keywords >= len(test["expected_answer"]) * 0.5:
            results["relevant"] += 1
        else:
            results["irrelevant"] += 1

    results["relevance_rate"] = results["relevant"] / results["total"]
    return results

Advanced Strategies

Multi-Vector-Store

Query multiple document bases in parallel:

response = client.responses.create(
    model="gpt-5.6-terra",
    input="Compare our HR policy with legal obligations",
    tools=[{
        "type": "file_search",
        "vector_store_ids": [
            vs_internal_policies,
            vs_legal_texts
        ],
        "max_num_results": 15
    }]
)

Decomposed Queries

A question containing multiple requests produces an average vector that resembles none of them, leading to poor search results for all. Breaking it down into separate sub-questions yields significantly better results—this is the same principle as web search: a precise query beats an exhaustive one.

def decomposed_search(complex_question: str, vs_id: str) -> str:
    """Decomposes a complex question into sub-searches."""
    # Step 1: Decompose the question
    decomposition = client.responses.create(
        model="gpt-5.6-terra",
        input=f"Decompose this question into 2-3 simple sub-questions: {complex_question}",
    )

    # Step 2: Search for each sub-question
    sub_results = []
    for sub_q in decomposition.output_text.split("\n"):
        if sub_q.strip():
            r = client.responses.create(
                model="gpt-5.6-terra",
                input=sub_q.strip(),
                tools=[{"type": "file_search", "vector_store_ids": [vs_id]}]
            )
            sub_results.append(r.output_text)

    # Step 3: Synthesize
    synthesis = client.responses.create(
        model="gpt-5.6-terra",
        input=f"Original question: {complex_question}\n\n"
              f"Intermediate results:\n" + "\n---\n".join(sub_results) +
              "\n\nSynthesize a complete and coherent answer."
    )
    return synthesis.output_text

Key Takeaways

• static chunking with custom size and overlap improves precision
• Small chunks for FAQs, large chunks for legal documents
• score_threshold filters out irrelevant results
• Clean and structure your documents before indexing
• Evaluate retrieval quality with a test set