Retrieval-augmented prompting

Lesson updated November 3, 2026 based on OpenAI documentation: temperature and top_p are only accepted with a reasoning effort of none, whereas GPT-5.6 models reason by default (medium). Calls that set these parameters must therefore specify reasoning={"effort": "none"} (Responses API) or reasoning_effort="none" (Chat Completions). Sources: Using GPT-6 · GPT-5.6 Terra.

Retrieval-Augmented Prompting (RAP) combines document search with prompting to provide the model with relevant, up-to-date context. It is the natural evolution of grounding: instead of manually injecting context, you automate the search for relevant passages before every LLM call.

The difference becomes obvious at scale. Manually pasting three useful paragraphs from your documentation works for a demo; but once your documentation base reaches a few hundred pages and questions arrive continuously, no one knows which passages to paste anymore. RAP delegates this choice to an automatic component, called before every generation, whose sole role is to find what to put in the prompt.

Architecture of a RAP system

A RAP pipeline breaks down into three steps that operate at different speeds and costs. Indexing transforms your documents into vector embeddings; this happens once, offline, turning each document into a numerical vector that encodes its meaning, stored alongside the original text. Retrieval, by contrast, runs on every question: you transform the question into a vector using the same embedding model, then search for the closest document vectors. Finally, augmentation is just standard prompting—you build a system prompt containing the retrieved passages and instruct the model not to go outside this scope.

Remember this division: everything expensive happens during indexing, where you can afford to be demanding, while everything that repeats for every question must stay fast. The code below implements the first two steps using cosine similarity as the proximity measure.

from openai import OpenAI
import numpy as np

client = OpenAI()

# Step 1: Generate document embeddings
def embed_texts(texts: list[str]) -> list[list[float]]:
    """Generates embeddings for a list of texts."""
    response = client.embeddings.create(
        model="text-embedding-3-large",
        input=texts
    )
    return [item.embedding for item in response.data]

# Step 2: Search by cosine similarity
def cosine_similarity(a: list[float], b: list[float]) -> float:
    """Calculates cosine similarity between two vectors."""
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

def retrieve(query: str, documents: list[str],
             doc_embeddings: list[list[float]],
             top_k: int = 3) -> list[str]:
    """Returns the top_k most relevant documents."""
    query_emb = embed_texts([query])[0]

    scores = [
        (i, cosine_similarity(query_emb, doc_emb))
        for i, doc_emb in enumerate(doc_embeddings)
    ]
    scores.sort(key=lambda x: x[1], reverse=True)

    return [documents[i] for i, _ in scores[:top_k]]

Complete pipeline

Assembling these building blocks yields a class that fits in about thirty lines. Most of the decisions happen in the generated system prompt, and it is worth reading line by line. The model first receives the order to answer ONLY from the provided passages, which limits hallucinated answers when the documentation is silent. It then receives the instruction to cite the passage number for every claim, and this is what makes verification possible: a reviewer can trace back to the source at a glance. A temperature of 0.1 completes the setup—on an answer anchored in documents, creativity is a bug.

class RAGPipeline:
    """Minimalist RAP pipeline."""

    def __init__(self, documents: list[str]):
        self.documents = documents
        self.embeddings = embed_texts(documents)

    def query(self, question: str, top_k: int = 3) -> str:
        """Answers a question based on the documents."""
        # Retrieval
        relevant_docs = retrieve(
            question, self.documents,
            self.embeddings, top_k
        )

        # Prompt augmentation
        context = "\n\n---\n\n".join(
            f"[Passage {i+1}]\n{doc}"
            for i, doc in enumerate(relevant_docs)
        )

        system = f"""You are a technical assistant.
Answer ONLY based on the provided passages.
If the passages do not contain the answer, say so.
Cite the passage number for every claim.

RELEVANT PASSAGES:
{context}"""

        # Generation
        response = client.responses.create(
            model="gpt-5.6-terra",
            reasoning={"effort": "none"},  # temperature requires effort none
            instructions=system,
            input=question,
            temperature=0.1
        )
        return response.output_text

Chunking: splitting documents

Splitting documents into chunks is crucial for retrieval quality. If chunks are too large, they drown out relevant information; if too small, they lose context. An API manual split into ten-page blocks will always return the same generic block, regardless of the question asked. Split into isolated sentences, it will return a sentence starting with “this method” without the model knowing which method is meant. Overlap specifically mitigates this risk: each chunk repeats the end of the previous one, so information straddling a boundary remains readable from at least one side.

When your documents have a heading structure, section-based chunking is almost always superior to fixed-size chunking: an “Authentication” section stays intact, with its introduction and examples.

def chunk_document(text: str, chunk_size: int = 500,
                   overlap: int = 50) -> list[str]:
    """Splits text into chunks with overlap."""
    words = text.split()
    chunks = []

    for i in range(0, len(words), chunk_size - overlap):
        chunk = " ".join(words[i:i + chunk_size])
        if chunk.strip():
            chunks.append(chunk)

    return chunks

def chunk_by_sections(text: str) -> list[str]:
    """Splits by sections (Markdown headings)."""
    sections = []
    current = []

    for line in text.split("\n"):
        if line.startswith("## ") and current:
            sections.append("\n".join(current))
            current = [line]
        else:
            current.append(line)

    if current:
        sections.append("\n".join(current))

    return sections

Advanced retrieval strategies

Reranking

After initial retrieval, use the LLM to reorder the results. The logic is that of a funnel: vector search is fast but approximate, bringing back a dozen plausible candidates; the model, slower but more nuanced, rereads these candidates knowing the question and ranks them. You then keep only the top of the list for generation. The gain is most visible on questions where several passages talk about the same topic but only one actually answers it—three sections mention the word “quota,” but only one explains how to raise it.

def rerank(question: str, passages: list[str]) -> list[str]:
    """Reorders passages by relevance using the LLM."""
    prompt = f"""Question: {question}

Here are candidate passages. Rank them from most relevant
to least relevant for answering the question.
Return only the numbers in order (e.g., 3, 1, 2).

{chr(10).join(f"[{i+1}] {p[:200]}" for i, p in enumerate(passages))}"""

    response = client.responses.create(
        model="gpt-5.6-terra",
        reasoning={"effort": "none"},  # temperature requires effort none
        input=prompt,
        temperature=0.0
    )
    # Parse returned order and reorganize
    return response.output_text

Hypothetical Document Embedding (HyDE)

Generate an ideal hypothetical document, then search for similar documents. The problem this technique solves is vocabulary mismatch: a question asked in everyday language does not look, vectorially, like a paragraph of technical documentation. By first asking the model to write the answer it imagines, you get a text written in the register of your documents, and it is this text that you use as the query. It does not matter if the hypothetical paragraph contains approximations: it never serves as the answer, only as a probe.

def hyde_retrieve(question: str, documents: list[str],
                  doc_embeddings: list[list[float]]) -> list[str]:
    """Retrieval via HyDE."""
    # Generate a hypothetical passage
    hypo_response = client.responses.create(
        model="gpt-5.6-terra",
        reasoning={"effort": "none"},  # temperature requires effort none
        input=f"Write a short technical paragraph that would perfectly "
              f"answer this question: {question}",
        temperature=0.5
    )
    hypothetical_doc = hypo_response.output_text

    # Search for documents similar to the hypothetical passage
    return retrieve(hypothetical_doc, documents,
                    doc_embeddings, top_k=3)

Using OpenAI’s native File Search

For rapid deployment, the API offers integrated retrieval. You upload the file, create a vector store, attach the file to it, and declare the file_search tool in the call: chunking, embedding, and search are handled server-side. This is the right choice when your documents are standard and you do not care where the data lives. The homegrown pipeline keeps the advantage as soon as you need to master chunking, host the index yourself, or plug in specific reranking.

# Upload file
file = client.files.create(
    file=open("documentation.pdf", "rb"),
    purpose="assistants"
)

# Create a vector store
vector_store = client.vector_stores.create(
    name="Technical Documentation"
)

client.vector_stores.files.create(
    vector_store_id=vector_store.id,
    file_id=file.id
)

# Query with File Search
response = client.responses.create(
    model="gpt-5.6-terra",
    instructions="Answer based on the provided documentation.",
    input="How do I configure authentication?",
    tools=[{
        "type": "file_search",
        "vector_store_ids": [vector_store.id]
    }]
)

Establish a baseline

Take five to ten pages of technical documentation and implement the complete pipeline, from chunking to generation. Write ten questions whose answers are in the documents, run the pipeline on each, then repeat with reranking enabled: it is by observing on which questions the ranking changes that you understand what vector search alone misses. Finish by measuring the percentage of correctly anchored answers, those where every claim effectively refers to a provided passage. This figure is your starting point; all subsequent optimizations will be judged against it, not against the impression left by two or three randomly read answers.

Key takeaways

RAP automates the injection of relevant context into the prompt
Chunking is critical: experiment with different sizes
Reranking with the LLM significantly improves relevance
HyDE is useful when the question differs greatly from the documents’ vocabulary
OpenAI File Search simplifies deployment for standard cases