Complete RAG Architecture

Lesson updated November 3, 2026 based on OpenAI documentation: temperature and top_p are only accepted with a reasoning effort of none, whereas GPT-5.6 models reason by default (medium). Calls that set these parameters must therefore specify reasoning={"effort": "none"} (API Responses) or reasoning_effort="none" (Chat Completions). Sources: Using GPT-6 · GPT-5.6 Terra.

Objectives

• Understand the end-to-end RAG architecture
• Implement a functional RAG pipeline
• Identify components to optimize

What is RAG?

RAG (Retrieval-Augmented Generation) combines document search with text generation by an LLM. Instead of relying solely on its internal knowledge, the model receives relevant documents as context, then drafts its response based on that context. Everything you have built since the start of this course—embeddings, indexing, search, evaluation—becomes the upstream half of a complete system here.

Why RAG?

Four reasons justify it, and they compound. First, proprietary data: your remote work policy, internal procedures, and product catalog never appeared in the model’s training data, and no prompt will make them appear there. Second, recent information: anything post-dating training is invisible to the model, whereas an index updates the same day. Verifiability matters just as much, because every response is anchored in identifiable sources that the user can open and check—transforming the nature of the trust placed in the system. Finally, cost works in its favor: adding knowledge via RAG costs far less than fine-tuning, and corrections happen by deleting a document rather than retraining a model.

Pipeline Stages

Read RAG in two distinct phases that you must not confuse. Ingestion happens offline, once and for all or whenever the corpus updates:

Documents → Chunking → Embeddings → Vector Storage

The query happens online, for every question asked, and must remain fast:

Question → Embedding → Search → Context + Question → LLM → Response

This separation has a practical consequence: ingestion costs are paid once and can be heavy, while query costs are paid per call and dictate the user experience. You can therefore afford sophisticated ingestion processing, but you must stay lean on the query side.

Complete Implementation

The following class materializes the three phases in three distinct methods, plus a fourth that chains them together. This separation isn’t cosmetic: it lets you test search without consuming generation tokens, and replay generation on a fixed context to compare two prompts.

from openai import OpenAI
import numpy as np

client = OpenAI()

class PipelineRAG:
    def __init__(
        self,
        model_embedding: str = "text-embedding-3-large",
        model_generation: str = "gpt-5.6-terra"
    ):
        self.client = OpenAI()
        self.model_embedding = model_embedding
        self.model_generation = model_generation
        self.chunks: list[dict] = []
        self.embeddings: np.ndarray | None = None

    # --- Ingestion Phase ---

    def ingest(self, documents: list[dict]):
        """Ingests documents {id, text, metadata}."""
        self.chunks = documents
        texts = [d["text"] for d in documents]

        # Generate embeddings in batches
        all_emb = []
        for i in range(0, len(texts), 100):
            batch = texts[i:i + 100]
            resp = self.client.embeddings.create(
                input=batch, model=self.model_embedding
            )
            batch_emb = sorted(resp.data, key=lambda x: x.index)
            all_emb.extend([e.embedding for e in batch_emb])

        self.embeddings = np.array(all_emb, dtype=np.float32)
        print(f"Ingestion complete: {len(self.chunks)} chunks indexed")

    # --- Retrieval Phase ---

    def search(self, question: str, k: int = 5) -> list[dict]:
        """Finds the k most relevant chunks."""
        query_emb = self.client.embeddings.create(
            input=question, model=self.model_embedding
        ).data[0].embedding

        scores = self.embeddings @ np.array(query_emb)
        top_k = np.argsort(scores)[-k:][::-1]

        return [
            {**self.chunks[i], "score": float(scores[i])}
            for i in top_k
        ]

    # --- Generation Phase ---

    def generate(self, question: str, contexts: list[dict]) -> str:
        """Generates a response based on retrieved contexts."""
        # Build the prompt
        formatted_context = "\n\n---\n\n".join(
            f"[Source: {c.get('id', 'unknown')}]\n{c['text']}"
            for c in contexts
        )

        messages = [
            {
                "role": "system",
                "content": (
                    "You are an assistant that answers questions "
                    "based solely on the provided documents. "
                    "If the information is not in the documents, "
                    "state this clearly. Cite your sources."
                )
            },
            {
                "role": "user",
                "content": (
                    f"Reference documents:\n\n{formatted_context}\n\n"
                    f"---\n\nQuestion: {question}"
                )
            }
        ]

        response = self.client.chat.completions.create(
            model=self.model_generation,
            reasoning_effort="none",  # temperature requires effort none
            messages=messages,
            temperature=0.2
        )

        return response.choices[0].message.content

    # --- Complete Pipeline ---

    def answer(self, question: str, k: int = 5) -> dict:
        """Complete RAG pipeline: search → generation."""
        contexts = self.search(question, k=k)
        response = self.generate(question, contexts)

        return {
            "question": question,
            "response": response,
            "sources": [
                {"id": c["id"], "score": c["score"],
                 "excerpt": c["text"][:150]}
                for c in contexts
            ]
        }

Three details in the generation phase drive most of the quality. The system message mandates answering only from the provided documents and admitting when information is missing: without this instruction, the model will fill gaps with its general knowledge, yielding a plausible but invented response. The [Source: id] prefix before each excerpt gives the model something specific to cite, rather than a vague “according to the documents.” Finally, a temperature of 0.2 reduces variability: for factual questions against a document base, you want consistency, not creativity.

Usage

Let’s test the pipeline on a minimal business case—three internal policies, and a question that doesn’t copy any of their exact wording.

# Prepare documents
documents = [
    {
        "id": "leave-policy",
        "text": "Employees receive 25 days of paid leave "
                 "per year. Leave must be requested at least 2 weeks "
                 "in advance via the HR portal.",
        "metadata": {"type": "hr", "date": "2026-01"}
    },
    {
        "id": "remote-work-policy",
        "text": "Remote work is authorized up to 3 days per week. "
                 "Tuesdays and Thursdays are mandatory office days. "
                 "Written agreement with the manager is required.",
        "metadata": {"type": "hr", "date": "2026-01"}
    },
    {
        "id": "reimbursement-procedure",
        "text": "Expense reports must be submitted within 30 days "
                 "of the expense. Attach the original receipt. "
                 "Validation by the direct supervisor required for amounts over 200 euros.",
        "metadata": {"type": "finance", "date": "2026-01"}
    },
]

# Create the pipeline
rag = PipelineRAG()
rag.ingest(documents)

# Ask a question
result = rag.answer("How many days of remote work per week?")
print(result["response"])
print("\nSources used:")
for s in result["sources"]:
    print(f"  [{s['score']:.3f}] {s['id']}: {s['excerpt']}")

Push the exercise further by asking a question no document answers, such as one about the company health plan. The pipeline will still return three sources—search always returns its k best candidates—and it is the system message that must prevent the model from inventing an answer. Verifying this behavior before production is better than discovering it through a complaint.

Optimization Points

Each pipeline step can be improved independently, allowing progress through small, measurable tweaks.

The following lessons detail each of these optimizations. Tackle them in this order: chunking determines what can be retrieved, and no reranking will fix a poorly chunked document.

Summary

• RAG = Search + Context + LLM Generation
• The pipeline splits into ingestion (offline) and query (online)
• Each component (chunking, embedding, search, prompt, generation) can be optimized independently
• RAG is more flexible and less costly than fine-tuning for adding knowledge