Corrective RAG

When retrieval isn’t enough

The RAG system you’ve built relies on an implicit assumption: the chunks most similar to the question are the right ones. This assumption often holds, but not always. Query your index on a topic it doesn’t cover, and FAISS will still return the five closest vectors, no matter how off-topic they are—a similarity engine doesn’t know how to say “I have nothing.” The model then receives unsuitable context, and since the prompt asks it to rely on that context, it produces an answer that looks authoritative but has no foundation. This is the most insidious hallucination scenario because the technical chain worked without error.

Corrective RAG (CRAG) inserts an evaluation step between retrieval and generation. An LLM examines each retrieved document and decides: relevant or not. If the verdict is negative, the system doesn’t just generate with what it has—it switches to a web search to find the information missing from your corpus elsewhere. The flow therefore has four stages: standard retrieval from the vector store, grading each document against the question, deciding whether to trigger a web search if at least one document is deemed irrelevant, and finally generating with the best available documents.

The document grader

Everything hinges on the grader. The trick is to ask the LLM for a binary judgment in JSON format rather than a free-form assessment: {"score": "yes"} or {"score": "no"}, nothing else. The parameter response_format={"type": "json_object"} guarantees a parsable output, and temperature=0 eliminates variability—the same question/document pair must always receive the same verdict, otherwise your pipeline behavior becomes unreproducible.

from mistralai.client import Mistral
import json

client = Mistral(api_key=api_key)

def grade_document(question, document):
    """Evaluate whether a document is relevant to a question."""
    prompt = f"""You are a relevance evaluator. Given a question
and a document, determine if the document contains relevant information
to answer the question.

Respond ONLY with a JSON object: {{"score": "yes"}} or {{"score": "no"}}

Question : {question}
Document : {document}

JSON Evaluation:"""

    response = client.chat.complete(
        model="mistral-large-latest",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        response_format={"type": "json_object"},
    )

    result = json.loads(response.choices[0].message.content)
    return result.get("score", "no") == "yes"

# Test
question = "What are the steps of RAG?"
doc_pertinent = "RAG breaks down into retrieval, augmentation, and generation."
doc_non_pertinent = "The tiramisu recipe requires mascarpone."

print(grade_document(question, doc_pertinent))      # True
print(grade_document(question, doc_non_pertinent))   # False

The test at the end of the block deserves attention. The tiramisu recipe illustrates exactly what vector similarity lets slip through: if your corpus only talks about cooking and you query it about RAG, this chunk will rank first for lack of competition. The grader, however, doesn’t compare vectors but reads the content, and rejects it.

Applied to all retrieved chunks, this judgment produces two pieces of information: the filtered list of documents to keep, and a boolean indicating whether to supplement with the web. The threshold chosen here is deliberately strict—a single discarded document is enough to trigger the fallback, on the grounds that the context is then incomplete.

def grade_documents(question, retrieved_chunks):
    """Evaluate and filter retrieved documents."""
    relevant_docs = []
    irrelevant_count = 0

    for chunk in retrieved_chunks:
        is_relevant = grade_document(question, chunk["text"])
        if is_relevant:
            relevant_docs.append(chunk)
        else:
            irrelevant_count += 1

    need_web_search = irrelevant_count > 0

    print(f"  Relevant: {len(relevant_docs)}/{len(retrieved_chunks)}")
    print(f"  Web search needed: {need_web_search}")

    return relevant_docs, need_web_search

The web safety net

When local documents aren’t enough, the system looks elsewhere. Mistral provides a built-in tool for this, web_search, which you enable through the Conversations API: the model runs the search itself, then writes its response citing its sources. This detail matters because a simple client.chat.complete call can’t search the web: the model would answer from memory, with the risk of inventing exactly what CRAG seeks to avoid. And since SDK version 3.0, chat completions actually reject the web_search tool.

The key design point is the return format: the function returns a list of dictionaries with the same shape as your local chunks, with a source field set to web_search, and adds the cited URLs. This uniformity lets downstream steps completely ignore the documents’ origin while preserving the trail for the end user.

def web_search_fallback(question):
    """Web search fallback when local documents are insufficient."""
    response = client.beta.conversations.start(
        model="mistral-medium-latest",
        inputs=question,
        instructions="Answer factually, relying on web search results.",
        tools=[{"type": "web_search"}],
        store=False,  # no need to keep the conversation on Mistral's side
    )

    texte, urls = "", []
    for entry in response.outputs:
        if entry.type != "message.output":
            continue  # ignore the entry describing tool execution
        if isinstance(entry.content, str):
            texte += entry.content
            continue
        for chunk in entry.content:
            if chunk.type == "text":
                texte += chunk.text
            elif chunk.type == "tool_reference" and chunk.url:
                urls.append(chunk.url)

    # Return as a "web" document, with cited sources
    return [{
        "text": texte,
        "source": "web_search",
        "urls": urls,
    }]

The response contains two types of entries: a tool.execution entry, which describes the launched search, and a message.output entry, whose content alternates text fragments and references (tool_reference) carrying the title and URL of the consulted pages. An organization administrator can restrict queried domains via an allowlist or blocklist; filtered results disappear without warning, so an overly narrow allowlist can leave the search with no results.

Official source: Websearch (Mistral documentation)

The assembled pipeline

The CorrectiveRAG class reprises the standard pipeline structure by inserting grading. The default k=4 is slightly more generous than the usual three chunks, which makes sense: since filtering will discard some, better to start with a margin. The ask method orchestrates the four stages and displays each one, letting you follow the system’s reasoning in real time—how many documents survived grading, whether the fallback triggered. The return dictionary also exposes web_search_used: this is a valuable indicator to monitor in production, because an abnormally high web usage rate doesn’t signal a CRAG flaw but a gap in your corpus.

class CorrectiveRAG:
    """Corrective RAG pipeline with evaluation and web fallback."""

    def __init__(self, chunks, embeddings, index):
        self.chunks = chunks
        self.embeddings = embeddings
        self.index = index

    def retrieve(self, question, k=4):
        """Step 1: standard retrieval."""
        response = client.embeddings.create(
            model="mistral-embed",
            inputs=[question],
        )
        query_vec = np.array([response.data[0].embedding], dtype=np.float32)
        distances, indices = self.index.search(query_vec, k)

        results = []
        for dist, idx in zip(distances[0], indices[0]):
            if idx >= 0:
                results.append({
                    "text": self.chunks[idx]["text"],
                    "source": self.chunks[idx].get("source", "local"),
                })
        return results

    def grade(self, question, documents):
        """Step 2: evaluate relevance."""
        relevant = []
        for doc in documents:
            if grade_document(question, doc["text"]):
                relevant.append(doc)
        need_web = len(relevant) < len(documents)
        return relevant, need_web

    def generate(self, question, documents):
        """Step 3: generate the answer."""
        context = "\n\n---\n\n".join([d["text"] for d in documents])
        prompt = f"""Context :\n{context}\n\nQuestion : {question}"""

        response = client.chat.complete(
            model="mistral-large-latest",
            messages=[
                {"role": "system", "content": "Answer based on the context."},
                {"role": "user", "content": prompt},
            ],
            temperature=0.1,
        )
        return response.choices[0].message.content

    def ask(self, question, k=4):
        """Full pipeline."""
        print(f"\n=== Question : {question} ===")

        # 1. Retrieval
        print("1. Retrieval...")
        docs = self.retrieve(question, k=k)

        # 2. Grading
        print("2. Grading...")
        relevant_docs, need_web = self.grade(question, docs)

        # 3. Web search fallback if needed
        if need_web:
            print("3. Fallback web search...")
            web_docs = web_search_fallback(question)
            relevant_docs.extend(web_docs)

        # 4. Generation
        print("4. Generation...")
        answer = self.generate(question, relevant_docs)

        return {
            "answer": answer,
            "sources_used": len(relevant_docs),
            "web_search_used": need_web,
        }

# Usage
crag = CorrectiveRAG(chunks, embeddings, index)
result = crag.ask("How does Mistral implement embeddings?")

print(f"\nAnswer : {result['answer']}")
print(f"Web search used : {result['web_search_used']}")

This reliability comes at a cost you must accept knowingly: with k=4, each question triggers four grading calls before generation even starts. The latency perceived by the user increases accordingly, and so does the bill. The trade-off differs by use case—for a legal or medical assistant, verification is non-negotiable; for an internal search engine where the user sees the sources and judges for themselves, it’s less critical.

Key takeaways

• Corrective RAG adds a relevance evaluation step after retrieval
• An LLM serves as a “grader” to judge each document: relevant or not
• If documents are deemed irrelevant, a fallback to web search triggers
• This approach significantly reduces hallucinations due to unsuitable context
• Grading adds latency but strongly improves answer reliability