Retrieval and Generation
On this page
From Vector to Answer
You know how to index documents and retrieve the ones closest to a question. One step remains: turning a list of chunks into a readable answer. This means injecting the retrieved context into a prompt so the LLM formulates a response based on your data rather than its training memories. It takes only a few lines of code, but it makes a decisive difference in quality. A well-presented context and a clear instruction matter as much as retrieval precision.
Building the RAG Prompt
The prompt structure is simple, but every choice has consequences. Number the chunks and annotate them with their source so the model can cite precisely. Separate them with a visible delimiter, ---, so the model does not merge two contradictory documents into a single claim. Most important, the instruction must explicitly restrict the answer to the provided context and allow the model to say it does not know. Without that permission, an LLM asked about missing information will fill the gap with whatever seems plausible. That is the very mechanism of hallucination.
def build_rag_prompt(question, retrieved_chunks):
"""Build the RAG prompt with retrieved context."""
# Format the context
context_parts = []
for i, chunk in enumerate(retrieved_chunks):
source = chunk.get("source", "unknown source")
context_parts.append(f"[Document {i+1} - {source}]\n{chunk['text']}")
context = "\n\n---\n\n".join(context_parts)
prompt = f"""You are an expert assistant. Answer the question using
ONLY the context provided below. If the information is not in the
context, say so clearly.
## Context
{context}
## Question
{question}
## Answer"""
return promptGenerating the Answer
The generation call is straightforward: a system message that restates the role, and a user message that carries the assembled prompt. Watch the temperature parameter, set here to 0.1. In creative writing, you want variety; in RAG, you want the opposite. You want the model to stay as close as possible to what the context says, and you want two runs on the same question to produce the same answer. A high temperature would push the model to rephrase, extrapolate, and drift away from the source.
from mistralai.client import Mistral
client = Mistral(api_key=api_key)
def generate_answer(question, retrieved_chunks, model="mistral-large-latest"):
"""Generate a RAG answer with Mistral."""
prompt = build_rag_prompt(question, retrieved_chunks)
response = client.chat.complete(
model=model,
messages=[
{
"role": "system",
"content": "You are a precise assistant that answers based on the provided context."
},
{
"role": "user",
"content": prompt,
}
],
temperature=0.1, # Low temperature for factual answers
)
return response.choices[0].message.content
# Example
question = "What are the main steps of RAG?"
results = search_index(question, index, chunks, k=3)
answer = generate_answer(question, results)
print(answer)Encapsulating the Flow in a Class
Scattered functions work for exploration, not for delivery. The RAGPipeline class bundles the state—the chunks, the embeddings matrix, and the FAISS index built once at instantiation—and three operations: retrieve encodes the question and queries the index, generate assembles the prompt and calls the model, and ask chains the two. Note what ask returns: not a string, but a dictionary containing the answer, the list of sources, and the number of chunks used. This traceability is what separates an auditable assistant from a black box. The day a user challenges an answer, you will know which passages it relied on.
import numpy as np
import faiss
from mistralai.client import Mistral
client = Mistral(api_key=api_key)
class RAGPipeline:
"""Complete RAG pipeline with Mistral and FAISS."""
def __init__(self, chunks, embeddings):
self.chunks = chunks
self.embeddings = embeddings.astype(np.float32)
# Create the FAISS index
dimension = embeddings.shape[1]
self.index = faiss.IndexFlatL2(dimension)
self.index.add(self.embeddings)
def retrieve(self, question, k=3):
"""Step 1: retrieve relevant chunks."""
response = client.embeddings.create(
model="mistral-embed",
inputs=[question],
)
query_vector = np.array(
[response.data[0].embedding], dtype=np.float32
)
distances, indices = self.index.search(query_vector, k)
results = []
for dist, idx in zip(distances[0], indices[0]):
if idx >= 0:
results.append({
"text": self.chunks[idx]["text"],
"source": self.chunks[idx].get("source", ""),
"distance": float(dist),
})
return results
def generate(self, question, context_chunks):
"""Step 2: generate the answer with context."""
prompt = build_rag_prompt(question, context_chunks)
response = client.chat.complete(
model="mistral-large-latest",
messages=[{"role": "user", "content": prompt}],
temperature=0.1,
)
return response.choices[0].message.content
def ask(self, question, k=3):
"""Full pipeline: question -> sourced answer."""
# Retrieval
context = self.retrieve(question, k=k)
# Generation
answer = self.generate(question, context)
return {
"question": question,
"answer": answer,
"sources": [c["source"] for c in context],
"n_chunks_used": len(context),
}
# Usage
rag = RAGPipeline(chunks, embeddings)
result = rag.ask("How does indexing work in FAISS?")
print(f"Question: {result['question']}")
print(f"Answer: {result['answer']}")
print(f"Sources: {result['sources']}")Three Prompt Variants by Use Case
You can specialize the base prompt depending on what you expect from the assistant. If your users must verify every claim, explicitly ask for citations in brackets. The model then attaches each sentence to a numbered document. If the answer feeds an interface or a report, a structured format enforces stable sections: a concise answer, details, sources, and a confidence level that reflects how well the context covers the question. The third variant plays a different role: it teaches the model to break down uncertainty by distinguishing what it can assert, what is missing, and where to look next. This often produces the most useful assistants in practice, because “I am missing X” beats a confident but wrong answer.
# 1. Prompt with citation instruction
prompt_cite = """Answer by citing source documents in brackets.
Example: "RAG improves accuracy [Document 1]."
Context: {context}
Question: {question}"""
# 2. Prompt with structured format
prompt_structured = """Answer in the following format:
**Answer**: [your concise answer]
**Details**: [detailed explanation]
**Sources**: [list of documents used]
**Confidence**: [high/medium/low based on context coverage]
Context: {context}
Question: {question}"""
# 3. Prompt with missing-information handling
prompt_honest = """If the context does not contain enough information to
answer completely, state clearly:
- What you can assert with certainty
- What is missing from the context
- Suggestions for obtaining the missing information
Context: {context}
Question: {question}"""On the call parameters side, three settings govern RAG behavior. A low temperature locks in factual output, max_tokens prevents answers that drift into essays, and a conservative top_p restricts sampled vocabulary to the most probable continuations.
# For RAG, use a low temperature
response = client.chat.complete(
model="mistral-large-latest",
messages=[{"role": "user", "content": prompt}],
temperature=0.1, # Factual, low creativity
max_tokens=1024, # Limit answer length
top_p=0.95, # Conservative nucleus sampling
)Key Takeaways
- The RAG prompt injects retrieved context before the question
- Use a low temperature (0.1–0.3) for factual answers
- Ask the model to cite sources for traceability
- The RAGPipeline class encapsulates the full flow: retrieve, generate, ask
- Handle cases where the context lacks sufficient information