Advanced RAG: Multi-Query, HyDE, Step-Back

Objectives

• Improve retrieval using advanced techniques
• Implement multi-query, HyDE, and step-back prompting
• Know when to use each technique

The Problem: A Single Query Isn’t Enough

The reranking from the previous lesson improved how candidates were ranked, but it couldn’t help with documents that retrieval never surfaced in the first place. The most common cause of these misses isn’t the model or the index—it’s the query itself. A user might type “it doesn’t work on my phone,” while the documentation discusses “incompatibility with Android versions prior to 12.” No embedding model will spontaneously bring those two formulations closer together.

The following techniques share a single principle: instead of accepting the query as-is, we transform or multiply it before searching. They all act on recall—that is, the ability to get the right document into the candidate list.

Multi-Query: Ask the Question Differently

The first approach is the most direct. We ask the LLM to rephrase the question from several angles, run a search with each variant, and merge the results. The question “how do I cancel my subscription?” also becomes “cancellation procedure,” “stop the monthly charge,” and “delete my paid account”—three different vocabularies, three chances to hit the wording the documentation author used.

The merge step deserves a note: the np.maximum line keeps, for each document, the best score achieved across all queries. A document that matches well with only one rephrasing still rises to the top, which wouldn’t happen if we averaged the scores. The temperature of 0.7 is intentional here, because we want variety, not reproducibility.

from openai import OpenAI
import numpy as np

client = OpenAI()

def generer_multi_queries(question: str, n: int = 3) -> list[str]:
    """Generate n rephrasings of the question."""
    response = client.chat.completions.create(
        model="gpt-5.6-terra",
        messages=[{
            "role": "user",
            "content": (
                f"Generate {n} different rephrasings of this question "
                f"for document search. Each rephrasing "
                f"should cover a different angle.\n\n"
                f"Original question: {question}\n\n"
                f"Return only the rephrasings, one per line."
            )
        }],
        temperature=0.7
    )
    queries = response.choices[0].message.content.strip().split("\n")
    return [q.strip().lstrip("0123456789.-) ") for q in queries if q.strip()]

def recherche_multi_query(
    question: str,
    corpus_embeddings: np.ndarray,
    documents: list[dict],
    k: int = 5,
    n_queries: int = 3
) -> list[dict]:
    """Search with multiple rephrasings."""
    queries = [question] + generer_multi_queries(question, n_queries)
    print(f"Queries: {queries}")

    tous_scores = np.zeros(len(documents))
    for query in queries:
        query_emb = client.embeddings.create(
            input=query, model="text-embedding-3-large"
        ).data[0].embedding
        scores = corpus_embeddings @ np.array(query_emb)
        tous_scores = np.maximum(tous_scores, scores)

    top_k = np.argsort(tous_scores)[-k:][::-1]
    return [
        {**documents[i], "score": float(tous_scores[i])}
        for i in top_k
    ]

HyDE: Hypothetical Document Embeddings

HyDE tackles the problem from the other end. Instead of multiplying questions, we have the model generate a plausible answer—even if it’s partially wrong, that doesn’t matter—and then search for documents close to that answer. The question “how do I configure SSO?” produces a paragraph talking about identity providers, SAML, and metadata, exactly the vocabulary found in the real documentation.

def recherche_hyde(
    question: str,
    corpus_embeddings: np.ndarray,
    documents: list[dict],
    k: int = 5
) -> list[dict]:
    """Search with a hypothetical document (HyDE)."""
    # 1. Generate a hypothetical document
    response = client.chat.completions.create(
        model="gpt-5.6-terra",
        messages=[{
            "role": "user",
            "content": (
                f"Write a short paragraph (3-5 sentences) that would answer "
                f"this question. Write as if it were an excerpt "
                f"from technical documentation.\n\n"
                f"Question: {question}"
            )
        }],
        temperature=0.3
    )
    doc_hypothetique = response.choices[0].message.content

    # 2. Encode the hypothetical document
    hyde_emb = client.embeddings.create(
        input=doc_hypothetique,
        model="text-embedding-3-large"
    ).data[0].embedding

    # 3. Search for similar documents
    scores = corpus_embeddings @ np.array(hyde_emb)
    top_k = np.argsort(scores)[-k:][::-1]

    return [
        {**documents[i], "score": float(scores[i])}
        for i in top_k
    ]

If this technique works, it’s because the embedding of an answer resembles the embedding of a document more than the embedding of a question does. A question is short, interrogative, and often phrased in the user’s own words; a document is long, affirmative, and written in domain jargon. HyDE bridges this query-document gap by moving the query onto the corpus’s turf. That’s also why the prompt specifies “as if it were an excerpt from technical documentation”: we’re mimicking the target register.

Step-Back Prompting

The third technique handles the opposite case: questions so specific that no document addresses them head-on. “Can I deduct an electric bike purchased in February?” appears nowhere, but the section on deductible business expenses does. So we ask the model for the general question hiding behind the specific one, then search with both, weighting them 0.7 for the original question and 0.3 for the broader one—the wide context helps, but it shouldn’t dominate.

def recherche_step_back(
    question: str,
    corpus_embeddings: np.ndarray,
    documents: list[dict],
    k: int = 5
) -> list[dict]:
    """Two-pass search: general then specific."""
    # 1. Generate a step-back question
    response = client.chat.completions.create(
        model="gpt-5.6-terra",
        messages=[{
            "role": "user",
            "content": (
                f"What is the more general question or underlying concept "
                f"behind this specific question?\n\n"
                f"Specific question: {question}\n\n"
                f"Return only the general question."
            )
        }],
        temperature=0.3
    )
    question_generale = response.choices[0].message.content.strip()

    # 2. Search with both questions
    resultats_combines = {}

    for q, poids in [(question, 0.7), (question_generale, 0.3)]:
        q_emb = client.embeddings.create(
            input=q, model="text-embedding-3-large"
        ).data[0].embedding
        scores = corpus_embeddings @ np.array(q_emb)

        for i, s in enumerate(scores):
            if i not in resultats_combines:
                resultats_combines[i] = 0.0
            resultats_combines[i] += float(s) * poids

    top_k = sorted(
        resultats_combines.items(),
        key=lambda x: x[1], reverse=True
    )[:k]

    return [
        {**documents[idx], "score": score}
        for idx, score in top_k
    ]

Comparing the Techniques

Each addresses a specific retrieval pathology, and each comes at the cost of extra calls and therefore higher latency.

Combining Techniques

You don’t have to choose just one. The following function accepts a list of strategies, accumulates the scores each one produces in a dictionary indexed by document, and keeps the maximum. A document found by HyDE alone thus ranks alongside a document found by direct search. Keep in mind, though, that stacking three strategies triples the budget and lengthens response time: reserve this combination for difficult queries, or for offline processing where latency doesn’t matter.

def recherche_avancee(
    question: str,
    corpus_embeddings: np.ndarray,
    documents: list[dict],
    k: int = 5,
    strategies: list[str] = ["direct", "multi_query", "hyde"]
) -> list[dict]:
    """Combine multiple search strategies."""
    tous_resultats = {}

    if "direct" in strategies:
        q_emb = client.embeddings.create(
            input=question, model="text-embedding-3-large"
        ).data[0].embedding
        scores = corpus_embeddings @ np.array(q_emb)
        for i, s in enumerate(scores):
            tous_resultats.setdefault(i, []).append(float(s))

    if "hyde" in strategies:
        res_hyde = recherche_hyde(
            question, corpus_embeddings, documents, k=k*2
        )
        for r in res_hyde:
            tous_resultats.setdefault(r["id"], []).append(r["score"])

    # Final score = max of scores
    scores_finaux = {
        idx: max(scores) for idx, scores in tous_resultats.items()
    }

    top_k = sorted(
        scores_finaux.items(), key=lambda x: x[1], reverse=True
    )[:k]

    return [
        {**documents[idx], "score": score}
        for idx, score in top_k
    ]

Summary

• Multi-query rephrases the question from multiple angles to cover more vocabulary
• HyDE generates a hypothetical answer that more closely resembles target documents
• Step-back asks a more general question to capture broad context
• These techniques add latency but significantly improve recall