Hybrid search (semantic + lexical)

Objectives

• Understand why combining semantic and lexical search matters
• Implement hybrid search using Reciprocal Rank Fusion
• Use hybrid search in vector databases

The problem: no single approach is perfect

Semantic search excels at handling synonyms but can miss precise technical terms; lexical search does the exact opposite. Consider a system administrator’s query, “nginx error 503,” and observe how both engines behave against a technical documentation base.

# Query: "nginx error 503"
# Semantic → finds "web server issues" (good) but misses "code 503" (missed)
# Lexical   → finds "nginx error 503" exactly (good) but misses "reverse proxy timeout" (missed)
# Hybrid    → finds both ✅

Semantic search understands this refers to a server incident and surfaces general articles, but the number 503 escapes it: for an embedding model, 503 and 504 are just two three-digit numbers, semantically close even though they denote different outages. Lexical search retrieves the exact code but misses the article on reverse proxy timeouts, which often describes the root cause. Neither method alone returns the right results page; their union does.

Lexical search with BM25

BM25 is the standard algorithm for keyword search. It weights each term based on its rarity in the corpus and its frequency in the document, naturally surfacing discriminative terms like “503” or “nginx.” However, you must feed it clean tokens: the tokenization function lowercases text, extracts words, and removes French stop words, which add no relevance signal.

from rank_bm25 import BM25Okapi
import re

def tokenize(text: str) -> list[str]:
    """Simple tokenization for French."""
    text = text.lower()
    tokens = re.findall(r"\b\w+\b", text)
    # Remove common stop words
    stop_words = {"le", "la", "les", "de", "du", "des", "un", "une",
                  "et", "ou", "est", "dans", "pour", "avec", "sur"}
    return [t for t in tokens if t not in stop_words]

documents = [
    "Comment résoudre l'erreur 503 dans nginx",
    "Configuration du reverse proxy avec timeout",
    "Guide d'installation de nginx sur Ubuntu",
    "Les bases de la cuisine française",
]

# Index with BM25
tokenized_corpus = [tokenize(doc) for doc in documents]
bm25 = BM25Okapi(tokenized_corpus)

# Search
query_tokens = tokenize("erreur 503 nginx")
bm25_scores = bm25.get_scores(query_tokens)

for i, score in enumerate(bm25_scores):
    print(f"BM25 {score:.3f} | {documents[i]}")

Reciprocal Rank Fusion (RRF)

Now you need to combine the two rankings, which is trickier than it looks: a cosine score lives between 0 and 1, while a BM25 score has no upper bound. Adding them directly would let BM25 overwhelm the semantic signal. RRF sidesteps this by ignoring raw scores and keeping only ranks: each document receives 1 / (k + rank) in each list, and you sum these contributions. A document that ranks well everywhere thus beats a document that excels in only one list.

def reciprocal_rank_fusion(
    result_lists: list[list[tuple[int, float]]],
    k: int = 60
) -> list[tuple[int, float]]:
    """Fuses multiple result lists using RRF.

    Args:
        result_lists: list of lists of (doc_id, score)
        k: RRF constant (60 is standard)
    """
    fused_scores: dict[int, float] = {}

    for results in result_lists:
        for rank, (doc_id, _) in enumerate(results):
            if doc_id not in fused_scores:
                fused_scores[doc_id] = 0.0
            fused_scores[doc_id] += 1.0 / (k + rank + 1)

    # Sort by score descending
    return sorted(
        fused_scores.items(),
        key=lambda x: x[1],
        reverse=True
    )

The constant k, conventionally set to 60 based on literature, dampens the gap between top positions: without it, the first rank would weigh twice as much as the second, making the fusion too sensitive to minor ranking differences.

Complete hybrid pipeline

The class below maintains both indexes in parallel over the same corpus, runs both searches on every query, and fuses the rankings. It also returns individual scores—a valuable detail during tuning, since comparing them shows which engine carried a given result.

from openai import OpenAI
from rank_bm25 import BM25Okapi
import numpy as np

class HybridSearch:
    def __init__(self, model: str = "text-embedding-3-large"):
        self.client = OpenAI()
        self.model = model
        self.documents: list[str] = []
        self.embeddings: np.ndarray | None = None
        self.bm25: BM25Okapi | None = None

    def index(self, documents: list[str]):
        self.documents = documents

        # Semantic index
        response = self.client.embeddings.create(
            input=documents, model=self.model
        )
        vectors = sorted(response.data, key=lambda x: x.index)
        self.embeddings = np.array(
            [v.embedding for v in vectors], dtype=np.float32
        )

        # BM25 index
        tokenized_corpus = [tokenize(d) for d in documents]
        self.bm25 = BM25Okapi(tokenized_corpus)

    def search(
        self, query: str, k: int = 5,
        semantic_weight: float = 0.7,
        lexical_weight: float = 0.3
    ) -> list[dict]:
        # Semantic search
        query_emb = self.client.embeddings.create(
            input=query, model=self.model
        ).data[0].embedding
        sem_scores = self.embeddings @ np.array(query_emb)
        sem_ranking = [
            (i, float(sem_scores[i]))
            for i in np.argsort(sem_scores)[::-1]
        ]

        # Lexical search
        query_tokens = tokenize(query)
        lex_scores = self.bm25.get_scores(query_tokens)
        lex_ranking = [
            (i, float(lex_scores[i]))
            for i in np.argsort(lex_scores)[::-1]
        ]

        # RRF Fusion
        fused = reciprocal_rank_fusion(
            [sem_ranking, lex_ranking]
        )

        return [
            {
                "text": self.documents[doc_id],
                "rrf_score": score,
                "sem_score": float(sem_scores[doc_id]),
                "lex_score": float(lex_scores[doc_id]),
            }
            for doc_id, score in fused[:k]
        ]

# Usage
engine = HybridSearch()
engine.index([
    "Comment résoudre l'erreur 503 dans nginx",
    "Timeout et configuration du reverse proxy",
    "Guide d'installation nginx sur Ubuntu",
    "Diagnostiquer les problèmes de serveur web",
])

results = engine.search("erreur 503 nginx")
for r in results:
    print(f"RRF: {r['rrf_score']:.4f} | "
          f"Sem: {r['sem_score']:.3f} | "
          f"Lex: {r['lex_score']:.3f} | "
          f"{r['text']}")

On this four-document corpus, the article “nginx error 503” dominates the lexical ranking, while “web server issues” dominates the semantic one; fusion brings both into the top three, something neither method would achieve alone.

Native hybrid search in Weaviate

If you already use Weaviate, you don’t need to write all this: the database integrates fusion and exposes it in a single query, with an alpha parameter that balances the two engines—0 for pure lexical, 1 for pure semantic.

import weaviate

wclient = weaviate.connect_to_local()
collection = wclient.collections.get("Document")

# Native hybrid search
results = collection.query.hybrid(
    query="erreur 503 nginx",
    alpha=0.7,  # 0 = pure lexical, 1 = pure semantic
    limit=5
)

for obj in results.objects:
    print(f"[{obj.metadata.score:.3f}] {obj.properties['texte']}")

An alpha of 0.7 is a reasonable starting point, but it isn’t universal: a corpus rich in technical references and error codes benefits from dropping toward 0.5, while a knowledge base written in plain language can handle 0.8. Only tune this slider with an evaluation set in front of you—that’s precisely the subject of the next lesson.

Summary

• Hybrid search combines semantic (meaning) and lexical (exact words)
• Reciprocal Rank Fusion (RRF) merges rankings effectively
• The alpha parameter controls the balance between the two approaches
• Weaviate offers hybrid search natively