Semantic Similarity

Measuring Text Proximity

Once you transform your texts into vectors using mistral-embed, you can quantify how semantically close two texts are. This is the core of semantic search: comparing meanings, not just words. While a SQL LIKE '%invoice%' query returns a simple yes or no, a similarity metric gives you a number, enabling ranking. That ranking is what makes the whole system valuable.

In this lesson, you will implement the main similarity metrics and see how they work in practice with concrete Python examples.

Euclidean Distance

Euclidean distance measures the length of the segment between two points in vector space. The smaller the distance, the more similar the texts. The example below compares three sentences: the first two say the same thing using different vocabulary, while the third discusses apple pie. Sentences A and B share no common words, yet their distance will be significantly smaller than the distance between A and C.

import numpy as np
from mistralai.client import Mistral

client = Mistral(api_key=api_key)

def get_embedding(text):
    """Get the embedding for a text via mistral-embed."""
    response = client.embeddings.create(
        model="mistral-embed",
        inputs=[text],
    )
    return np.array(response.data[0].embedding)

# Three sentences to compare
phrase_a = "Machine learning transforms data into predictions."
phrase_b = "Automatic learning converts data into results."
phrase_c = "The recipe for apple pie is simple."

emb_a = get_embedding(phrase_a)
emb_b = get_embedding(phrase_b)
emb_c = get_embedding(phrase_c)

# Euclidean distances
dist_ab = np.linalg.norm(emb_a - emb_b)
dist_ac = np.linalg.norm(emb_a - emb_c)

print(f"Distance A-B (semantically close): {dist_ab:.4f}")
print(f"Distance A-C (semantically distant): {dist_ac:.4f}")
# A-B will be significantly lower than A-C

Cosine Similarity

Cosine similarity is the most widely used metric for embeddings. It measures the angle between two vectors, independent of their magnitude. The result ranges from -1 to 1, providing an immediately readable scale: you can set a threshold and communicate it to a business team without having to explain vector norms.

from sklearn.metrics.pairwise import cosine_similarity

# With scikit-learn
sim_ab = cosine_similarity([emb_a], [emb_b])[0][0]
sim_ac = cosine_similarity([emb_a], [emb_c])[0][0]

print(f"Cosine similarity A-B: {sim_ab:.4f}")  # ~0.92 (very close)
print(f"Cosine similarity A-C: {sim_ac:.4f}")  # ~0.35 (distant)

The gap between 0.92 and 0.35 is the useful information: it’s not the absolute value that matters, but the margin separating relevant results from off-topic ones in your specific dataset. You can also implement the metric manually, if only to verify what scikit-learn is doing or to avoid dependencies in a constrained environment:

def cosine_sim(vec_a, vec_b):
    """Manual cosine similarity."""
    dot = np.dot(vec_a, vec_b)
    norm_a = np.linalg.norm(vec_a)
    norm_b = np.linalg.norm(vec_b)
    return dot / (norm_a * norm_b)

print(f"Manual verification A-B: {cosine_sim(emb_a, emb_b):.4f}")

Similarity Matrix

To compare multiple documents against each other, you can build a similarity matrix. The corpus below intentionally mixes two themes—AI and gastronomy—so the structure jumps out at you: documents 0, 1, and 3 relate to each other, while documents 2 and 4 form a separate cluster. This is exactly the signal that clustering algorithms exploit.

from sklearn.metrics.pairwise import cosine_similarity

documents = [
    "Mistral AI develops language models.",
    "LLMs generate text from prompts.",
    "French cuisine is renowned worldwide.",
    "NLP is a branch of artificial intelligence.",
    "Croissants taste best in the morning.",
]

# Encode all documents
response = client.embeddings.create(
    model="mistral-embed",
    inputs=documents,
)
embeddings = np.array([d.embedding for d in response.data])

# Similarity matrix (5x5)
sim_matrix = cosine_similarity(embeddings)

print("Similarity Matrix:")
for i, doc in enumerate(documents):
    for j, doc2 in enumerate(documents):
        if j > i:
            print(f"  [{i}]-[{j}]: {sim_matrix[i][j]:.3f}")

Simple Semantic Search

All of the above combines into a mini search engine. The query is encoded with the same model as the corpus—a non-negotiable condition, since two different models produce incomparable spaces—then argsort reverses the ranking to bring the highest scores to the top.

def recherche_semantique(query, documents, embeddings, top_k=3):
    """Finds the most relevant documents for a query."""
    # Encode the query
    query_response = client.embeddings.create(
        model="mistral-embed",
        inputs=[query],
    )
    query_embedding = np.array(query_response.data[0].embedding)

    # Calculate similarities
    similarities = cosine_similarity(
        [query_embedding], embeddings
    )[0]

    # Sort by descending similarity
    ranked_indices = np.argsort(similarities)[::-1][:top_k]

    results = []
    for idx in ranked_indices:
        results.append({
            "document": documents[idx],
            "score": similarities[idx],
            "index": int(idx),
        })
    return results

# Usage example
query = "How does Mistral's AI work?"
results = recherche_semantique(query, documents, embeddings)

for r in results:
    print(f"  Score: {r['score']:.3f} -- {r['document']}")

Relevance Threshold

This engine has a flaw you will quickly discover in production: it always returns three results, even when the question has nothing to do with the corpus. Ask it about the weather, and it will return the three least distant documents, with derisory scores but a seemingly normal ranking. In practice, you must therefore define a similarity threshold below which a document is considered irrelevant:

SEUIL_PERTINENCE = 0.7

def recherche_avec_seuil(query, documents, embeddings, seuil=SEUIL_PERTINENCE):
    """Search with filtering by relevance threshold."""
    results = recherche_semantique(query, documents, embeddings, top_k=len(documents))
    return [r for r in results if r["score"] >= seuil]

# A threshold of 0.7 is a good starting point
# Adjust based on your use case:
# - 0.8+: very strict (few results, high relevance)
# - 0.6-0.7: balanced
# - 0.5: permissive (many results, variable relevance)

Take the time to calibrate this threshold on about thirty real queries before opening the service to users. Set it too high, and your engine says “I don’t know” to legitimate questions. Set it too low, and you feed the LLM off-topic context, creating the very hallucinations RAG was supposed to prevent.

Key Takeaways

• Cosine similarity is the standard metric for comparing embeddings
• Use sklearn.metrics.pairwise.cosine_similarity for fast calculations
• A similarity matrix allows you to compare all documents against each other
• Define a relevance threshold adapted to your use case (0.7 is a good default)
• Semantic search finds results by meaning, not by exact word matching