Vector Store with FAISS — Embeddings, RAG, and Semantic Search

FAISS: High-Performance Vector Search

You have calculated and saved your embeddings, but a NumPy matrix is not a search engine: comparing a query to 100,000 vectors using a Python loop would take several seconds per question. FAISS (Facebook AI Similarity Search) solves exactly this problem. This open-source library from Meta is designed for similarity search across large vector sets; it is fast, memory-efficient, and works just as well with a few hundred vectors as with millions. Installation takes a single command. faiss-cpu is sufficient in the vast majority of cases; faiss-gpu is only justified for volumes where computation becomes the bottleneck.

# pip install faiss-cpu
# or faiss-gpu for GPU acceleration

import faiss
import numpy as np

Create Your First Index

The simplest index is IndexFlatL2, which performs an exhaustive search using L2 distance (Euclidean). “Exhaustive” means it actually compares the query to every stored vector: no approximation, so no missed results. You initialize it with the vector dimension—1024 for mistral-embed, a value you read directly from the matrix shape rather than hard-coding. Converting to float32 is not optional: FAISS rejects the float64 that NumPy produces by default, and forgetting this detail is the first error every beginner encounters.

# Assume our embeddings are a NumPy matrix
# embeddings.shape = (n_chunks, 1024)

# Create the index
dimension = embeddings.shape[1]  # 1024 for mistral-embed
index = faiss.IndexFlatL2(dimension)

# Add the vectors
index.add(embeddings.astype(np.float32))  # FAISS requires float32

print(f"Index created: {index.ntotal} vectors of dimension {dimension}")

Choosing the Right Index Type

FAISS does not offer a single index but a family of them, and the choice involves a trade-off between accuracy and speed. The first two, IndexFlatL2 and IndexFlatIP, are exact: they differ only in the metric used, Euclidean distance for one, dot product for the other. IndexIVFFlat changes the logic by partitioning the space into clusters and searching only the most promising ones, which significantly speeds up search at the cost of a slight loss in recall—and requires prior training on your data. Finally, IndexHNSWFlat builds a neighbor graph and proves very fast during queries, with the parameter 32 setting the number of neighbors retained per node here.

dimension = 1024

# 1. IndexFlatL2: Exact search, L2 distance
# Most accurate, but slowest for large volumes
index_flat_l2 = faiss.IndexFlatL2(dimension)

# 2. IndexFlatIP: Exact search, dot product (inner product)
# Use with normalized vectors for cosine similarity
index_flat_ip = faiss.IndexFlatIP(dimension)

# 3. IndexIVFFlat: Approximate search with partitioning
# Faster for large volumes, slight loss of accuracy
nlist = 100  # Number of partitions (clusters)
quantizer = faiss.IndexFlatL2(dimension)
index_ivf = faiss.IndexIVFFlat(quantizer, dimension, nlist)
# Requires training before use:
# index_ivf.train(embeddings)
# index_ivf.add(embeddings)

# 4. IndexHNSWFlat: Neighbor graph (very fast search)
index_hnsw = faiss.IndexHNSWFlat(dimension, 32)  # 32 = number of neighbors

Querying the Index

Search happens in three steps. First, encode the user’s question with the same model as the corpus—a query encoded by a different model would yield meaningless distances. FAISS then returns two parallel arrays: the distances and the indices of the k nearest neighbors. These indices are positions in your list of chunks: this is where the discipline from the previous lesson pays off, since the alignment between the matrix and the metadata is what allows you to trace back to the text and its source. The if idx >= 0 test covers the case where the index contains fewer than k vectors, in which case FAISS fills the missing spots with -1.

from mistralai.client import Mistral

client = Mistral(api_key=api_key)

def search_index(query, index, chunks, k=5):
    """Search for chunks closest to a query."""
    # 1. Encode the query
    response = client.embeddings.create(
        model="mistral-embed",
        inputs=[query],
    )
    query_vector = np.array([response.data[0].embedding], dtype=np.float32)

    # 2. Search in FAISS
    distances, indices = index.search(query_vector, k)

    # 3. Format results
    results = []
    for i, (dist, idx) in enumerate(zip(distances[0], indices[0])):
        if idx >= 0:  # FAISS returns -1 if not enough results
            results.append({
                "rank": i + 1,
                "distance": float(dist),
                "text": chunks[idx]["text"],
                "source": chunks[idx].get("source", ""),
            })

    return results

# Example search
results = search_index("How does RAG work?", index, chunks)
for r in results:
    print(f"  #{r['rank']} (dist: {r['distance']:.4f}) [{r['source']}]")
    print(f"    {r['text'][:100]}...")

Switching to Cosine Similarity

FAISS does not offer an explicit “cosine” index, but algebra provides a free equivalent: for vectors with norm 1, the dot product is the cosine. You simply divide each vector by its norm, then use IndexFlatIP. This change is far from theoretical: L2 distance is sensitive to vector length, so a very long chunk can end up artificially far from a short query even if their topics match. Cosine measures only orientation, hence meaning, and that is generally what you want to compare.

def create_cosine_index(embeddings):
    """Create a FAISS index with cosine similarity."""
    # Normalize vectors (L2 norm = 1)
    norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
    normalized = (embeddings / norms).astype(np.float32)

    # Use IndexFlatIP (dot product = cosine for normalized vectors)
    dimension = normalized.shape[1]
    index = faiss.IndexFlatIP(dimension)
    index.add(normalized)

    return index

cosine_index = create_cosine_index(embeddings)

As with embeddings, the index persists to disk. faiss.write_index produces a binary file that faiss.read_index reloads as-is, including internal structures—which matters most for trained indexes, where reconstruction would otherwise need to be redone.

# Save
faiss.write_index(index, "rag_faiss.index")
print("FAISS index saved")

# Load
index_loaded = faiss.read_index("rag_faiss.index")
print(f"Index loaded: {index_loaded.ntotal} vectors")

When the Corpus Becomes Massive

Starting at several hundred thousand vectors, exhaustive search becomes too slow and the IVF (Inverted File) index becomes necessary. Its lifecycle includes a step that flat indexes do not have: training, which analyzes your data distribution to place the nlist partition centroids. Next comes adding the vectors, then tuning nprobe, the knob that decides how many partitions will be explored per query. Increasing nprobe improves recall and slows down search; this is the parameter you will adjust by measuring, on a representative set of questions, the point at which results stop improving.

def create_ivf_index(embeddings, nlist=100, nprobe=10):
    """Create an IVF index for large volumes."""
    dimension = embeddings.shape[1]
    data = embeddings.astype(np.float32)

    # Create the quantizer and IVF index
    quantizer = faiss.IndexFlatL2(dimension)
    index = faiss.IndexIVFFlat(quantizer, dimension, nlist)

    # Train the index (required for IVF)
    print("Training index...")
    index.train(data)

    # Add the vectors
    index.add(data)

    # Number of partitions to explore during search
    index.nprobe = nprobe  # Higher = more accurate, slower

    print(f"IVF index created: {index.ntotal} vectors, {nlist} partitions")
    return index

# For 100K+ vectors
# large_index = create_ivf_index(embeddings, nlist=256, nprobe=20)

Key Takeaways

FAISS is the standard for fast vector search
IndexFlatL2 for small corpora (under 100K vectors), exact search
IndexIVFFlat for large corpora, approximate but fast search
Normalize vectors and use IndexFlatIP for cosine similarity
Save the index to disk with faiss.write_index to avoid rebuilding