Batching and Chunking

The Challenge of Large Volumes

When you work with thousands of documents, you cannot encode everything in a single API call. You must manage batching (processing in groups) and chunking (splitting text). These two techniques are essential for building semantic search systems at scale.

The two concepts sound similar but solve opposite problems. Chunking handles a single document that is too large to be encoded effectively as a whole; batching handles too many documents to fit in a single request. A three-hundred-page technical manual falls into the first category, an export of twenty thousand tickets into the second, and a real-world project almost always involves both.

Batching: Processing in Groups

The mistral-embed API accepts a list of inputs, but practical limits exist (request size, token count). The solution is to split your corpus into batches and loop through them. The function below does exactly that, displaying progress—a critical detail when encoding a full corpus takes several minutes and you need to know whether the process is advancing or stuck.

import numpy as np
from mistralai.client import Mistral

client = Mistral(api_key=api_key)

def get_embeddings_by_chunks(data, chunk_size=25):
    """Encode a large list of texts in batches."""
    chunks = [data[x:x + chunk_size] for x in range(0, len(data), chunk_size)]
    embeddings = []

    for i, chunk in enumerate(chunks):
        print(f"Batch {i+1}/{len(chunks)} ({len(chunk)} texts)...")
        response = client.embeddings.create(
            model="mistral-embed",
            inputs=chunk,
        )
        embeddings.extend([d.embedding for d in response.data])

    return embeddings

# Example with 1000 documents
documents = [f"Document number {i} with varied content." for i in range(1000)]
all_embeddings = get_embeddings_by_chunks(documents, chunk_size=50)

print(f"Total: {len(all_embeddings)} embeddings of {len(all_embeddings[0])} dimensions")

Using extend instead of append ensures the vector order matches the input text order exactly. This allows you to retrieve the original document from a simple index later on.

Optimizing Batch Size

The optimal batch size depends on the average length of your texts: fifty article titles and fifty full reports do not weigh the same in tokens. Rather than picking a constant at random, estimate it from the corpus itself.

def optimal_batch_size(documents, max_tokens_per_batch=16000):
    """Estimate the optimal batch size."""
    # Rough estimate: 1 token ~ 4 characters in English
    avg_chars = sum(len(d) for d in documents) / len(documents)
    avg_tokens = avg_chars / 4

    batch_size = max(1, int(max_tokens_per_batch / avg_tokens))
    return min(batch_size, 100)  # Cap at 100

# Adjust based on your corpus
batch_size = optimal_batch_size(documents)
print(f"Recommended batch size: {batch_size}")

The cap at 100 is not cosmetic: beyond that, a single failed batch causes significant work loss, and the memory consumed by the response becomes substantial.

Chunking: Splitting Text

Long texts must be split into chunks before encoding. Several strategies exist, each with its advantages, and the choice directly impacts response quality: a chunk that starts in the middle of a sentence produces a fuzzy vector, leading to fuzzy search results.

Character-Based Splitting (Simple)

The most direct method cuts every N characters, regardless of content. Its merit is predictability and speed, making it acceptable for a prototype or for text without clear structure.

def chunk_by_chars(text, chunk_size=2048, overlap=200):
    """Split text into chunks with overlap."""
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk)
        start = end - overlap  # Overlap for context
    return chunks

long_text = "A very long document..." * 500
chunks = chunk_by_chars(long_text, chunk_size=2048, overlap=200)
print(f"{len(chunks)} chunks created")

Paragraph-Based Splitting (Recommended)

Paragraph-based splitting respects the unit of meaning intended by the author. In product documentation, each paragraph generally covers a specific point; keeping it intact yields a chunk that can answer a question on its own. The function accumulates paragraphs until the maximum size is exceeded, then closes the current chunk.

def chunk_by_paragraphs(text, max_chunk_size=2048):
    """Split while respecting paragraph boundaries."""
    paragraphs = text.split("\n\n")
    chunks = []
    current_chunk = ""

    for para in paragraphs:
        if len(current_chunk) + len(para) + 2 > max_chunk_size:
            if current_chunk:
                chunks.append(current_chunk.strip())
            current_chunk = para
        else:
            current_chunk += "\n\n" + para if current_chunk else para

    if current_chunk:
        chunks.append(current_chunk.strip())

    return chunks

Sentence-Based Splitting

When text lacks usable paragraphs—a call transcript, a running log—the sentence becomes the smallest available unit of meaning. Splitting then follows the same accumulation logic, relying on end-of-sentence punctuation.

import re

def chunk_by_sentences(text, max_chunk_size=1500):
    """Split while respecting sentence boundaries."""
    # Split on end-of-sentence punctuation
    sentence_pattern = r'(?<=[.!?])\s+'
    sentences = re.split(sentence_pattern, text)
    chunks = []
    current_chunk = ""

    for sentence in sentences:
        if len(current_chunk) + len(sentence) + 1 > max_chunk_size:
            if current_chunk:
                chunks.append(current_chunk.strip())
            current_chunk = sentence
        else:
            current_chunk += " " + sentence if current_chunk else sentence

    if current_chunk:
        chunks.append(current_chunk.strip())

    return chunks

Overlap

Overlap between chunks prevents cutting an idea in half. By adding an overlap of 10% to 20% of the chunk size, you preserve context between adjacent pieces. Imagine a procedure where step 4 ends at character 2040 and the consequence is explained immediately after: without overlap, no chunk contains both halves of the reasoning, and search will never retrieve the complete answer.

The values below serve as proven starting points depending on content type:

# Recommended parameters by content type
PARAMS = {
    "articles":     {"chunk_size": 1500, "overlap": 200},
    "documentation": {"chunk_size": 1000, "overlap": 150},
    "conversations": {"chunk_size": 500,  "overlap": 50},
    "code":          {"chunk_size": 2000, "overlap": 300},
}

Conversations drop to 500 characters because a turn of speech is short, and a larger chunk would mix multiple topics; code rises to 2000 with generous overlap because a function loses meaning if its signature is separated from its body.

Complete Pipeline: Chunk + Batch + Embed

Here is how to combine chunking and batching in a production pipeline. The element you must not neglect is chunk_metadata: without it, you get orphan vectors unable to indicate which document they come from, and you will never be able to cite your sources in the final response.

def pipeline_embed(documents, chunk_size=1500, overlap=200, batch_size=50):
    """Complete pipeline: chunk, batch, embed."""
    # 1. Chunk all documents
    all_chunks = []
    chunk_metadata = []  # Track source document

    for doc_idx, doc in enumerate(documents):
        chunks = chunk_by_paragraphs(doc, max_chunk_size=chunk_size)
        for chunk_idx, chunk in enumerate(chunks):
            all_chunks.append(chunk)
            chunk_metadata.append({
                "doc_index": doc_idx,
                "chunk_index": chunk_idx,
            })

    print(f"{len(documents)} documents -> {len(all_chunks)} chunks")

    # 2. Encode in batches
    embeddings = get_embeddings_by_chunks(all_chunks, chunk_size=batch_size)

    # 3. Return structured results
    return {
        "chunks": all_chunks,
        "embeddings": np.array(embeddings),
        "metadata": chunk_metadata,
    }

# Usage
result = pipeline_embed(documents)
print(f"Embeddings: {result['embeddings'].shape}")

Run this pipeline on a dozen of your actual documents and inspect all_chunks by eye before encoding anything at scale. A bad split is immediately visible when reading, whereas it never shows up in the matrix shape.

Key Takeaways

• Batching lets you encode thousands of documents without exceeding API limits
• Chunking splits long texts for better semantic granularity
• A 10–20% overlap between chunks preserves context
• Paragraph or sentence splitting is preferable to character splitting
• Keep metadata (source document, position) for each chunk