Multimodal Architecture of Mistral Models

How does a model process text and images?

When you send an image to a Mistral model, it doesn’t “see” it the way a human does. Processing relies on a two-stage architecture: first, a visual encoder transforms the image into a sequence of numerical tokens; then, the language model processes these visual tokens alongside the text tokens from your prompt.

This lesson might seem theoretical, but it’s actually the most cost-effective part of the course. Understanding this mechanism explains why a 4K image doesn’t yield better results than a 1080p image, why your monthly bill spikes faster with five images than with five paragraphs, and why tiny text on a screenshot often gets transcribed poorly.

The Visual Encoder (Vision Encoder)

The visual encoder is a specialized neural network—typically a Vision Transformer (ViT)—that slices the image into small square patches, for example 14×14 pixels, and converts them into numerical embeddings. The pipeline runs in five steps:

1. Resizing: the image is scaled to a standard resolution (e.g., 384×384 or 768×768 pixels)
2. Patch extraction: the resized image is divided into regular patches
3. Projection: each patch is converted into a numerical vector (embedding)
4. Positional encoding: position information for each patch is added
5. Attention: the visual transformer processes the complete sequence

The result is a sequence of visual tokens, fully comparable to text tokens, except they represent regions of the image rather than word fragments. The first step deserves your attention: it explains why a 4,000-pixel-wide invoice scan gets reduced to a few hundred pixels before analysis even begins. Characters in fine print, already at the limit of legibility, become a few blurry pixels.

Text-Image Fusion in the LLM

Once the visual tokens are produced, they are inserted into the language model’s input sequence alongside the text tokens. The model processes everything uniformly thanks to the cross-attention mechanism.

[text tokens: "Describe this image"] + [visual tokens: patch1, patch2, ..., patchN]
                                    ↓
                          Language Model (LLM)
                                    ↓
                    [output tokens: "This image shows..."]

The model can thus “read” the image and text simultaneously, allowing it to answer specific questions about visual content. This also explains why a question asked before the image and one asked after don’t yield exactly the same result: everything sits in the same sequence, and order matters.

Impact on Token Counting

Each image consumes a certain number of tokens, depending on its resolution after resizing. This is the key takeaway for estimating your costs. A standard image, around 768×768, consumes between 800 and 1,500 tokens—equivalent to one or two pages of text. Larger images are resized before processing: sending a 4K image costs no more than a 1080p image, but it gains you nothing either. To this total, add the tokens from your prompt and the generated response.

# Example: estimate the cost of a vision call
tokens_image = 1000  # average estimate
tokens_prompt = 50   # "Describe this image in detail"
tokens_response = 200  # estimated response

total_tokens = tokens_image + tokens_prompt + tokens_response
print(f"Estimated total tokens: {total_tokens}")
# → Estimated total tokens: 1250

Run this calculation on your own volume before industrializing: a thousand images per day represents at least a million input tokens, shifting the conversation from prototype to budget.

Size and Resolution Limits

Mistral accepts JPEG, PNG, GIF (first frame), and WebP formats, with a maximum size of about 20 MB per image. Multiple images can coexist in a single message, a topic we’ll cover in Lesson 6. The most counterintuitive constraint remains internal resizing: very high resolution doesn’t improve results; it just lengthens transfer time.

The practical consequence is that you should resize images yourself upstream rather than letting the API do it after an unnecessarily heavy upload:

from PIL import Image

def prepare_image(path, max_size=1024):
    """Resizes an image before sending to optimize latency."""
    img = Image.open(path)
    img.thumbnail((max_size, max_size))
    img.save(path, quality=85)
    return path

Insert this function just before Base64 encoding or URL hosting. On a batch of 4 MB smartphone photos, it cuts transferred volume by a significant factor without significant quality loss for analysis.

Simplified Architecture

End-to-end, a request journey boils down to five steps: you send a message containing text and one or more images; the API resizes each image and sends it to the visual encoder; the resulting visual tokens merge with text tokens; the model generates a natural language response; you receive it just like a standard text call. Nothing changes in your API-side code—Mistral handles all the complexity—but that doesn’t mean you can ignore what it costs you.

Key Takeaways

• Images are converted into visual tokens by a Vision Transformer encoder
• The LLM processes these visual tokens just like text tokens
• A standard image consumes roughly 800 to 1,500 tokens
• Images are resized internally—no need to send very high resolution
• Resizing images on the client side reduces latency without quality loss