Next Steps: Document AI, Function Calling, and Multimodal Pipelines

Updated November 3, 2026 based on the Mistral Document QnA page and mistralai SDK 3.7.0: the example queries a document using a chat model (mistral-small-latest), not the OCR model, and document_image_limit is not a parameter for chat.complete (the call would have failed); page limits are set in client.ocr.process. Source: Document QnA.

Beyond the Vision API

You have now mastered the fundamentals of the Mistral Vision API: sending images via URL and Base64, analyzing charts, performing OCR, extracting structured data, and comparing images. This foundation has a natural boundary—until now, the model has always looked at an image and returned text, never taking action. This final lesson crosses that boundary in three stages: the specialized service for documents, function calling that turns analysis into action, and pipelines that chain them together.

Mistral Document AI (OCR 4)

For advanced OCR needs, Mistral offers Document AI, a service specialized in document processing. Where the Vision API handles individual images, Document AI reasons at the document scale: it handles multi-page PDFs and analyzes long files in a single pass. It manages complex layouts—columns, headers and footers, nested tables—and preserves structure, with output maintaining the document hierarchy of titles, paragraphs, and lists. Finally, its OCR accuracy is optimized for both printed and handwritten text.

Document AI Call Example

from mistralai.client import Mistral
import os

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])

# Analyze a PDF with Document AI
response = client.chat.complete(
    model="mistral-small-latest",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Extract the full content of this document while preserving the structure."
                },
                {
                    "type": "document_url",
                    "document_url": "https://example.com/annual-report.pdf"
                }
            ]
        }
    ]
)

print(response.choices[0].message.content)

The difference from everything you have written so far lies in the element type: document_url instead of image_url, read by a standard chat model (mistral-small-latest in Mistral’s documentation). The document must be no larger than 50 MB and contain at most 1,000 pages. To process only part of a 300-page annual report, use the OCR endpoint (client.ocr.process) instead, where the pages and image_limit parameters bound the work.

When to Use Document AI vs. Vision API

Remember the rule of thumb: use the Vision API for what was born as an image, and Document AI for what was born as a document.

Vision + Function Calling

Until now, your code received a response and decided what to do with it. Function calling reverses this relationship: you provide the model with a list of available actions, and it chooses which one to call based on the image. This is the key to building autonomous visual agents.

The cycle has four steps:

1. You send an image + function definitions
2. The model analyzes the image and decides which function to call
3. You execute the function and return the result
4. The model formulates a final response

Example: Invoice Processing Agent

Consider an accounting department that receives invoices by mail. Two outcomes are possible: the invoice is valid and must be recorded, or it has an anomaly and must be flagged. We therefore define two tools, with their parameters and required fields.

import json

# Define available tools
outils = [
    {
        "type": "function",
        "function": {
            "name": "enregistrer_facture",
            "description": "Enregistre une facture dans le système comptable",
            "parameters": {
                "type": "object",
                "properties": {
                    "emetteur": {"type": "string", "description": "Nom de l'émetteur"},
                    "numero": {"type": "string", "description": "Numéro de facture"},
                    "date": {"type": "string", "description": "Date d'émission (YYYY-MM-DD)"},
                    "montant_ttc": {"type": "number", "description": "Montant TTC en euros"},
                    "tva": {"type": "number", "description": "Montant de la TVA"}
                },
                "required": ["emetteur", "numero", "date", "montant_ttc"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "signaler_anomalie",
            "description": "Signale une anomalie sur une facture (doublons, montants incohérents)",
            "parameters": {
                "type": "object",
                "properties": {
                    "type_anomalie": {"type": "string"},
                    "description": {"type": "string"}
                },
                "required": ["type_anomalie", "description"]
            }
        }
    }
]

# Send the image with the tools
response = client.chat.complete(
    model="mistral-large-latest",
    messages=[
        {
            "role": "system",
            "content": "Vous êtes un assistant comptable. Analysez les factures et utilisez les outils appropriés."
        },
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Traitez cette facture."},
                {"type": "image_url", "image_url": encoder_image("factures/facture-2024-0042.jpg")}
            ]
        }
    ],
    tools=outils,
    tool_choice="auto"
)

# Handle the function call
if response.choices[0].message.tool_calls:
    appel = response.choices[0].message.tool_calls[0]
    nom_fonction = appel.function.name
    arguments = json.loads(appel.function.arguments)
    print(f"Fonction appelée : {nom_fonction}")
    print(f"Arguments : {json.dumps(arguments, indent=2, ensure_ascii=False)}")

Craft the description field for each function carefully: the model bases its choice on this, not on the name. A vague description produces an agent that calls the wrong function.

Multimodal Pipelines

A multimodal pipeline chains multiple processing steps by combining vision, text, and tools. Its value is both economic and organizational: each step employs strictly the model necessary.

Example: Mail Processing Pipeline

Incoming mail for an SME mixes invoices, quotes, contracts, and advertisements. Classifying them costs little; analyzing them in detail costs more. The pipeline therefore starts with a sort using a lightweight model, and only engages Large on documents that justify it.

def mail_pipeline(image_path):
    """Pipeline complet : classification -> extraction -> action."""

    # Étape 1 : Classification (modèle léger)
    classification = client.chat.complete(
        model="mistral-small-latest",
        response_format={"type": "json_object"},
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": 'Classifiez ce document : {"type": "facture|devis|contrat|courrier|autre", "urgence": "haute|moyenne|basse"}'},
                {"type": "image_url", "image_url": encoder_image(image_path)}
            ]
        }]
    )
    info = json.loads(classification.choices[0].message.content)
    print(f"Type : {info['type']} | Urgence : {info['urgence']}")

    # Étape 2 : Extraction détaillée (modèle puissant, seulement si nécessaire)
    if info["type"] == "facture":
        extraction = client.chat.complete(
            model="mistral-large-latest",
            response_format={"type": "json_object"},
            messages=[{
                "role": "user",
                "content": [
                    {"type": "text", "text": "Extrayez toutes les données de cette facture en JSON : emetteur, numero, date, lignes, total_ht, tva, total_ttc."},
                    {"type": "image_url", "image_url": encoder_image(image_path)}
                ]
            }]
        )
        data = json.loads(extraction.choices[0].message.content)
        return {"classification": info, "données": data}

    # Étape 3 : Résumé pour les autres types
    summary = client.chat.complete(
        model="mistral-small-latest",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "Résumez le contenu de ce document en 3 phrases maximum."},
                {"type": "image_url", "image_url": encoder_image(image_path)}
            ]
        }]
    )
    return {"classification": info, "summary": summary.choices[0].message.content}

# Traiter un lot de courrier
from pathlib import Path

for file in sorted(Path("courrier/").glob("*.jpg")):
    print(f"\n--- {file.name} ---")
    result = mail_pipeline(file)
    print(json.dumps(result, indent=2, ensure_ascii=False))

In a batch where one in five items is an invoice, four out of five documents never see the expensive model, for identical final quality.

Course Summary

Over these 12 lessons, you moved from mechanism to application. You first understood the multimodal architecture and what actually happens when an image enters the model, then learned to choose the variant suited to your workload—Large, Small, or Ministral. You then learned to send images via URL and Base64, singly or in batches. On this basis, you tackled the uses that make up the bulk of real work: analyzing charts, reading text via OCR, extracting structured data, comparing images for quality control and tracking evolution. Finally, you learned to optimize your calls for cost, latency, and reliability, and to integrate vision into larger pipelines thanks to function calling.

Next Steps

Four directions naturally extend this path. First, Document AI, whose OCR 4 handles documents the Vision API cannot support. Next, agents, combining vision and function calling to build systems that decide and act. Then fine-tuning, if your images have specifics no generalist model knows—an industrial parts catalog, an internal form. Finally, the Batch API, designed to process thousands of images at reduced cost when response time is not critical. In the longer term, advanced multimodal capabilities will open the way to video and audio when available.

Key Takeaways

• Document AI (OCR 4) is the natural complement to the Vision API for complex documents and PDFs
• Function calling turns vision into an autonomous agent capable of acting on extracted data
• A multimodal pipeline combines classification (Small) and extraction (Large) to optimize quality and cost
• The Mistral Vision API is a fundamental building block for building modern AI applications

Test Your Knowledge

Vision via the API: five questions to anchor the pipeline.

1. How do you send an image to the Mistral API?

Answer: Via URL or Base64-encoded, in a message with mixed text + image content—and multiple images can travel in the same message for comparison.

2. What does vision enable beyond description?

Answer: Chart analysis, OCR (transcribing text from images), and structured data extraction—the image becomes a data source, not an illustration.

3. How do you make structured extraction from an image reliable?

Answer: Specify the expected schema precisely (fields, format), request completeness or admission of absence, and verify critical values—vision is prompted like text.

4. What limits should you keep in mind?

Answer: Image quality conditions everything (resolution, sharpness); very small text and complex layouts tire the model—optimize the input before blaming the output.

5. What are the natural follow-ups to this course?

Answer: Document AI for complete documents, function calling to connect extraction to your systems, and multimodal pipelines that combine it all.

URL or Base64, precise prompts, careful inputs: the vision pipeline is ready—the next steps integrate it into your applications.