The Annotations API — Document AI: OCR and Document Intelligence

Updated November 2, 2026 — OCR. OCR 4.0 (mistral-ocr-4-0) was retired on September 30, 2026. Use OCR 4.1 (mistral-ocr-4-1), which has been generally available since August 31 at the same price, or the alias mistral-ocr-latest, which has pointed to it since July 16. OCR 4.1 adds confidence scores at the block level. Addendum from November 3: The previous example read page.document_annotation on each page; in mistralai SDK 3.7.0, this field belongs to the OCR response (ocr_response.document_annotation), while pages contain only their content and images; image annotations are read from image.image_annotation.

Beyond Text Extraction

OCR gives you raw text: faithful and complete, but undifferentiated. You get Markdown, not data. Annotations take the next step: they extract structured data from your documents based on a schema you define in advance. You describe what you’re looking for, the model extracts it automatically, and your application receives typed fields instead of a wall of text to parse with regular expressions.

Two Levels of Analysis

Document AI distinguishes between two types of annotations based on the scale at which they operate.

Bounding box annotations (bbox_annotation) focus on the images and graphics that OCR has isolated. The vision model processes each bounding box individually to describe its content. In an annual report, this mechanism lets you describe charts and diagrams, classify images (photo, logo, schematic, graph), and extract textual content from embedded images—three pieces of information that standard text extraction systematically overlooks.

Document annotations (document_annotation) treat the file as a whole. The model analyzes the extracted Markdown along with the first eight images, then returns what you asked for: structured fields like a name, date, or amount; the document type; a content summary; or an answer to a specific question. This is the tool for bulk processing, the one you’ll plug into your invoice, form, and contract workflows.

The Pydantic Schema Acts as the Contract

Annotations rely on Pydantic models to define the output structure. Each attribute becomes a field to extract, and its description serves as an instruction to the model: it carries as much weight as the field name itself.

from pydantic import BaseModel, Field
from typing import Optional

class InvoiceExtraction(BaseModel):
    vendor: str = Field(..., description="Vendor name")
    invoice_number: str = Field(..., description="Invoice number")
    issue_date: str = Field(..., description="Issue date (YYYY-MM-DD)")
    amount_excl_tax: float = Field(..., description="Amount excluding tax")
    amount_incl_tax: float = Field(..., description="Amount including tax")
    currency: str = Field(default="EUR", description="Currency (ISO code)")

Annotating an Entire Document

The call remains the same as for OCR: you simply add the document_annotation_format parameter, populated by response_format_from_pydantic_model(), which translates your model into a schema the API can use. On return, the annotation arrives as a JSON string, which you revalidate with Pydantic to get a safe Python object.

import os
from mistralai.client import Mistral
from mistralai.extra import response_format_from_pydantic_model
from pydantic import BaseModel, Field

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])

class InvoiceExtraction(BaseModel):
    vendor: str = Field(..., description="Vendor name")
    invoice_number: str = Field(..., description="Invoice number")
    issue_date: str = Field(..., description="Issue date (YYYY-MM-DD)")
    amount_excl_tax: float = Field(..., description="Amount excluding tax")
    amount_incl_tax: float = Field(..., description="Amount including tax")

ocr_response = client.ocr.process(
    model="mistral-ocr-latest",
    document={
        "type": "document_url",
        "document_url": "https://example.com/invoice.pdf"
    },
    document_annotation_format=response_format_from_pydantic_model(
        InvoiceExtraction
    )
)

# The document annotation is carried by the response, not by each page
if ocr_response.document_annotation:
    invoice = InvoiceExtraction.model_validate_json(
        ocr_response.document_annotation
    )
    print(f"Vendor: {invoice.vendor}")
    print(f"Amount incl. tax: {invoice.amount_incl_tax}")

Describing Visuals One by One

For images, the principle is identical but the parameter changes, and results are read from page.images rather than the page itself. Note include_image_base64=True: without it, you lose the visuals the model just described.

from pydantic import BaseModel, Field
from mistralai.extra import response_format_from_pydantic_model

class ImageDescription(BaseModel):
    image_type: str = Field(
        ..., description="Type: graph, table, photo, logo, schematic"
    )
    description: str = Field(
        ..., description="Short description of the image in English"
    )
    relevant_content: str = Field(
        ..., description="Summary of the informative content of the image"
    )

ocr_response = client.ocr.process(
    model="mistral-ocr-latest",
    document={
        "type": "document_url",
        "document_url": "https://example.com/report.pdf"
    },
    bbox_annotation_format=response_format_from_pydantic_model(
        ImageDescription
    ),
    include_image_base64=True
)

# Access image annotations
for page in ocr_response.pages:
    for image in page.images:
        if image.image_annotation:
            desc = ImageDescription.model_validate_json(image.image_annotation)
            print(f"Image {image.id}: {desc.image_type}")
            print(f"  Description: {desc.description}")

Nothing forces you to choose between the two: on an illustrated report, you’ll generally want both the document metadata and the description of its charts, which a single call covers by passing both formats.

ocr_response = client.ocr.process(
    model="mistral-ocr-latest",
    document={
        "type": "document_url",
        "document_url": "https://example.com/report.pdf"
    },
    bbox_annotation_format=response_format_from_pydantic_model(
        ImageDescription
    ),
    document_annotation_format=response_format_from_pydantic_model(
        InvoiceExtraction
    ),
    include_image_base64=True
)

Guiding Extraction with a Prompt

A schema describes the expected shape, not the document’s context. When your fields are ambiguous—multiple dates, multiple amounts, two similar sections—document_annotation_prompt helps resolve the doubt by telling the model the nature of the document and what to prioritize. On a French commercial contract, this framing sentence often makes the difference between usable extraction and approximate filling.

ocr_response = client.ocr.process(
    model="mistral-ocr-latest",
    document={
        "type": "document_url",
        "document_url": "https://example.com/contract.pdf"
    },
    document_annotation_format=response_format_from_pydantic_model(
        ContractExtraction
    ),
    document_annotation_prompt=(
        "This document is a French commercial contract. "
        "Extract information about the contracting parties, "
        "the contract duration, and the main clauses."
    )
)

Key Takeaways

Two types of annotations: bbox_annotation for images, document_annotation for the entire document
Pydantic schemas define the structure of the data to extract
Use response_format_from_pydantic_model() to convert the schema
The document_annotation_prompt guides extraction with contextual instructions
Both types can be combined in a single call