Annotation Use Cases

Update November 2, 2026 — OCR. OCR 4.0 (mistral-ocr-4-0) was retired on September 30, 2026. Use OCR 4.1 (mistral-ocr-4-1), generally available since August 31 at the same price, or the alias mistral-ocr-latest, which has pointed to it since July 16. OCR 4.1 adds confidence scores at the block level. Addendum November 3, 2026: A local PDF is transmitted in base64 as {"type": "document_url", "document_url": "data:application/pdf;base64,…"}; the document_base64 type from older examples exists neither in the documentation nor in the mistralai SDK, and those examples would have failed. Sources: OCR Processor · Document QnA. Corrected November 3, 2026: The code previously read page.document_annotation page by page; in mistralai SDK 3.7.0, document annotation is a single field in the response (response.document_annotation), and image annotation is read from image.image_annotation.

Annotations for Every Role

Document AI annotations transform unstructured documents into actionable data. The mechanism is always the same—a schema, a prompt, a call—but what you extract changes radically depending on the role. This lesson walks through four concrete implementations: classification upstream in a pipeline, contract clause analysis, regulatory compliance checks, and enriching a RAG index.

Sort Before Processing

In a real-world workflow, documents arrive mixed together: an invoice, a purchase order, an ID card, a letter. Before applying the appropriate extraction schema, you need to know what you are dealing with. Annotation serves as the first sorting stage. The TypeDocument enum locks in admissible categories, the confidence score lets you route anything below a threshold for human review, and the one-sentence summary gives operators enough context to make a decision without opening the file.

import os
from enum import Enum
from pydantic import BaseModel, Field
from mistralai.client import Mistral
from mistralai.extra import response_format_from_pydantic_model

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])

class TypeDocument(str, Enum):
    INVOICE = "facture"
    PURCHASE_ORDER = "bon_commande"
    CONTRACT = "contrat"
    REPORT = "rapport"
    LETTER = "courrier"
    FORM = "formulaire"
    ID_DOCUMENT = "piece_identite"
    OTHER = "autre"

class Classification(BaseModel):
    document_type: TypeDocument = Field(
        ..., description="Identified document type"
    )
    confidence: float = Field(
        ..., description="Confidence score between 0 and 1"
    )
    language: str = Field(
        ..., description="Primary language (ISO 639-1 code)"
    )
    summary: str = Field(
        ..., description="One-sentence summary of the content"
    )

def classify_document(document_base64: str) -> Classification:
    """Automatically classifies a document."""
    response = client.ocr.process(
        model="mistral-ocr-latest",
        document={
            "type": "document_url",
            "document_url": f"data:application/pdf;base64,{document_base64}"
        },
        document_annotation_format=response_format_from_pydantic_model(
            Classification
        ),
        document_annotation_prompt=(
            "Identify the type of this document and provide "
            "a summary of its content."
        )
    )

    if response.document_annotation:
        return Classification.model_validate_json(
            response.document_annotation
        )

    raise ValueError("No annotation returned")

Read a Contract Like a Lawyer

Contract analysis is a major use case in legal and finance, and it is where nested schemas truly shine. Each Clause carries its title, a content summary, its category—confidentiality, termination, non-compete, liability, payment—and a risk assessment on three levels. The contract itself aggregates these clauses along with the parties, signature date, duration, amount, and applicable jurisdiction. On a portfolio of two hundred supplier contracts, you obtain in a few hours a mapping that no one had time to produce manually.

The risk field warrants a warning: it is an assessment produced by a model, useful for prioritizing a lawyer’s review, never for replacing it.

from pydantic import BaseModel, Field
from typing import Optional

class Clause(BaseModel):
    title: str = Field(..., description="Title or number of the clause")
    content: str = Field(..., description="Summary of the clause content")
    clause_type: str = Field(
        ..., description="Type: confidentiality, termination, "
        "non-compete, liability, payment, other"
    )
    risk: str = Field(
        ..., description="Risk level: low, medium, high"
    )

class ContractAnalysis(BaseModel):
    parts: list[str] = Field(
        ..., description="Names of the contracting parts"
    )
    signature_date: Optional[str] = Field(
        None, description="Signature date (YYYY-MM-DD)"
    )
    duration: Optional[str] = Field(
        None, description="Contract duration"
    )
    key_clauses: list[Clause] = Field(
        ..., description="Identified main clauses"
    )
    amount: Optional[float] = Field(
        None, description="Total amount if mentioned"
    )
    jurisdiction: Optional[str] = Field(
        None, description="Applicable jurisdiction"
    )

def analyze_contract(document_base64: str) -> ContractAnalysis:
    """Analyzes a contract and extracts main clauses."""
    response = client.ocr.process(
        model="mistral-ocr-latest",
        document={
            "type": "document_url",
            "document_url": f"data:application/pdf;base64,{document_base64}"
        },
        document_annotation_format=response_format_from_pydantic_model(
            ContractAnalysis
        ),
        document_annotation_prompt=(
            "Analyze this commercial contract. Identify the parts, "
            "the main clauses, and evaluate the risk level "
            "of each clause."
        )
    )

    if response.document_annotation:
        return ContractAnalysis.model_validate_json(
            response.document_annotation
        )

    raise ValueError("No annotation returned")

Verify Compliance Automatically

Regulatory control is particularly well-suited to annotation because the criteria are written in black and white in the law. For a French invoice, the prompt lists the obligations to verify—mandatory mentions (SIRET, intra-community VAT, payment terms, late payment penalties), amount formatting, issue date—and the schema enforces, for each criterion, a boolean verdict accompanied by its justification. You retrieve a global score and improvement recommendations, which is enough to block a non-compliant invoice before it enters the payment circuit.

from pydantic import BaseModel, Field

class CompliancePoint(BaseModel):
    criterion: str = Field(..., description="Compliance criterion checked")
    compliant: bool = Field(..., description="True if compliant, False otherwise")
    justification: str = Field(
        ..., description="Explanation of compliance or non-compliance"
    )

class ComplianceAudit(BaseModel):
    document_type: str = Field(..., description="Type of document audited")
    compliance_points: list[CompliancePoint] = Field(
        ..., description="Compliance points checked"
    )
    global_score: float = Field(
        ..., description="Global compliance score (0-100)"
    )
    recommendations: list[str] = Field(
        ..., description="Improvement recommendations"
    )

def audit_invoice_compliance(document_base64: str) -> ComplianceAudit:
    """Verifies invoice compliance according to French rules."""
    response = client.ocr.process(
        model="mistral-ocr-latest",
        document={
            "type": "document_url",
            "document_url": f"data:application/pdf;base64,{document_base64}"
        },
        document_annotation_format=response_format_from_pydantic_model(
            ComplianceAudit
        ),
        document_annotation_prompt=(
            "Verify the compliance of this invoice according to "
            "French legal obligations: mandatory mentions "
            "(SIRET, intra-community VAT, payment terms, "
            "late payment penalties), amount formatting, issue date."
        )
    )

    if response.document_annotation:
        return ComplianceAudit.model_validate_json(
            response.document_annotation
        )

    raise ValueError("No annotation returned")

Lose Nothing Before Indexing

One last use case, more discreet but decisive: preparing content for a RAG index. An index built solely on Markdown loses everything that was visual, and your searches will never find the chart that held the key figure. The function below recomposes, page by page, the text and the description of each image, then adds the metadata extracted from the document, in a single block ready to be split into chunks.

def enrich_for_rag(ocr_response) -> str:
    """Enriches OCR content with annotations for RAG."""
    parts = []

    for page in ocr_response.pages:
        # Text content
        parts.append(page.markdown)

        # Image annotations (replace images with their description)
        for image in page.images:
            if image.image_annotation:
                parts.append(
                    f"[Description of {image.id}]: {image.image_annotation}"
                )

    # Document annotation: a single one, carried by the response
    if ocr_response.document_annotation:
        parts.append(
            f"[Extracted Metadata]: {ocr_response.document_annotation}"
        )

    return "\n\n".join(parts)

By adding image descriptions and extracted metadata to the indexed content, you lose no information from the original document. You will notice that these four scenarios share the same backbone: what distinguishes them is the schema and the prompt, not the calling code.

Key Takeaways

• Automatic classification routes documents to the correct processing pipeline
• Contract clause extraction identifies risks and key points
• Compliance verification automates regulatory audits
• Annotation enrichment improves RAG indexing quality
• Each use case boils down to a well-designed Pydantic schema and an adapted prompt