OCR and Text Transcription in Images

Reading Text in Images

OCR (Optical Character Recognition) is one of the most common use cases for the Vision API. With Mistral’s multimodal models, you can extract text from photos, screenshots, scanned documents, invoices, or receipts without installing a dedicated OCR library—no Tesseract, no language models, no preprocessing pipeline.

The advantage isn’t just cutting back on a dependency. Classic OCR returns characters; a multimodal model understands what it reads and can, in the same call, distinguish an invoice number from a purchase order number. This lesson covers common use cases and techniques that concretely improve accuracy.

Simple OCR: Reading Visible Text

The most direct approach comes down to a single prompt, provided it’s unambiguous:

from mistralai.client import Mistral
import os

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])

response = client.chat.complete(
    model="mistral-small-latest",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Transcribe exactly all visible text in this image. Do not comment, do not summarize--return only the text as it appears."},
                {"type": "image_url", "image_url": "https://exemple.com/document-scanne.jpg"}
            ]
        }
    ]
)

print(response.choices[0].message.content)

The keyword is “exactly”. Without this instruction, the model tends to paraphrase or summarize the content because that’s what a helpful assistant does spontaneously: here you’re asking for the opposite of a service, a literal restitution.

Extracting Text from an Invoice

Invoices contain structured information—issuer, recipient, amounts, dates—that the model can identify by their role, not just their position. This is precisely the advantage over fixed-zone OCR, which breaks as soon as the supplier changes the layout:

prompt_facture = """Analyze this invoice and extract the following information:
- Issuer (name, address, SIRET if visible)
- Recipient (name, address)
- Invoice number
- Issue date
- Due date
- Billing lines (description, quantity, unit price, total)
- Subtotal excluding tax
- VAT (rate and amount)
- Total including tax

Return the result as structured JSON."""

response = client.chat.complete(
    model="mistral-large-latest",
    messages=[
        {
            "role": "system",
            "content": "You are a certified accountant. Extract billing data with precision. Return only valid JSON."
        },
        {
            "role": "user",
            "content": [
                {"type": "text", "text": prompt_facture},
                {"type": "image_url", "image_url": data_uri_facture}
            ]
        }
    ]
)

Transcribing Screenshots

Another frequent need is retrieving content from an interface or terminal shared as a screenshot, typically in a support ticket. Here, formatting is part of the information: a dedented terminal trace becomes unusable.

import base64

def encoder_image(chemin):
    with open(chemin, "rb") as f:
        b64 = base64.standard_b64encode(f.read()).decode("utf-8")
    return f"data:image/png;base64,{b64}"

# Transcribe a terminal screenshot
response = client.chat.complete(
    model="mistral-small-latest",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Transcribe the terminal output visible in this screenshot. Preserve formatting and indentation."
                },
                {"type": "image_url", "image_url": encoder_image("captures/terminal.png")}
            ]
        }
    ]
)

Reading Receipts and Cash Register Tickets

Thermal receipts are the toughest case: faint printing, crumpled paper, fading ink. The decisive technique is to explicitly allow the model to declare defeat on an item; otherwise, it will fill gaps with plausible values.

prompt_recu = """Read this cash register ticket and extract:
1. Store name
2. Date and time
3. List of items (name, price)
4. Subtotal
5. Payment method
6. Total amount

If an item is illegible, indicate [illegible].
Return the result as JSON."""

response = client.chat.complete(
    model="mistral-large-latest",  # Large for best reading
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": prompt_recu},
                {"type": "image_url", "image_url": encoder_image("recu-supermarche.jpg")}
            ]
        }
    ]
)

A [illegible] in the result is infinitely better than a made-up amount: the former triggers human verification, the latter goes unnoticed until accounting reconciliation. For these degraded documents, use Mistral Large 3—the difference in accuracy with Small is significant on blurry or partially legible text.

Batch Document Processing

Scaling up follows the pattern we’ve already seen: a unit function parameterized by the model, then a loop over the folder.

import json
from pathlib import Path

def ocr_document(chemin_image, modele="mistral-small-latest"):
    """Performs OCR on a document and returns the text."""
    response = client.chat.complete(
        model=modele,
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "Transcribe the entire text of this document. Preserve formatting (headings, paragraphs, lists)."
                    },
                    {"type": "image_url", "image_url": encoder_image(chemin_image)}
                ]
            }
        ]
    )
    return response.choices[0].message.content

# Process a folder
dossier = Path("documents_scannes/")
resultats = {}

for fichier in sorted(dossier.glob("*.jpg")):
    print(f"OCR of {fichier.name}...")
    resultats[fichier.name] = ocr_document(fichier)

# Save
with open("transcriptions.json", "w", encoding="utf-8") as f:
    json.dump(resultats, f, ensure_ascii=False, indent=2)

print(f"{len(resultats)} documents transcribed.")

The modele parameter in the signature isn’t cosmetic: it lets you rerun only doubtful documents with Large without rewriting the function.

Improving OCR Accuracy

Two levers act upstream of the model. The first is image preprocessing: a gray, low-contrast scan becomes legible after conversion to grayscale, contrast enhancement, and sharpening.

from PIL import Image, ImageEnhance, ImageFilter

def pretraiter_pour_ocr(chemin):
    """Improves an image for OCR: contrast, sharpness, grayscale."""
    img = Image.open(chemin)

    # Convert to grayscale
    img = img.convert("L")

    # Increase contrast
    enhancer = ImageEnhance.Contrast(img)
    img = enhancer.enhance(2.0)

    # Increase sharpness
    img = img.filter(ImageFilter.SHARPEN)

    return img

The 2.0 factor works for a faded scan; pushed too high, it creates artifacts that degrade reading. Test on three representative documents before locking in the value.

The second lever is prompt specialization by document type. A form calls for “Read each field and its corresponding value,” a letter for “Transcribe the body of the letter preserving paragraphs,” a table for “Extract table data in markdown format.” Three formulations, three output structures: classify your documents before transcribing them, and quality jumps a notch.

Vision API vs Document AI

For advanced OCR and document parsing, Mistral also offers Document AI (OCR 4), a specialized service. The following table outlines the boundary between the two.

The decisive line is multi-page PDFs: the Vision API processes images, not paginated documents. As soon as your source is a PDF of several dozen pages, the question no longer arises. We’ll cover Document AI in Lesson 12 (Next Steps).

Key Takeaways

• The Vision API performs adequate OCR for simple cases (invoices, screenshots, receipts)
• Use the word “exactly” in your prompt to avoid paraphrasing
• Mistral Large 3 is significantly better than Small for poor-quality documents
• Image preprocessing (contrast, grayscale) improves results
• For advanced OCR and multi-page PDFs, consider Mistral’s Document AI (OCR 4)