Vision API Limits and Optimization

Know the limits to work around them

The previous ten lessons covered what the Vision API does well. This one covers where it struggles. This isn’t an exercise in humility: the difference between a demo prototype that wows and an application that holds up in production comes down to exactly this. Here, you’ll learn about technical constraints, common pitfalls—visual hallucinations chief among them—and strategies to optimize cost and latency.

Technical limits

We’ve touched on format constraints before, but let’s bring them together. Each image is capped at roughly 20 MB and will be resized internally anyway, typically to 768x768 or equivalent: a 4K image won’t improve results. Accepted formats are JPEG, PNG, GIF (first frame only), and WebP. The Vision API does not support PDF directly—you need to call Document AI for that, as discussed in Lesson 8.

The most structural constraint remains token consumption. Each image consumes a significant number of tokens, and this function translates that estimate into dollars rather than abstract units:

# Estimate the cost of a vision call
def estimate_cost(num_images=1, text_tokens=100, response_tokens=500,
                 tokens_per_image=1000, input_price=0.10, output_price=0.30):
    """Estimates the cost in dollars for a vision call (Small 3.2)."""
    input_tokens = (num_images * tokens_per_image) + text_tokens
    output_tokens = response_tokens

    input_cost = (input_tokens / 1_000_000) * input_price
    output_cost = (output_tokens / 1_000_000) * output_price
    total_cost = input_cost + output_cost

    print(f"Input tokens: {input_tokens} (${input_cost:.4f})")
    print(f"Output tokens: {output_tokens} (${output_cost:.4f})")
    print(f"Total cost: ${total_cost:.4f}")
    return total_cost

# 1 image with Small 3.2
estimate_cost(num_images=1)

# 5 images with Large 3
estimate_cost(num_images=5, input_price=2.00, output_price=6.00)

Compare the two calls at the end of the script: the second costs dozens of times more than the first. Multiply that by your daily volume before making a decision. On top of that, there’s the context window limit, which the total of images + prompt + response must not exceed—a ceiling that multiple images reach much faster than a long text.

Visual hallucinations

The model can “invent” details absent from the image. This is the most critical issue in production because it’s silent: the response is well-formed, plausible, and wrong.

Four patterns recur. Invented text, where the model reads a word that doesn’t exist in the image. Approximate numbers, where values read from a chart can deviate 5–15% from reality. Added details, where the model fills in what it doesn’t see clearly with plausible information—a blurry SIRET number becomes a perfectly formatted but fake SIRET. Finally, element confusion, where two nearby objects are merged into one.

Three countermeasures exist; combine them based on criticality. First, force the model to qualify its own certainty. Second, have it reread the image after transcribing. Third, pit two models against the same question and keep only what they agree on.

# 1. Ask for a confidence level
prompt_confiance = """Analyze this image. For each piece of extracted information, indicate your confidence level:
- [CERTAIN]: clearly visible
- [PROBABLE]: partially visible or inferred
- [INCERTAIN]: barely legible, estimated

If an element is illegible, mark it [ILLISIBLE] rather than guessing."""

# 2. Ask for cross-verification
prompt_verification = """Read the text in this document. After transcribing, reread the image and verify every number and proper noun. Correct any detected errors."""

# 3. Double reading with two models
def double_verification(image_uri, question):
    """Sends the same image to two models and compares results."""
    resultats = {}
    for modele in ["mistral-small-latest", "mistral-large-latest"]:
        response = client.chat.complete(
            model=modele,
            messages=[{
                "role": "user",
                "content": [
                    {"type": "text", "text": question},
                    {"type": "image_url", "image_url": image_uri}
                ]
            }]
        )
        resultats[modele] = response.choices[0].message.content

    return resultats

Double reading doubles the cost: reserve it for amounts, identifiers, and anything that triggers an irreversible action.

Optimize costs

The most cost-effective optimization was covered in Lesson 5, and it’s the simplest: reduce image size before sending. A 6 MB photo resized to 1024 pixels loses nothing for analysis purposes and lightens the entire chain.

from PIL import Image
from io import BytesIO
import base64

def optimiser_image(chemin, taille_max=1024, qualite=80):
    """Resizes and compresses an image for the Vision API."""
    img = Image.open(chemin)
    taille_originale = img.size

    # Resize
    if max(img.size) > taille_max:
        img.thumbnail((taille_max, taille_max))

    # Compress as JPEG
    buffer = BytesIO()
    img.convert("RGB").save(buffer, format="JPEG", quality=qualite)
    taille_fichier = buffer.tell()

    b64 = base64.standard_b64encode(buffer.getvalue()).decode("utf-8")

    print(f"Original: {taille_originale} | Optimized: {img.size} | Size: {taille_fichier/1024:.0f} KB")
    return f"data:image/jpeg;base64,{b64}"

Next comes routing by complexity: not every task deserves the most expensive model. A three-line function is enough to decide, provided you’ve classified your tasks upstream.

def choose_model(task):
    """Selects the optimal model based on the task."""
    simple_tasks = ["classification", "ocr_simple", "description", "triage"]
    complex_tasks = ["extraction_json", "comparaison", "analyse_graphique", "raisonnement"]

    if task in simple_tasks:
        return "mistral-small-latest"  # ~$0.10/M tokens
    else:
        return "mistral-large-latest"  # ~$2.00/M tokens

The third lever is the most often forgotten: don’t pay twice for the same analysis. In production, the same image returns more often than you’d think—re-running a process, duplicate sends, or retrying after an incident. A disk cache indexed on the file hash and prompt eliminates these redundant calls.

import hashlib
import json
from pathlib import Path

CACHE_DIR = Path("cache_vision/")
CACHE_DIR.mkdir(exist_ok=True)

def analyser_avec_cache(image_path, prompt, modele="mistral-small-latest"):
    """Analyzes an image with disk caching."""
    # Generate a cache key
    with open(image_path, "rb") as f:
        image_hash = hashlib.md5(f.read()).hexdigest()
    cache_key = hashlib.md5(f"{image_hash}{prompt}{modele}".encode()).hexdigest()
    cache_file = CACHE_DIR / f"{cache_key}.json"

    # Check cache
    if cache_file.exists():
        return json.loads(cache_file.read_text())

    # API call
    response = client.chat.complete(
        model=modele,
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": prompt},
                {"type": "image_url", "image_url": encoder_image(image_path)}
            ]
        }]
    )

    resultat = response.choices[0].message.content

    # Save to cache
    cache_file.write_text(json.dumps(resultat, ensure_ascii=False))
    return resultat

The key includes the prompt and the model, not just the image: changing the question on the same photo should indeed trigger a new call.

Optimize latency

For an interactive application, four settings affect perceived response time. Resize images on the client side to a maximum of 1024 pixels, which shortens upload time. Choose Small or Ministral, as their generation is significantly faster. Pass images by URL when they are already hosted to avoid sending a large Base64 request body. Finally, enable streaming, which doesn’t reduce total time but displays the response as it builds:

# Streaming for a progressive response
stream = client.chat.stream(
    model="mistral-small-latest",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {"type": "image_url", "image_url": image_url}
        ]
    }]
)

for chunk in stream:
    if chunk.data.choices[0].delta.content:
        print(chunk.data.choices[0].delta.content, end="", flush=True)

It’s chat.stream replacing chat.complete; the rest of the message is exactly the same.

Production checklist

Before deploying a Vision application to production, verify each of these points:

• Automatic image resizing (max 1024px)
• Format and size validation before sending
• API error handling (retry, timeout, fallback)
• Caching results for identical images
• Intelligent routing (Small for simple, Large for complex)
• Cost and latency monitoring
• Testing with poor-quality images (blurry, low resolution, rotated)
• Anti-hallucination strategy (confidence levels, double verification)

The second-to-last point is the one most often skipped, and the one that causes the most damage: build a dataset of deliberately bad images and run it before every deployment.

Key takeaways

• Images are resized internally—no need to send very high resolution
• Visual hallucinations are the primary risk—ask for confidence levels
• Client-side resizing is the most cost-effective optimization
• Routing by complexity (Small vs. Large) cuts costs by 90% on simple tasks
• Disk caching avoids redundant API calls on the same images