Extract Structured Data from Images
On this page
Turn Images into Actionable Data
The previous lesson focused on transcribing text; this one goes further by producing structured data. Extracting structured data means transforming the visual content of an image—a table, form, business card, or label—into JSON, CSV, or any other format your code can manipulate directly, without manual review.
The stakes are higher. An approximate transcription might still help a human reader, but missing a single key in a JSON response will crash the program consuming it. This lesson covers the two guarantees you need to implement: the API’s JSON mode for syntactic validity, and schema validation for business compliance.
The API’s JSON Mode
Mistral offers a response_format parameter that forces the model to return valid JSON. Without it, you’ll waste time cleaning up responses wrapped in Markdown blocks or preceded by polite introductory sentences.
from mistralai.client import Mistral
import os
import json
client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])
response = client.chat.complete(
model="mistral-large-latest",
response_format={"type": "json_object"},
messages=[
{
"role": "system",
"content": "Always return a valid JSON object with the data extracted from the image."
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "Extract the following data from this business card as JSON: name, title, company, email, phone, address."
},
{"type": "image_url", "image_url": data_uri_carte}
]
}
]
)
# The result is directly parsable
donnees = json.loads(response.choices[0].message.content)
print(json.dumps(donnees, indent=2, ensure_ascii=False))The json.loads call runs without special precautions: response_format={"type": "json_object"} guarantees syntactically valid JSON output. However, it does not guarantee which keys are present. That’s why you must describe the expected format in the prompt itself, as shown in the examples below.
Extract a Table from an Image
Photographed tables or screenshots are a common use case, and this is where an explicit schema pays off the most. Without clear instructions, the model will invent its own structure, which may differ with every call.
prompt_tableau = """Extract the content of this table as JSON.
Expected format:
{
"columns": ["Column1", "Column2", ...],
"rows": [
{"Column1": "value", "Column2": "value"},
...
]
}
Read all visible rows. If a cell is empty, use null."""
response = client.chat.complete(
model="mistral-large-latest",
response_format={"type": "json_object"},
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": prompt_tableau},
{"type": "image_url", "image_url": encoder_image("captures/tableau-excel.png")}
]
}
]
)
donnees = json.loads(response.choices[0].message.content)
print(f"Columns: {donnees['columns']}")
print(f"Number of rows: {len(donnees['rows'])}")The instruction regarding empty cells is critical. Without it, the model might shift values to fill gaps, distorting the table. Displaying the row count provides an immediate sanity check: if the image shows twelve rows but the script reports eight, the extraction is incomplete.
Forms and Administrative Documents
Filled-out forms introduce two challenges: handwriting and checkboxes, which have no text value. Your schema must therefore include dedicated fields for these.
prompt_formulaire = """Analyze this filled form and extract all fields and their values.
Expected JSON format:
{
"form_type": "...",
"fields": {
"field_name": "entered_value",
...
},
"checked_boxes": ["field1", "field2"],
"signature_present": true/false
}
If a field is empty or illegible, use null."""
response = client.chat.complete(
model="mistral-large-latest",
response_format={"type": "json_object"},
messages=[
{
"role": "system",
"content": "You are a precise form extractor. Return structured JSON."
},
{
"role": "user",
"content": [
{"type": "text", "text": prompt_formulaire},
{"type": "image_url", "image_url": encoder_image("formulaires/cerfa-12345.jpg")}
]
}
]
)Extract Product Label Data
Food labels follow a standard structure: a nutrition facts table per 100 g, an ingredient list, and allergens. You can define the schema down to the unit name, avoiding guesswork later about whether “12” referred to grams or kilocalories.
prompt_etiquette = """Extract the nutritional information from this label.
JSON format:
{
"product_name": "...",
"net_weight": "...",
"nutrition_per_100g": {
"energy_kcal": ...,
"fat_g": ...,
"saturated_fat_g": ...,
"carbohydrates_g": ...,
"sugars_g": ...,
"protein_g": ...,
"salt_g": ...
},
"ingredients": "...",
"allergens": ["..."]
}"""
response = client.chat.complete(
model="mistral-large-latest",
response_format={"type": "json_object"},
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": prompt_etiquette},
{"type": "image_url", "image_url": encoder_image("produits/etiquette.jpg")}
]
}
]
)
produit = json.loads(response.choices[0].message.content)Extraction Pipeline with Validation
In production, syntactic guarantees aren’t enough. The model might return perfectly valid JSON where email is missing, or where phone contains an integer instead of a string. Pydantic turns these cases into explicit exceptions at the extraction stage, rather than three steps downstream in your pipeline.
from pydantic import BaseModel, Field
from typing import Optional
import json
class BusinessCard(BaseModel):
last_name: str
first_name: str
title: Optional[str] = None
company: Optional[str] = None
email: Optional[str] = None
phone: Optional[str] = None
address: Optional[str] = None
website: Optional[str] = None
def extract_business_card(image_path):
"""Extract and validate business card data."""
response = client.chat.complete(
model="mistral-large-latest",
response_format={"type": "json_object"},
messages=[
{
"role": "system",
"content": "Extract data from this business card. Fields: last_name, first_name, title, company, email, phone, address, website. Return JSON."
},
{
"role": "user",
"content": [
{"type": "text", "text": "Extract information from this business card."},
{"type": "image_url", "image_url": encoder_image(image_path)}
]
}
]
)
# Parse and validate with Pydantic
raw_data = json.loads(response.choices[0].message.content)
card = BusinessCard(**raw_data)
return card
# Usage
card = extract_business_card("cartes/contact-dupont.jpg")
print(f"Contact: {card.first_name} {card.last_name} - {card.email}")Note the distinction between last_name: str and title: Optional[str] = None. A card without a name is a failed extraction; a card without a job title is a normal case. You draw that line, and Pydantic enforces it.
Convert to CSV or DataFrame
The final step is feeding your data processing pipeline. The JSON produced above converts directly:
import pandas as pd
# From an extracted table
donnees = json.loads(response.choices[0].message.content)
# Create a pandas DataFrame
df = pd.DataFrame(donnees["rows"])
print(df.to_string())
# Export to CSV
df.to_csv("export/tableau_extrait.csv", index=False, encoding="utf-8")The encoding="utf-8" argument during export is mandatory whenever your data contains accented characters or currency symbols.
Key Takeaways
- The
response_format={"type": "json_object"}parameter guarantees valid JSON output - Always provide an expected JSON schema in your prompt to guide extraction
- Use Pydantic to validate extracted data in production
- Tables, forms, labels, and business cards are ideal targets
- Combine extraction with pandas for direct integration into your data workflows