OCR: Text Extraction

Read Text in Images

OCR (Optical Character Recognition) is one of the most powerful use cases for Grok Vision. Unlike traditional tools that only extract characters, Grok understands the context of the text in the image, its layout, and its meaning.

Classic OCR vs. Multimodal OCR

Traditional OCR tools (Tesseract, Google Vision OCR, AWS Textract) excel at raw character extraction. Grok adds an extra dimension: it understands what the text means within the context of the image.

For example, when processing a scanned invoice, a classic OCR tool extracts all characters. Grok can also:

• Identify fields (invoice number, date, total amount)
• Structure data in a usable format
• Correct OCR errors using context
• Interpret tables and complex layouts

Basic Extraction

For simple extraction of visible text:

response = client.responses.create(
    model="grok-4.5",
    input=[
        {
            "type": "input_image",
            "image_url": url_document,
            "detail": "high"  # Essential for OCR
        },
        {
            "type": "input_text",
            "text": "Extract all visible text in this image. Preserve the layout as much as possible."
        }
    ],
    store=False
)

The detail: "high" parameter is essential for OCR. In low mode, the model may miss fine text or misread certain characters.

Structured Document Extraction

For professional documents, request structured extraction:

def extraire_facture(image_url):
    prompt = """This image is an invoice. Extract the following
information in JSON:
- "invoice_number": the invoice number
- "date": the issue date
- "issuer": name and address of the issuer
- "recipient": name and address of the recipient
- "lines": table with columns [description, quantity, unit_price, total]
- "subtotal": amount excluding tax
- "vat": VAT amount
- "total_incl_tax": amount including tax

If a field is not visible, indicate null."""

    response = client.responses.create(
        model="grok-4.5",
        input=[
            {"type": "input_image", "image_url": image_url, "detail": "high"},
            {"type": "input_text", "text": prompt}
        ],
        store=False
    )
    return response.output_text

Business Card Extraction

def extraire_carte_visite(image_url):
    prompt = """Extract the information from this business card in JSON:
- "name": full name
- "title": job title/position
- "company": company name
- "email": email address
- "phone": phone number(s)
- "address": postal address
- "website": website URL
- "social_media": links to social media profiles"""

    response = client.responses.create(
        model="grok-4.5",
        input=[
            {"type": "input_image", "image_url": image_url, "detail": "high"},
            {"type": "input_text", "text": prompt}
        ],
        store=False
    )
    return response.output_text

Screenshot Processing

Screenshots are an excellent use case because they often contain text in complex layouts (menus, sidebars, popups):

def analyser_screenshot(image_url):
    prompt = """Analyze this screenshot and identify:
1. The application or website displayed
2. The visible page or section
3. The main interface elements (buttons, menus, forms)
4. Any important text displayed
5. Any error messages or notifications"""

    response = client.responses.create(
        model="grok-4.5",
        input=[
            {"type": "input_image", "image_url": image_url, "detail": "high"},
            {"type": "input_text", "text": prompt}
        ],
        store=False
    )
    return response.output_text

Batch Document Processing

To process a large volume of documents, combine Grok OCR with a processing pipeline:

import json

async def pipeline_ocr(documents):
    resultats = []

    for doc in documents:
        # 1. Validate the image
        if not valider_image(doc["chemin"]):
            resultats.append({"fichier": doc["chemin"], "erreur": "Invalid image"})
            continue

        # 2. Encode to base64
        image_url = encoder_base64(doc["chemin"])

        # 3. Extract with Grok
        response = client.responses.create(
            model="grok-4.5",
            input=[
                {"type": "input_image", "image_url": image_url, "detail": "high"},
                {"type": "input_text", "text": doc["prompt"]}
            ],
            store=False
        )

        # 4. Parse the result
        try:
            data = json.loads(response.output_text)
            resultats.append({"fichier": doc["chemin"], "data": data})
        except json.JSONDecodeError:
            resultats.append({"fichier": doc["chemin"], "texte_brut": response.output_text})

    return resultats

Grok OCR Limitations

• Handwritten text: recognition works well for legible writing, but may fail on cursive or illegible handwriting
• Non-Latin languages: accuracy varies depending on the language and font used
• Low-quality images: low-resolution scans or blurry photos degrade results
• Very small text: even in high mode, text smaller than 8 pixels in height may be misread

Key Takeaways

• Always use detail: "high" for OCR tasks
• Request structured extraction (JSON) to use data automatically
• Grok goes beyond raw OCR: it understands context and layout
• Use store: false for sensitive documents (invoices, business cards)
• Validate and preprocess images to maximize extraction quality