Vision: Analyze Images with Grok

Image Understanding with Chat Completions

Grok models (starting from grok-4.5) include vision capabilities that let you analyze, describe, and reason about images. You use this feature via the array format of the content field, combining text and image_url objects.

Use cases are numerous: extracting text from scanned documents (OCR), analyzing charts, describing user interfaces, visual comparison, or interpreting technical diagrams.

Send an Image via URL

The simplest way is to provide the public URL of the image:

from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("XAI_API_KEY"),
    base_url="https://api.x.ai/v1"
)

response = client.chat.completions.create(
    model="grok-4.20-0309-reasoning",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe what you see in this image."},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://example.com/photo-architecture.jpg",
                        "detail": "high"
                    }
                }
            ]
        }
    ]
)

print(response.choices[0].message.content)

xAI servers download the image at the time of the request. It must be publicly accessible.

Send an Image in Base64

For local or private images, use base64 encoding:

import base64

def encode_image(image_path: str) -> str:
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_data = encode_image("capture-ecran.png")

response = client.chat.completions.create(
    model="grok-4.20-0309-reasoning",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What text is visible in this screenshot?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{image_data}"
                    }
                }
            ]
        }
    ]
)

The URL format is data:image/{format};base64,{data} where {format} is jpeg, jpg, or png.

Detail Levels: auto, low, and high

The detail parameter controls the resolution at which the model analyzes the image:

detail: “low”

{
  "type": "image_url",
  "image_url": {
    "url": "https://example.com/icone.png",
    "detail": "low"
  }
}

The image is downscaled to a low resolution before analysis. This is the fastest mode and costs the least tokens. Use it for simple images: icons, logos, basic diagrams.

detail: “high”

{
  "type": "image_url",
  "image_url": {
    "url": "https://example.com/document-scan.jpg",
    "detail": "high"
  }
}

The image is analyzed at high resolution. The model can read small text, identify fine details, and analyze specific areas. This is the recommended mode for documents, screenshots, and detailed photographs.

detail: “auto”

The model automatically chooses the detail level based on the size and complexity of the image. This is the default value if you omit the parameter.

Analyze Multiple Images

You can send multiple images in a single message for comparison or cross-analysis tasks:

response = client.chat.completions.create(
    model="grok-4.20-0309-reasoning",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Compare these two versions of the design. What are the differences?"},
                {"type": "image_url", "image_url": {"url": "https://example.com/v1.png", "detail": "high"}},
                {"type": "image_url", "image_url": {"url": "https://example.com/v2.png", "detail": "high"}}
            ]
        }
    ]
)

There is no limit on the number of images per request, but each image consumes additional tokens.

Technical Constraints

Keep these limits in mind when using vision:

• Maximum size: 20 MiB per image
• Supported formats: JPG, JPEG, and PNG only
• Compatible models: grok-4 and higher (older models do not support vision)
• Tokens: images consume tokens that add to the text cost of the request

Key Takeaways

• Vision uses the array format of the content field with image_url objects
• Images can be sent via public URL or encoded in base64
• Three detail levels: auto (default), low (fast/economical), and high (detailed)
• Maximum size of 20 MiB per image, JPG and PNG formats only
• Multiple images can be sent in a single message for comparisons