Multi-Modal Prompts

Text and images in the same prompt

Grok 4.5 and later models can analyze images in addition to text. This multimodal capability opens up a wide range of possibilities: image description, OCR, chart analysis, visual comparison, and more.

Mastering multi-modal prompts lets you use Grok as a visual assistant in addition to a text assistant.

Sending an image to Grok

With the Responses API (main endpoint), images are integrated into the input field as typed objects:

{
  "model": "grok-4.5",
  "input": [
    {"type": "input_image", "image_url": "https://exemple.com/dashboard.png"},
    {"type": "input_text", "text": "Analyze this dashboard and identify declining metrics."}
  ]
}

With the Chat Completions API (legacy endpoint), the structure is slightly different:

{
  "model": "grok-4.5",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "image_url", "image_url": {"url": "https://exemple.com/dashboard.png", "detail": "high"}},
      {"type": "text", "text": "Analyze this dashboard and identify declining metrics."}
    ]
  }]
}

The detail parameter: controlling accuracy

The detail level influences the depth of analysis and the number of tokens consumed:

detail: low

The image is processed quickly at a reduced resolution. Use this mode to:

• Classify images (photo, illustration, diagram)
• Describe general content
• Identify main objects
• Sort images by category

Consumption: fewer tokens, faster response.

detail: high

The image is analyzed in high resolution. Use this mode to:

• Read text in an image (OCR)
• Analyze details in a chart or diagram
• Compare precise visual elements
• Detect defects or anomalies

Consumption: more tokens (between 256 and 1,792 depending on size), slower response.

detail: auto

The model chooses the appropriate level itself. This is the default behavior.

Image sources

Direct URL

The simplest method: provide an HTTPS link to the image.

{"type": "input_image", "image_url": "https://exemple.com/photo.jpg"}

Base64 (data URI)

For images that are not hosted online, encode them in base64:

{"type": "input_image", "image_url": "data:image/png;base64,iVBORw0KGgo..."}

This method is useful for screenshots, programmatically generated images, or images from a processing pipeline.

Practical use cases

OCR and text extraction

{
  "input": [
    {"type": "input_image", "image_url": "https://exemple.com/facture.jpg"},
    {"type": "input_text", "text": "Extract from this invoice: the invoice number, date, amount excluding tax, VAT, and amount including tax. Return the result in JSON."}
  ]
}

Chart analysis

{
  "input": [
    {"type": "input_image", "image_url": "https://exemple.com/graphique-ventes.png"},
    {"type": "input_text", "text": "This chart shows our monthly sales over 12 months. Identify trends, peaks, and troughs. Suggest possible explanations for the variations."}
  ]
}

Image comparison

{
  "input": [
    {"type": "input_text", "text": "Compare these two versions of our logo:"},
    {"type": "input_image", "image_url": "https://exemple.com/logo-v1.png"},
    {"type": "input_image", "image_url": "https://exemple.com/logo-v2.png"},
    {"type": "input_text", "text": "Identify the differences and recommend the best version for web use."}
  ]
}

User interface analysis

{
  "input": [
    {"type": "input_image", "image_url": "https://exemple.com/mockup.png"},
    {"type": "input_text", "text": "Evaluate this mobile interface mockup based on the following criteria: readability, visual hierarchy, accessibility, consistency. Rate each criterion out of 10."}
  ]
}

Key technical constraints

• Maximum size: 20 MB per image
• Formats: JPG, JPEG, and PNG only (no GIF, SVG, or WebP)
• No limit on the number of images per request
• xAI tip: use store: false in the Responses API when working with images to avoid errors related to history storage

Writing a good multi-modal prompt

The same clarity principles apply, with specifics for images:

• Announce the image: tell the model what it will see before the image
• Be precise about the task: “Describe this image” is vague; “Extract data from the bar chart and return it in CSV” is precise
• Guide attention: if only part of the image interests you, indicate it (“Focus on the top right corner”)
• Specify output format: as with text prompts, specify if you expect JSON, a list, or a table

Hands-on practice

Take a screenshot of a dashboard, chart, or document, and send it to Grok with a specific request. Test with detail: low then detail: high and compare the quality and speed of the responses.

Key takeaways

• grok-4.5+ models analyze images in addition to text
• The detail parameter (low, auto, high) controls analysis depth and token cost
• Images can be provided via URL or base64
• Announce the image, be precise about the task, and specify the output format
• Use store: false to avoid storage issues with images
• Accepted formats are limited to JPG, JPEG, and PNG, with a maximum size of 20 MB