Image Formats: URL and Base64

Updated November 3, 2026. With the Responses API, input_image and input_text objects go in the content of a {"role": "user", ...} message, not directly in input. The Python example previously called client.responses.create() on the xai-sdk client, which does not have this method, and wrote the API key in the code: it now uses user(...) and image(...) from the SDK, with the key read from the environment. Added xAI’s warning about history storage. Code verified with xai-sdk 1.21.0. Source: Image Understanding.

Sending Images to Grok

Grok models from grok-4.5 onward include vision capabilities. You can send one or more images in your API requests, and the model will analyze them to answer your questions. This is called image understanding, or multimodal vision.

Before diving into use cases, it’s essential to master the two image submission formats supported by the xAI API.

Direct URL

The simplest method is to provide the public URL of an image. The model will download and analyze it automatically.

With the Responses API (main endpoint):

{
  "model": "grok-4.5",
  "input": [
    {
      "role": "user",
      "content": [
        {
          "type": "input_image",
          "image_url": "https://exemple.com/photo-produit.jpg"
        },
        {
          "type": "input_text",
          "text": "Describe this image in detail."
        }
      ]
    }
  ]
}

xAI recommends that when you send images, you do not store the request and response history on their servers, otherwise the request may fail: with the Responses API, pass "store": false.

With the Chat Completions API (legacy, OpenAI compatible):

{
  "model": "grok-4.5",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "https://exemple.com/photo-produit.jpg"
          }
        },
        {
          "type": "text",
          "text": "Describe this image in detail."
        }
      ]
    }
  ]
}

The URL must point to a publicly accessible image file. If the image is behind authentication or a protected CDN, use the Base64 format instead.

Base64 Encoded (Data URI)

The second format involves encoding the image directly in the request as a Base64 string preceded by a MIME prefix. This is the ideal method when working with local images, screenshots, or dynamically generated files.

{
  "type": "input_image",
  "image_url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ..."
}

The prefix data:image/jpeg;base64, tells the model the MIME type of the image. Adjust it according to the format:

• data:image/jpeg;base64, for JPG/JPEG files
• data:image/png;base64, for PNG files

Encoding an Image in Python

import base64

with open("photo.jpg", "rb") as f:
    image_data = base64.standard_b64encode(f.read()).decode("utf-8")

image_url = f"data:image/jpeg;base64,{image_data}"

Encoding in JavaScript/Node.js

import { readFileSync } from 'fs';

const imageData = readFileSync('photo.jpg').toString('base64');
const imageUrl = `data:image/jpeg;base64,${imageData}`;

Key Differences Between URL and Base64

Mixing Text and Images

The content of a message accepts a mixed array of input_image and input_text objects, in any order: you can freely alternate text and images. With the native xAI SDK, the same message is built with user(), which accepts text and images created by image() in sequence:

from xai_sdk import Client
from xai_sdk.chat import image, user

client = Client()  # key read from XAI_API_KEY

chat = client.chat.create(model="grok-4.5")
chat.append(
    user(
        "Compare these two photos:",
        image(image_url="https://exemple.com/photo1.jpg", detail="high"),
        image(image_url="https://exemple.com/photo2.jpg", detail="high"),
        "What are the main differences?",
    )
)

response = chat.sample()
print(response.content)

image() accepts a public URL or a Base64 Data URI.

Key Takeaways

• Two image formats are supported: Direct URL and Base64 (Data URI)
• Accepted formats are JPG, JPEG, and PNG only
• Each image must not exceed 20 MiB
• The number of images per request is unlimited
• The Responses API places input_image in the content of a message; Chat Completions uses image_url parts; the xAI SDK uses image() inside user()
• Prefer URL for public images and Base64 for local files