Image Understanding in Web Search

Analyze images found on the web

Web Search goes beyond text. With the enable_image_understanding parameter, you can ask the model to analyze images it encounters during its web search. This feature significantly extends search capabilities, especially for visual questions or topics where images provide information that complements the text.

Activation

Enable this feature by adding the enable_image_understanding parameter to your Web Search configuration:

{
  "tools": [{
    "type": "web_search",
    "enable_image_understanding": true
  }]
}

When enabled, the model can use the internal view_image tool to analyze images encountered during the search. It automatically decides which images are relevant to examine.

SDK Syntax

xAI SDK

response = client.responses.create(
    model="grok-4.20-0309-reasoning",
    input="Show me the Transformer architecture and explain each component",
    tools=[client.tools.web_search(
        enable_image_understanding=True
    )]
)

OpenAI SDK

response = client.responses.create(
    model="grok-4.20-0309-reasoning",
    input="Show me the Transformer architecture",
    tools=[{
        "type": "web_search",
        "enable_image_understanding": True
    }]
)

Vercel AI SDK

tools: [xai.tools.webSearch({
  enableImageUnderstanding: true
})]

Use Cases

Technical diagram analysis

The model can find and interpret architecture diagrams, flow charts, or technical graphs. This is particularly useful for questions about software architectures or industrial processes.

Visual product comparison

For comparative queries, the model can analyze screenshots or product photos found on the web and draw relevant observations from them.

Visual data analysis

If your question concerns data presented as charts or visual tables, image understanding allows the model to extract information from these visual representations.

Combining with other parameters

Image understanding combines with domain filtering:

response = client.responses.create(
    model="grok-4.20-0309-reasoning",
    input="Analyze the latest LLM performance benchmarks",
    tools=[{
        "type": "web_search",
        "allowed_domains": ["arxiv.org", "huggingface.co"],
        "enable_image_understanding": True
    }]
)

The model will search only within allowed domains and can analyze benchmark charts found in publications.

Cost impact

Enabling image understanding increases token consumption. Analyzing each image generates additional tokens for input (the encoded image) and output (the description/analysis). The cost per invocation remains the same ($0.005), but the token volume per request can increase significantly.

Enable this feature only when images add real value to your use case. For purely text-based searches, leave this parameter disabled to optimize costs.

Difference from X Search

Image understanding is available for both Web Search and X Search. However, video understanding (enable_video_understanding) is exclusive to X Search. Web Search cannot analyze videos found on the web.

Key takeaways

• Enable with enable_image_understanding: true in the Web Search configuration
• The model uses view_image internally to analyze relevant images
• Useful for diagrams, charts, screenshots, and visual comparisons
• Increases token consumption—enable only when relevant
• Combines with domain filtering for precise source control
• Video understanding is exclusive to X Search