Architecture: Screenshots, Actions, and the Interaction Loop

Lesson rewritten on November 3, 2026 based on OpenAI’s Computer use guide and Computer use integration recipes: This course previously relied on Anthropic’s API (import anthropic, tool computer_20241022). It now uses OpenAI’s Responses API computer tool, with actions grouped in a computer_call and screenshots returned via computer_call_output. We have removed the unsourced token counts and duration estimates from the previous version. Sources: Computer use · Integration recipes.

The perception-action loop

The entire architecture of Computer Use boils down to a single cycle: observe → decide → act → observe. Until this cycle is clear in your mind, your automations will remain erratic; once it clicks, the rest is just plumbing.

The difference from a Selenium script lies in where the knowledge resides. Selenium blindly executes a pre-determined sequence: click here, then there, then type this. Computer Use chooses the next step only after seeing the result of the previous one. A popup appears, a page takes longer to load, or a button label changes overnight: in all three cases, the model sees the actual situation and adjusts, whereas the script continues reciting into the void.

With OpenAI’s computer tool, the exchange happens in three steps: you send the task, you execute the actions the model returns, and then you send back a screenshot. Your application holds the mouse and keyboard; the model only makes the decisions.

The starting point: the screenshot

Every turn relies on an image of the current screen state. This screenshot comes from a browser controlled by Playwright—the most common case for web automation—a virtual desktop when automating a desktop application, or a Docker container that isolates the agent. OpenAI recommends using an isolated browser or virtual machine in all cases.

import base64

def capture_screen(page) -> str:
    """Capture the screen with Playwright and return the PNG image as base64."""
    return base64.b64encode(page.screenshot()).decode("utf-8")

The screenshot goes back to the model as a PNG data URL. It counts as an image input, so it affects the cost of every turn: the window size you capture is a real design parameter.

Analysis and decision

The model receives the task, the conversation history, and the latest screenshot. Its response isn’t a sentence but a computer_call item, which carries an ordered list of actions to execute. Here is an example from OpenAI’s documentation that selects a search field and then types a word:

{
  "type": "computer_call",
  "call_id": "call_002",
  "actions": [
    { "type": "click", "button": "left", "x": 405, "y": 157 },
    { "type": "type", "text": "penguin" }
  ],
  "status": "completed"
}

Two details matter. The first call might contain only a screenshot action: the model is asking to see the screen before acting, and you return a screenshot without changing anything. Also, status: "completed" means the model has finished generating the call, not that the actions have been performed: executing them is up to you.

The action repertoire

The vocabulary available to the model is intentionally limited; combined, these primitives cover most of what a human does with a mouse and keyboard.

The full flow

Put together, these elements create the following loop, modeled on OpenAI’s documentation. As long as the response contains a computer_call, you execute its actions, capture the screen, and return the screenshot in a computer_call_output that reuses the same call_id; previous_response_id links each turn to the previous one.

from openai import OpenAI

client = OpenAI()

def computer_use_loop(page, task: str):
    """Computer Use loop with the computer tool from the Responses API."""
    response = client.responses.create(
        model="gpt-6.1-sol",
        tools=[{"type": "computer"}],
        input=task,
    )

    while True:
        call = next(
            (item for item in response.output if item.type == "computer_call"),
            None,
        )
        if call is None:
            return response.output_text  # no more actions requested: task is done

        execute_actions(page, call.actions)  # your action handler (Lesson 4)
        screenshot = capture_screen(page)

        response = client.responses.create(
            model="gpt-6.1-sol",
            tools=[{"type": "computer"}],
            previous_response_id=response.id,
            input=[{
                "type": "computer_call_output",
                "call_id": call.call_id,
                "output": {
                    "type": "computer_screenshot",
                    "image_url": f"data:image/png;base64,{screenshot}",
                    "detail": "original",
                },
            }],
        )

This loop has no guardrails yet: in production, limit the number of turns, duration, and cost; stop if a call includes pending safety checks (pending_safety_checks); and verify the actual result rather than trusting only the model’s final message. These are OpenAI’s safety guidelines, covered in Lesson 12.

What a loop turn costs

Every request to the Responses API charges for input and output tokens, and every screenshot counts as an image input; a hosted environment may be billed separately. The cost of a task therefore depends on the number of turns, screenshot size, and model—and it grows faster than the number of turns, because even with previous_response_id, all input tokens from previous responses in the chain are re-charged as input on every request. OpenAI provides an image cost calculator and exposes usage for each request: measure on your own workflows before budgeting. A Selenium script will always do the same thing faster and cheaper—but you’ll have to rewrite it the next time the form changes, which the agent does not require.

Key takeaways

• The task → actions → screenshot loop is the heart of Computer Use; your application executes the actions
• With OpenAI, declare the tool with tools=[{"type": "computer"}] in the Responses API
• A computer_call carries an ordered list of actions (click, type, scroll, keypress, etc.) to execute in sequence
• The screenshot returns to the model in a computer_call_output with the same call_id, chained by previous_response_id
• Every turn costs tokens and an image: limit turns and measure before budgeting