First workflow: navigate and interact
On this page
Lesson rewritten on November 3, 2026 based on OpenAI’s Computer use guide and Computer use integration recipes: the agent uses the
computertool from the Responses API, an action handler modeled on the documentation (grouped actions,keypress,scroll_x/scroll_y,dragpaths), and chaining viaprevious_response_id, replacing code written for Anthropic’s API. Sources: Computer use · Integration recipes.
Assembling the pieces
You have the loop in mind and the environment configured. Now you need to bring them together into a working program: an agent that opens an online encyclopedia, searches for a term, opens the first result, and reports back the title of the page it reached. The scenario seems modest, but it engages the fundamental gestures—navigation, typing, clicking, reading information—and everything you build next will just be a longer variation of this.
The action handler
The model decides, but it doesn’t execute anything itself: it returns a list of actions, and your code translates them into Playwright gestures, in order. This function follows the handler published by OpenAI in its integration recipes, simplified for a browser.
import time
KEYS = {
"ENTER": "Enter", "RETURN": "Enter", "ESC": "Escape", "ESCAPE": "Escape",
"TAB": "Tab", "SPACE": "Space", "BACKSPACE": "Backspace", "DELETE": "Delete",
"UP": "ArrowUp", "DOWN": "ArrowDown", "LEFT": "ArrowLeft", "RIGHT": "ArrowRight",
"CTRL": "Control", "CONTROL": "Control", "SHIFT": "Shift", "ALT": "Alt",
"CMD": "Meta", "META": "Meta",
}
BUTTONS = {"left": "left", "right": "right", "wheel": "middle"}
def execute_actions(page, actions):
"""Executes the actions from a computer_call in order."""
for action in actions:
if action.type in ("click", "double_click", "drag", "move", "scroll") and getattr(action, "keys", None):
raise ValueError("Mouse action with modifier key: not handled here")
match action.type:
case "click":
button = getattr(action, "button", "left")
if button not in BUTTONS: # back and forward don't exist in Playwright
raise ValueError(f"Unsupported button: {button}")
page.mouse.click(action.x, action.y, button=BUTTONS[button])
case "double_click":
page.mouse.dblclick(action.x, action.y)
case "drag":
points = [(p.x, p.y) for p in action.path]
page.mouse.move(*points[0])
page.mouse.down()
for x, y in points[1:]:
page.mouse.move(x, y)
page.mouse.up()
case "move":
page.mouse.move(action.x, action.y)
case "scroll":
page.mouse.move(action.x, action.y)
page.mouse.wheel(action.scroll_x, action.scroll_y)
case "keypress":
page.keyboard.press("+".join(KEYS.get(k, k) for k in action.keys))
case "type":
page.keyboard.type(action.text)
case "wait":
time.sleep(2)
case "screenshot":
continue # screenshot is taken after each batch of actions
case _:
raise ValueError(f"Unsupported action: {action.type}")Two choices deserve explanation. The key and button names returned by the model (ENTER, CTRL, wheel…) aren’t what Playwright expects (Enter, Control, middle…): the mapping tables, taken from the documentation, bridge the gap, and the back and forward buttons, which the API can also request, are rejected as in OpenAI’s handler. And an unknown action raises an error rather than being silently ignored: a hard stop is better than an agent that thinks it acted when it didn’t.
The complete loop
The agent then orchestrates the cycle. The first call sends the task; if it contains only a screenshot request, you simply return the screen. Each screenshot comes back to the model as the result of the call that requested it, referenced by its call_id, and previous_response_id links the turns together.
import base64
from openai import OpenAI
from playwright.sync_api import sync_playwright
client = OpenAI()
MODEL = "gpt-6.1-sol"
MAX_TURNS = 15
def launch_agent(task: str, start_url: str = "https://en.wikipedia.org"):
"""Launches a Computer Use agent on a given task."""
pw = sync_playwright().start()
browser = pw.chromium.launch(headless=False) # False to observe
page = browser.new_page(viewport={"width": 1280, "height": 720})
page.goto(start_url)
page.wait_for_load_state("networkidle")
try:
response = client.responses.create(
model=MODEL, tools=[{"type": "computer"}], input=task,
)
for turn in range(MAX_TURNS):
call = next((i for i in response.output if i.type == "computer_call"), None)
if call is None:
print(f"Task complete: {response.output_text}")
return response.output_text
if call.pending_safety_checks: # see lesson 12
raise RuntimeError("Pending safety checks: human intervention required")
for action in call.actions:
print(f" Action: {action.type}")
execute_actions(page, call.actions)
screenshot = base64.b64encode(page.screenshot()).decode("utf-8")
response = client.responses.create(
model=MODEL,
tools=[{"type": "computer"}],
previous_response_id=response.id,
input=[{
"type": "computer_call_output",
"call_id": call.call_id,
"output": {
"type": "computer_screenshot",
"image_url": f"data:image/png;base64,{screenshot}",
"detail": "original",
},
}],
)
raise RuntimeError("Maximum number of turns reached: check the page state")
finally:
browser.close()
pw.stop()Three safeguards are already in place. MAX_TURNS bounds the loop: without this limit, a disoriented agent would consume your budget until exhaustion. A call with pending safety checks (pending_safety_checks) stops everything, as in OpenAI’s official example: lesson 12 shows how to have them validated by a human. And normal completion is signaled by the absence of a computer_call: the model has stopped acting and is responding to you, and you retrieve the page title from output_text.
Launching the workflow
if __name__ == "__main__":
launch_agent(
"Search for 'Computer Use' in Wikipedia's search engine, "
"open the first result, and give me the title of the page. "
"Use the computer tool to act on the interface."
)With headless=False, the show is instructive: the window opens, the click lands in the search field, the text appears, the first link opens, and the title comes back in your console. Watch it run two or three times before moving on to more ambitious tasks; you’ll see precisely where the agent hesitates and what your instructions should address.
A reusable function for the rest of the course
The following lessons work on an already-open page—after a login, in the middle of a workflow. They rely on run_agent(task, page=page): the same loop, extracted from launch_agent, with optional instructions returned on each turn (lesson 5 explains why).
def run_agent(task: str, page, instructions: str | None = None, max_turns: int = MAX_TURNS) -> str:
"""Computer Use loop on an already-open page; returns the final message."""
request = {"model": MODEL, "tools": [{"type": "computer"}]}
if instructions:
request["instructions"] = instructions # to be returned on each request
response = client.responses.create(**request, input=task)
for turn in range(max_turns):
call = next((i for i in response.output if i.type == "computer_call"), None)
if call is None:
return response.output_text
if call.pending_safety_checks:
raise RuntimeError("Pending safety checks: human intervention required")
execute_actions(page, call.actions)
screenshot = base64.b64encode(page.screenshot()).decode("utf-8")
response = client.responses.create(
**request,
previous_response_id=response.id,
input=[{
"type": "computer_call_output",
"call_id": call.call_id,
"output": {
"type": "computer_screenshot",
"image_url": f"data:image/png;base64,{screenshot}",
"detail": "original",
},
}],
)
raise RuntimeError("Maximum number of turns reached: check the page state")Three failures you will encounter
Does the agent systematically click next to its target? The culprit is almost always a scale mismatch between the screenshot sent and the page you’re clicking on: resized screenshot, browser zoom, high-density screen. The model’s coordinates refer to the image it received; if you shrink it, convert its coordinates in the opposite direction.
Is the agent going in circles and repeating the same gesture? The turn limit will eventually stop it, but it’s better to detect the loop sooner: if the same screenshot comes back several times in a row, the agent isn’t making progress, and interrupting it saves you unnecessary calls.
Is the agent acting on a page that’s still empty? On a heavy site, page.goto() returns before rendering finishes, and the first screenshot shows only a white background. Wait for network inactivity before taking the picture:
page.goto(url)
page.wait_for_load_state("networkidle")Key takeaways
- A Computer Use agent combines an action handler, a control loop, and response chaining
- The first call sends the task; each screenshot comes back in a
computer_call_outputwith the samecall_id - The handler executes actions in order and rejects what it doesn’t know how to do
- Always limit the number of turns, stop on
pending_safety_checks, and useheadless=Falseduring development run_agent(task, page=page)reuses this loop for an already-open page; this is what the following lessons use