Web Browsing

Lesson updated November 3, 2026 based on OpenAI’s Computer use guide, its Integration recipes, and the Responses API reference: instructions must be passed via the instructions parameter and resent with every request, as they are not carried over by previous_response_id; the script that removed target="_blank" contained a JavaScript syntax error and only covered links already present; image recommendations now follow OpenAI’s guidelines (detail: "original", resizing with coordinate conversion). Sources: Integration recipes · Computer use.

Beyond the Simple Click

For an agent, browsing isn’t just opening a URL. It’s navigating a real web filled with redirects, pages that take four seconds to render, consent banners that cover everything, and content that only appears after multiple scrolls. An agent that ignores these frictions fails by the second page; one that anticipates them chains together a ten-screen journey without any intervention.

Two Ways to Get Somewhere

When you know the exact address, don’t ask the AI to find it: send the browser there directly.

page.goto("https://example.com/products")
page.wait_for_load_state("networkidle")

Conversely, when the address depends on what’s displayed—the third product in a price-sorted list, a dynamically generated “View Invoice” link—let the model look and click.

# The AI sees the screenshot and decides to click a link
# This happens automatically in the Computer Use loop

The right approach is to combine both in a single workflow: direct navigation for known milestones in your journey, and visual navigation for anything discovered along the way. This saves you costly iterations where they add no value.

The Cookie Banner: Obstacle Number One

In Europe, nearly every site displays a consent banner on the first load. As long as it’s present, it captures clicks, and your agent spins its wheels without understanding why the page isn’t reacting. Don’t improvise a fix site by site: bake it once and for all into the agent’s instructions—the instructions parameter of the Responses API, which acts as the system prompt—with a clear directive on what to refuse.

INSTRUCTIONS = """You are an agent that browses the web.
Rules:
- If you see a cookie or consent banner,
  refuse optional cookies or click "Refuser" / "Reject"
- If only "Accept" is offered, accept only if the task allows it;
  otherwise, stop and report it
- Don't waste time on popups: handle them immediately
"""

# Pass this to EVERY request in the loop: with previous_response_id,
# instructions from the previous response are not carried over.
response = client.responses.create(
    model="gpt-6.1-sol",
    tools=[{"type": "computer"}],
    instructions=INSTRUCTIONS,
    input=tache,
)

The order of the first two rules matters: the agent refuses by default, and accepting remains a decision for the person assigning the task, not the model. Also, note the code comment: the Responses API documentation states that instructions from one response are not carried over to the next when chaining via previous_response_id. A loop that sends them only on the first turn loses them on the second.

Three Ways to Wait

Modern pages load content asynchronously, and your agent might click where a button hasn’t arrived yet. The most reliable fix remains Playwright’s network wait, which returns control once all requests are finished.

page.goto(url)
page.wait_for_load_state("networkidle")
# All network requests are complete

This isn’t enough everywhere: on an app that maintains a persistent connection—websocket, polling, live chat—the network is never idle, and the wait eventually times out. Fall back to a simple delay before capturing.

import time

def wait_and_screenshot(page, wait_seconds=2):
    """Wait, then capture a screenshot."""
    time.sleep(wait_seconds)
    return page.screenshot()

There’s a third, often overlooked path: entrust the judgment to the model itself. It sees the spinner or half-rendered content and understands it’s better to request an additional screenshot than to act blindly.

task = """
Navigate to https://example.com/dashboard.
If the page still appears to be loading 
(spinner visible, partial content), 
wait for an additional screenshot before acting.
"""

Guiding a Long Journey

Over a sequence of several pages, a task formulated in a single sentence falls apart along the way: the agent forgets half the instructions by the third screen. Break it down into explicit steps—thanks to response chaining via previous_response_id, the model keeps track of the ones it has already completed and always knows where it stands.

task = """
Step 1: Go to https://shop.example.com
Step 2: Search for "mechanical keyboard" in the search bar
Step 3: Sort results by ascending price
Step 4: Open the first product
Step 5: Note the price and availability
"""

Tabs Are Not Managed by the AI

Computer Use works on only one viewport. If a link opens in a new tab, the agent keeps photographing the old one and doesn’t understand why nothing has changed—a classic situation with target="_blank" links, found on almost every news site. Fix this upstream in Playwright, so the model doesn’t have to worry about it.

REMOVE_BLANK = """
document.addEventListener("click", (e) => {
  const link = e.target.closest && e.target.closest('a[target="_blank"]');
  if (link) link.removeAttribute("target");
}, true);
"""

def handle_new_tab(context, page):
    """Keep navigation in the tab the agent photographs."""
    # On every loaded page, a click on a target="_blank" link
    # opens it in the current tab...
    context.add_init_script(REMOVE_BLANK)
    # ... and any window opened by script is closed immediately
    page.on("popup", lambda popup: popup.close())

The initialization script runs in every page before its own scripts: the listener, set during the capture phase, removes the target attribute before the browser follows the link. The page’s popup event covers what remains: windows opened by window.open.

Image Quality Conditions Everything Else

A blurry or cropped screenshot produces an agent that clicks the wrong button, and you’ll spend more time debugging its errors than fixing the screenshot. Capture the visible viewport rather than the full page: a full_page capture produces an oversized image, and worse, its coordinates no longer match the screen where page.mouse clicks.

import base64

def optimized_screenshot(page) -> str:
    """Capture visible viewport as PNG, in base64."""
    screenshot = page.screenshot(
        type="png",
        full_page=False,  # Only the visible viewport
    )
    return base64.b64encode(screenshot).decode("utf-8")

On the API side, OpenAI recommends sending captures with detail: "original", which preserves resolution and improves click precision, and advises against high and low for Computer Use. A large capture consumes more tokens, and the API may resize it, or even reject it if it exceeds limits. If you must reduce your captures, do it before sending and convert the model’s coordinates into the original image space; OpenAI reports good results with desktops at 1440×900 and 1600×900. The 1280×720 viewport used in this course does not need reduction.

Key Takeaways

• Combine direct navigation (known URLs) and visual navigation (dynamic links) based on context
• Handle GDPR popups in the agent’s instructions, resent with every request
• Wait for full loading before capturing a screenshot
• Capture the visible viewport and send it with detail: "original"; if you resize the image, convert the coordinates
• Handle new tabs in Playwright, not in the AI