Extract Data from Web Pages

Lesson updated November 3, 2026 based on OpenAI’s Computer Use guide: a task ends when the response contains no computer_call, not based on the stop_reason from Anthropic’s API used in the previous version. The DOM extraction script had unquoted selectors, causing a JavaScript error; this is now fixed. Unsourced figures (tokens per screenshot, duration per page) have been replaced with cost factors documented by OpenAI. Sources: Computer Use · Image Cost Calculator.

Vision-Based Scraping

A classic extractor reads HTML: it traverses the DOM, follows a hierarchy of tags, and breaks the day the site switches to client-side rendering or renames its classes. Computer Use works differently—the agent reads the screen just like a human would, then returns what it understood in the format you requested. The practical impact is significant: sites previously inaccessible to scrapers become available, since what the agent sees is exactly what the user sees.

Read and Report

The simplest case involves asking for a reading of visible content and a structured return. The critical point is to enforce the output format in the prompt: without instructions, the model responds in prose, forcing you to parse free text, which ruins the entire benefit of the operation.

task = """
Go to https://news.example.com and extract the first 5 article 
headlines visible on the homepage.

Return the result in JSON format:
[
  {"rank": 1, "title": "..."},
  {"rank": 2, "title": "..."},
  ...
]
"""

The agent sees the screenshot, identifies the headlines, and returns the JSON in its final text response—the one that arrives when the response no longer contains a computer_call. You read this from response.output_text.

Extract Content Beyond the Screen

A 720-pixel-high viewport shows only a fraction of a results page. You must therefore describe not only what to collect, but also how to navigate the page and the stopping condition. Otherwise, the agent will scroll until your iterations run out, returning truncated data.

task = """
On this search results page:
1. Read all visible results and note them
2. Scroll down to see more results
3. Continue until you have collected 20 results or reached 
   the bottom of the page

For each result, extract:
- Title
- URL
- Short description

Return everything in JSON.
"""

Tables are the most favorable terrain because the model immediately grasps visual tabular structures: it associates a cell with its column by simple alignment, whereas a DOM extractor must count <td> elements and gets lost as soon as a cell is merged. Describe the expected columns, and it will fill the rows.

task = """
Open https://stats.example.com/prices and extract the price table 
displayed on the page.

The table has columns: Product, Price ex-VAT, VAT, Price inc-VAT.
Extract all rows and return in JSON:
{
  "headers": ["Product", "Price ex-VAT", "VAT", "Price inc-VAT"],
  "rows": [
    ["Widget A", "10.00", "20%", "12.00"],
    ...
  ]
}
"""

The Hybrid Approach: Navigate by Vision, Extract via DOM

For sensitive data—amounts, product references—visual reading remains approximate. The most robust combination divides roles according to each tool’s strengths: the agent handles the part that is difficult to script, namely reaching the right page through searches, filters, and consent dialogs; Playwright then takes over where precision matters most, reading values directly from the DOM.

def extract_with_dom(page, task_description: str):
    """Use Computer Use to navigate, Playwright to extract."""
    # Phase 1: Computer Use navigates and identifies the area
    run_agent(task_description, page=page)
    
    # Phase 2: Playwright extracts precise data
    data = page.evaluate("""
        () => {
            const rows = document.querySelectorAll('table tbody tr');
            return Array.from(rows).map(row => {
                const cells = row.querySelectorAll('td');
                return Array.from(cells).map(c => c.textContent.trim());
            });
        }
    """)
    
    return data

Non-Standard Structures

Product cards, profiles, listings: these layouts escape regular templates, and this is precisely where vision takes the lead. The prompt must then name each expected field, specify its type, and provide a complete output example. Otherwise, you might retrieve “€250,000” in one record and 250000 in the next, causing downstream processing to break on the first line.

task = """
On this real estate results page, extract for each listing:
- price (number, in euros)
- area (in m²)
- rooms (integer)
- city
- postal_code
- listing_url

Scroll to load more listings if necessary.
Stop when you have 10 listings or when there are no more.

Strict JSON output format:
{
  "listings": [
    {
      "price": 250000,
      "area": 65,
      "rooms": 3,
      "city": "Paris",
      "postal_code": "75011",
      "listing_url": "https://..."
    }
  ]
}
"""

Preserve Extraction Traceability

An extraction without metadata becomes unusable three weeks later: you no longer know where the figures came from or when they were dated, making comparison with the next collection meaningless. Systematically record the source and timestamp alongside the data.

import json
from datetime import datetime, timezone

def save_extraction(data: dict, source_url: str):
    """Save extracted data with metadata."""
    now = datetime.now(timezone.utc)
    output = {
        "source": source_url,
        "extracted_at": now.isoformat(),
        "data": data,
    }

    filename = f"extraction_{now:%Y%m%d_%H%M%S}.json"
    with open(filename, "w", encoding="utf-8") as f:
        json.dump(output, f, ensure_ascii=False, indent=2)
    
    print(f"Data saved to {filename}")

What Vision Won’t Give You

Numerical precision is the first limitation. The model’s implicit OCR can confuse characters of similar shape—0 and O, 1 and l—which is inconsequential for an article headline but unacceptable for an IBAN or order reference: always verify critical data, or use DOM extraction for those fields.

Cost comes next. Each screenshot counts as an image input, the price of which depends on its size and the model—OpenAI provides a calculator to estimate this—and an extraction with scrolling chains many turns together. Since the entire chain history is re-billed as input on every request, the cost of a long collection grows faster than its number of screenshots. Therefore, any large volume to be collected daily warrants measurement on a few real pages before going to production. Time follows the same logic: each turn waits for a model response, which suits overnight processing but rarely a user interface waiting for a reply.

Key Takeaways

• Computer Use excels at extracting data from visually complex pages
• Always request a structured output format (JSON) in the prompt
• Combine AI navigation with Playwright DOM extraction for higher reliability
• Plan for scrolling on long pages
• Verify critical numerical data because implicit OCR isn’t perfect