What Is Computer Use

Lesson updated November 3, 2026 based on OpenAI’s Computer use guide: this course now follows the computer tool in the Responses API, replacing the previous Anthropic API approach. The section on ecosystem players covers both paths documented by OpenAI (code execution, recommended for GPT-6 Astra, and the computer tool). We have removed unsourced claims about competing products. Source: Computer use.

A New Way to Interact with Machines

For years, language models remained purely text-based interlocutors: you asked a question, received an answer, and anything involving the real world was left to you. Computer Use breaks down that barrier. The model sees a screen and decides which gestures to perform—clicking, typing, scrolling, switching applications—and your application executes those actions on its behalf.

This shift runs deeper than it appears. Traditional automation clings to technical anchors: a CSS selector, a DOM ID, or an API that a software vendor chose to expose. The day the front-end team renames a class, the script halts, and often no one notices until the next day’s report arrives empty. With Computer Use, you issue an instruction in natural language, and the AI navigates the interface exactly as a human perceives it. A “Submit” button moved three inches or repainted green remains the “Submit” button.

The Loop That Makes It Possible

The mechanism operates in five steps, and their repetition creates its power:

1. Your application captures the screen or browser window
2. The model analyzes the image to understand what it sees
3. It decides on one or more actions (click, keystroke, scroll)
4. Your application executes these actions on the machine
5. A new capture goes back to the model, and the loop restarts

This perception-action loop is the core of the system. Consider a common scenario: your agent must fill out a contact form, but a cookie banner pops up over it on load. A Selenium script that didn’t anticipate this clicks into thin air and throws an error that is hard to interpret. The Computer Use agent, however, sees the banner in its screenshot, handles it, takes another capture, and finds the form again. It does not recite a rigid plan; it re-evaluates the situation at every iteration.

What We Actually Automate

The primary use case is end-to-end testing. You describe the journey to verify—“log in, add an item to the cart, proceed to checkout”—and the agent plays it out like a user, without requiring you to maintain a library of selectors that expire with every site redesign.

Next comes RPA, the automation of repetitive tasks on software lacking APIs: a legacy internal ERP, a supplier extranet, or a business application from the 2000s whose vendor has vanished. Where technical integration is impossible, the agent works through the screen, exactly like the employee who used to copy data by hand.

Scraping also becomes accessible on dynamic sites that resist traditional extractors, since the agent reads the rendered page rather than raw HTML. Multi-application workflows—pulling an order from a CRM, entering it into a spreadsheet, sending a confirmation email—chain together in a single session. This same capability serves user support: instead of explaining where to click, the agent actually navigates the interface.

The Players in 2026

This course follows OpenAI’s offering, which opens two paths in the Responses API. With code execution, the model writes a script—Playwright or PyAutoGUI, for example—that your isolated environment executes before returning the result and screenshots; a single call can then chain loops and conditions, making this the path OpenAI recommends for GPT-6 Astra. With the computer tool, the model returns structured mouse and keyboard actions that your application translates into gestures; this tool remains supported, and it is the one you will use here because it exposes the loop in its raw form. OpenAI SDKs also provide ready-made helpers (ComputerToolset, BrowserToolset) that manage this loop for you.

Other providers, notably Anthropic, offer a comparable tool in their own API with a different exchange format: the code in this course does not transpose directly, but the principles—loop, action manager, isolation, human confirmation—remain the same.

What You Gain, What You Pay

Read this table as a trade-off, not a ranking. You exchange raw speed for resilience and near-zero maintenance costs. For fifty thousand daily executions on a stable page where you control the HTML, a script remains the right choice. For a journey tweaked every month, or for software without an API, the balance shifts decisively the other way—and this course teaches you to handle that second case.

Key Takeaways

• Computer Use lets AI see and control a screen via a screenshot-action loop
• It is a different paradigm from classic scripts: more resilient but slower
• Primary use cases include E2E testing, RPA, scraping, and multi-application workflows
• OpenAI offers two paths: code execution, recommended for GPT-6 Astra, and the computer tool, used in this course