Computer-Using Agent (CUA)
Concept
A computer-using agent is an AI agent that reads a graphical user interface and operates it like a human by clicking, typing and navigating, which lets agents work with legacy systems that have no API.
נושאים שנדונים גם
בפרקים על Computer-Using Agent (CUA) עולים גם: Process Mining, API-first architecture, Copilot Studio topics, תאום דיגיטלי, Foundry Agent Service ו-Service account.
Description
A Computer-Using Agent (CUA) is an AI agent that reads a graphical user interface and operates it in the same way a person would: by looking at the screen, clicking, typing and moving through menus and pages. Because it works at the interface level, a CUA can handle software that offers no API, which makes it relevant for legacy systems that would otherwise be difficult to automate.
The idea behind a CUA is that a user gives instructions in natural language and the agent carries out the task on their behalf. The M365.FM episode "Computer-Using Agents: How CUA Could Transform SaaS" describes a CUA as an AI system that interacts with software for the user, completes tasks, automates workflows and reduces the need to use traditional user interfaces. The episode frames this as a possible shift in how SaaS products are built and used, with agent-driven software taking over routine work so that people can focus on higher-value tasks.
Microsoft Learn documents a concrete implementation under the name Computer Use in Azure OpenAI in Microsoft Foundry. According to that documentation, Computer Use is a specialized AI tool, backed by its own model, that interacts with the user interfaces of computer systems and applications. It interprets what is visible on the screen and acts on it, for example by opening applications, clicking buttons, filling in forms and working through multi-page workflows. It can also adapt when the interface changes, which is one of the practical differences from fixed, script-based screen automation.
The tool is called through the Responses API and runs in a loop. The model returns a computer call containing a list of actions such as click, type, scroll, drag, wait or screenshot. The application then executes those actions and sends a screenshot back to the model. This repeats until the model no longer requests further actions. An important detail is that the model itself does not execute anything: the calling application is responsible for performing the actions, so the surrounding code decides what is actually allowed to happen on the machine.
Several practical and safety points follow from this design. Access requires registering for the model through a limited-access form and then creating a deployment. The documentation defines safety checks for malicious instructions in screenshots, irrelevant domains and sensitive domains, and these checks must be acknowledged before the process continues. The sample provided is described as experimental and intended for a low-privilege test environment, with a human confirming decisions and no sensitive data exposed to the model. Anyone considering a CUA for real workflows therefore has to weigh the convenience of API-free automation against the need for oversight, restricted permissions and careful scoping.
Typical decisions around a CUA include which tasks are suitable for delegation, which systems the agent may touch, and how humans stay in the loop. The M365.FM episode also points to organizational preparation, noting that pricing models evolve and require clear definitions of what counts as an agent and what counts as an outcome, and it advises focusing on adaptive workflows and agent-driven systems as preparation for the next phase of enterprise technology.
Overview compiled with AI from M365.FM episodes and the facts on this page. You can improve it with the edit button below.
| Type | Concept |
|---|
Community wiki
Short info above, community knowledge below: add practical tips, examples or corrections. New contributions are reviewed first; trusted members publish directly.
Experts and companies for this topic
Nobody has listed this topic yet. Add it to your profile.
Maintained by Mirko Peters (Microsoft MVP) · Last updated: