M365Con 2027 – free online conference · 11–22 Jan 2027 · 120 sessions m365con.net

Computer-Using Agent (CUA)

Concept

A computer-using agent is an AI agent that reads a graphical user interface and operates it like a human by clicking, typing and navigating, which lets agents work with legacy systems that have no API.

✦ CONCEPTComputer-Using Agent(CUA)M365.FM
Title graphic: Computer-Using Agent (CUA) (Concept)

Also discussed

Episodes about Computer-Using Agent (CUA) also cover Process Mining, API-first architecture, Copilot Studio topics, Digital twin, Foundry Agent Service and Service account.

Description

A Computer-Using Agent (CUA) is an AI agent that reads a graphical user interface and operates it in the same way a person would: by looking at the screen, clicking, typing and moving through menus and pages. Because it works at the interface level, a CUA can handle software that offers no API, which makes it relevant for legacy systems that would otherwise be difficult to automate.

The idea behind a CUA is that a user gives instructions in natural language and the agent carries out the task on their behalf. The M365.FM episode "Computer-Using Agents: How CUA Could Transform SaaS" describes a CUA as an AI system that interacts with software for the user, completes tasks, automates workflows and reduces the need to use traditional user interfaces. The episode frames this as a possible shift in how SaaS products are built and used, with agent-driven software taking over routine work so that people can focus on higher-value tasks.

Microsoft Learn documents a concrete implementation under the name Computer Use in Azure OpenAI in Microsoft Foundry. According to that documentation, Computer Use is a specialized AI tool, backed by its own model, that interacts with the user interfaces of computer systems and applications. It interprets what is visible on the screen and acts on it, for example by opening applications, clicking buttons, filling in forms and working through multi-page workflows. It can also adapt when the interface changes, which is one of the practical differences from fixed, script-based screen automation.

The tool is called through the Responses API and runs in a loop. The model returns a computer call containing a list of actions such as click, type, scroll, drag, wait or screenshot. The application then executes those actions and sends a screenshot back to the model. This repeats until the model no longer requests further actions. An important detail is that the model itself does not execute anything: the calling application is responsible for performing the actions, so the surrounding code decides what is actually allowed to happen on the machine.

Several practical and safety points follow from this design. Access requires registering for the model through a limited-access form and then creating a deployment. The documentation defines safety checks for malicious instructions in screenshots, irrelevant domains and sensitive domains, and these checks must be acknowledged before the process continues. The sample provided is described as experimental and intended for a low-privilege test environment, with a human confirming decisions and no sensitive data exposed to the model. Anyone considering a CUA for real workflows therefore has to weigh the convenience of API-free automation against the need for oversight, restricted permissions and careful scoping.

Typical decisions around a CUA include which tasks are suitable for delegation, which systems the agent may touch, and how humans stay in the loop. The M365.FM episode also points to organizational preparation, noting that pricing models evolve and require clear definitions of what counts as an agent and what counts as an outcome, and it advises focusing on adaptive workflows and agent-driven systems as preparation for the next phase of enterprise technology.

Overview compiled with AI from M365.FM episodes and the facts on this page. You can improve it with the edit button below.

TypeConcept

Episodes about Computer-Using Agent (CUA)

Episodes in which Computer-Using Agent (CUA) is discussed:

Timeline of episodes about Computer-Using Agent (CUA)

Frequently asked questions

Questions marked with AI were written with AI from the facts on this page.

What is a Computer-Using Agent (CUA)? (AI)

A Computer-Using Agent is an AI agent that reads a graphical user interface and operates it like a human by clicking, typing and navigating. This allows it to work with systems that have no API.

How does the Computer Use tool in Microsoft Foundry work? (AI)

It is called through the Responses API in a loop: the model returns a list of actions such as click, type or scroll, the application executes them and returns a screenshot, and this repeats until no further actions are requested. The model does not execute anything itself.

What safety considerations apply to Computer Use? (AI)

The documentation describes safety checks for malicious instructions in screenshots, irrelevant domains and sensitive domains that must be acknowledged. Its sample is experimental and meant for a low-privilege test environment with a human confirming decisions and no sensitive data exposed.

Glossary

Community wiki

Short info above, community knowledge below: add practical tips, examples or corrections. New contributions are reviewed first; trusted members publish directly.

Experts and companies for this topic

Nobody has listed this topic yet. Add it to your profile.

Sources

Maintained by Mirko Peters (Microsoft MVP) · Last updated: