Alibaba MAI-UI team releases OpenGCUA: an open full-stack for general computer use agents

The core question is: how should a model decide on its own how to observe, act, and organize across visual and programmatic interfaces to complete diverse computer tasks.

• Scenario: The authors argue the focus is shifting from pure GUI interaction to completing tasks on the user's own computer using visual and programmatic interfaces. Shared control, background windows, and mid-task requirement changes are situations not fully covered by benchmarks. The authors state that frontier systems such as GPT-6 Astra are largely closed, with weights, full prompts, real-machine harnesses, device SDKs, and trajectories unavailable.
• Fixed elements of existing systems: The authors state that GUI agents, CLI agents, multi-agent hybrids, and GUI+CLI with batched actions all pre-fix at least one of "what to observe, how to act, how to organize." The authors describe their earlier Qwen-UI-Agent as a restricted special case: it screenshots every step, actions are batched sequences without control flow, and working state is updated by fixed rules.
• The authors' definition: The authors define a generalist computer-use agent (GCUA) as a system that completes diverse computer tasks through visual interaction and programmatic access, with the model deciding how to observe, act, and organize work as the task requires. Programmatic access includes the shell, accessibility APIs, and available application or service APIs.
• What this paper provides: The authors propose OpenGCUA and state this report is the project's first release. It covers agent design and evaluation, trajectories from eight frontier models, and an initial real-machine harness and cross-device SDK. The authors state that no trained model is included yet.

Frontier computer-use systems

• OpenAI (2026): The authors state GPT-6 Astra targets difficult end-to-end work and recommends code execution for computer use while retaining a structured computer tool. Codex Computer Use can view and operate GUIs on macOS and Windows; on macOS it can run a limited range of background tasks. The authors state public materials lack key implementation details and execution trajectories.
• Anthropic (2026): Claude Cowork and Claude Code provide computer use; the authors state Claude first uses more precise connectors, and directly controls the browser, mouse, keyboard, and screen when no connector is available. Dispatch allows dispatching tasks from a phone and opening completed work on a computer. The authors likewise state details and trajectories are unavailable.

Computer-use agents

• GUI-oriented open models: UI-TARS and UI-TARS-2 (Qin et al., 2025; Wang et al., 2025a), OpenCUA (Wang et al., 2025b), ScaleCUA (Liu et al., 2025), EvoCUA (Huang et al., 2026), Fara (Microsoft, 2026). The authors state these open models remain largely GUI-centric and report strong results on relevant benchmarks.
• GUI and programmatic hybrids: CoAct-1, ComputerRL, UltraCUA, ToolCUA, and StateAct combine GUI with code, APIs, tool calls, or program state (Song et al., 2026; Lai et al., 2025; Yang et al., 2026b; Hu et al., 2026; Yang et al., 2026a). RecreationWorld and WeaveBench provide environments and benchmarks respectively (Bai et al., 2026; Li et al., 2026). Section 2 describes multi-agent hybrids as: an orchestrator selects an interface per subtask, with sub-agents' observation and action sets fixed.
• Qwen-UI-Agent (Zhou et al., 2026): The authors' own prior work, unifying GUI and CLI action spaces and supporting batched actions, with a mobile real-machine runtime. In this paper's formalization, the authors write it as a special case of GCUA rather than a separate definition.

Open infrastructure

• cua (trycua, 2026): Cross-platform desktop driver and sandbox SDK. This paper's macOS adapter uses its unmodified Cua Driver.
• Agent S series (Agashe et al., 2025a;b; Gonzalez-Pumariega et al., 2026): Open orchestration frameworks with memory, visual grounding, and increasingly code-based execution.
• Open Computer Use (iFurySt, 2026): A desktop control runtime wrapped in MCP, including macOS control via Accessibility. The authors state these works are complementary to this paper.

Baselines and benchmarks

• Baselines: GUI-only with batched actions; ablations also include single-action GUI, batched GUI+CLI, and code execution only. The benchmark is the 82-task offline subset of OSWorld-v2 v2.1 (Yuan et al., 2026). The real-machine portion is the authors' own four macOS tasks, not a published benchmark.

The authors use Agents' Last Exam's system hierarchy as a reference (Sun et al., 2026), but define GCUA as the model making three decisions per step, rather than merely whether it has five layers of components. The authors position this paper as an ongoing open full-stack project: this report tests the design with existing models, leaving trained models for later release.

The authors do not train new models; instead, they use prompts, tool interfaces, and an execution runtime to make existing frontier models exhibit two behaviors: code execution, and harness-to-model decision transfer (delegating selected observations and task-state decisions from fixed harness rules to the model).

Three Decisions and Four Interfaces

• Three decisions: Each step's policy jointly samples (q_t, p_t, m_t+1) ∼ π_θ(· ∣ m_t, o_t). q_t is the extra view, p_t is the program, and m_t+1 is the updated working state; m₀ contains the task instruction.
• Observations and actions: q_t is taken from screenshots and the AX tree and can be empty; CLI/API outputs are returned automatically. The statements of p_t call four types of primitives—GUI, AX, CLI, API—and can include loops, branches, and variables. A single primitive call is the simplest program.
• Four interfaces: GUI looks at screenshot pixels and operates on coordinates; AX tree reads the role, name, and value of window elements without bringing the window to the front or moving the user's pointer; CLI runs shell; API calls external services. The authors state that no single interface is suitable for every step.
• Special case: Batch GUI+CLI is written as three constraints: screenshot every step, p_t is a sequence of GUI and CLI primitives, and the working state is updated by fixed rules. GCUA removes these three.

Code Execution

• Implementation: The OSWorld-v2 configuration uses PyAutoGUI for desktop operations and bash() for running shell. The real-machine prompt additionally describes Playwright. Each step is one Python program; the harness extracts the code block, performs syntax checking, then executes it. Appendix C gives the full prompt.
• Feedback: Returns standard output, standard error, and execution status. When on-demand observation is enabled, the program calls screenshot() to attach a new screenshot in the next step. The real machine can also return the AX tree at the model's request.
• Five writing patterns summarized by the authors: combining across interfaces, loops, branching on intermediate results, organizing observations for the next step, and using assertions to check what should and should not change. Examples come from GPT-6 Astra trajectories on OSWorld-v2; the authors state they illustrate usage, are not mutually exclusive categories, and do not correspond to individual gains.
• Limits of visual feedback: The authors state that with screenshots alone, whether a button is enabled or whether an input box contains the expected value is not directly a program-readable value; checking them may require another model call. This is the reason the real-machine harness also provides the AX tree.

Delegating Harness Decisions to the Model

• On-demand screenshots: The model calls screenshot() to put a new image into the next step's input. The environment still captures an image every step; what changes is what the model receives, not what the environment records.
• Task ledger: todo.update() and todo.list() record progress, facts, and open questions. States include pending, attempted, changed_unverified, claimed_verified, blocked. The authors state these entries are the model's own judgments, not independent verification of the environment.
• intent: Before each program executes, the model writes an intent block describing what that step will do. The authors state this sentence can be shown to the user on the real machine. It is not a Python operation and does not override the user's task.
• Rules still in the harness: The scrolling screenshot buffer deletes old images in batches to limit visual context and reduce perturbation of the prompt cache. Micro-compaction performs a similar scrolling replacement for earlier text tool outputs. Appendix C also states: scripts run as new processes, timeout is 300 seconds, and printed output is truncated to 3200 characters.

Real-Machine Harness and Cross-Device SDK

• Differences from the benchmark: The authors list five: sharing control with the user; the screen changes continuously; requests evolve during execution; observation and control must target a specific goal; tasks start from an existing user environment. Background observability does not mean every operation can be completed in the background.
• Five layers: The model writes a program; a persistent interpreter runs it and assembles the next step's input; the Device SDK binds observations and actions to a device, window, or browser tab; the Driver translates them into accessibility actions or coordinate input and returns detectable typed errors; the application and operating system determine actual capabilities. The authors state that successful delivery of an action does not mean task-level effects have occurred.
• Low-disruption order: Prefer element-level accessibility actions, then window-oriented coordinate input, and finally foreground input. Coordinate with the user before foreground input. ask_user is used for clarification, login, or authorization.
• SDK: The same vocabulary of operations is implemented by platform adapters; the macOS adapter uses the unmodified Cua Driver. ui.ax and ui.vision are separate. Programs can inspect multiple targets, then use mu.emit to hand the selected content to the model. Platform differences remain in namespaces such as .android, .macos, .browser. The authors state that a phone, computer, and browser can be coordinated within one task, and give an illustrative flow, not measured results.

Experimental Setup

• Benchmark: OSWorld-v2 v2.1 offline subset, 82 tasks, excluding 26 tasks that require external network access; local web apps and desktop tasks are retained. The protocol follows the evaluation protocol used in the GPT-6 Astra release.
• Models: Claude Opus 5, GPT-6 Astra, Gemini 3.8 Flash, Qwen3.8-Max, Qwen3.8-Flash, Kimi K3, DeepSeek-V4.1-Flash, GLM-5.3-Flash.
• Baselines: The main comparison baseline is GUI-only with batch actions, i.e., GUI + batch in Table 2. Ablations also include single-action GUI, batch GUI+CLI, and code execution only.
• Metrics: partial score is the mean over tasks of the evaluator's partial score, on a 0–100 scale. success is the percentage of tasks with normalized score greater than 0.999999. One step is one model call. Output tokens include reasoning; K denotes 1,000 tokens. Cost is computed at public prices, including cache discounts.
• Aggregation: Each model is weighted equally; relative reductions are computed within each model first, then averaged. Within each run, the latest attempt with an evaluator result is kept, not the highest score (Appendix A).
• Repeats and real machine: The paper does not report the number of independent repeats for the OSWorld-v2 main comparison; several ablations are called selected runs. The real machine consists of four macOS tasks, with up to three recorded runs; Table 5 uses run 1. Complete is the number of tasks judged complete; the paper does not state who judged. The paper says seven real-machine configurations were studied, but Table 5 summarizes only five and does not give the names of the other two.

Main Results

The table below is from Table 2, OSWorld-v2 offline set. partial score is shown, and steps are the mean number of model calls per task.

Swipe horizontally to read the table

Model	GUI+batch score	OpenGCUA score	GUI+batch steps	OpenGCUA steps
Claude Opus 5	60.7	70.1	103.1	61.4
GPT-6 Astra	67.1	75.7	102.1	33.5
Gemini 3.8 Flash	50.1	67.0	166.8	101.2
Qwen3.8-Max	46.9	64.0	127.7	57.0
Qwen3.8-Flash	38.9	60.5	232.0	86.2
Kimi K3	40.7	59.2	160.1	81.9
DeepSeek-V4.1-Flash	28.0	57.7	167.3	84.9
GLM-5.3-Flash	29.6	64.9	190.8	75.5

• Summary: The Table 2 footnote states that the cross-model average partial score goes from 45.3 to 64.9, i.e., +19.6; steps decrease by an average relative 53.0%, and output tokens decrease by an average relative 27.2%. success ranges from 8.5%–36.6% for GUI+batch and 22.0%–46.3% for OpenGCUA, higher for all eight models.
• Authors' interpretation: The authors state the gains are not limited to low-scoring models and that the score gap between strong and weak models narrows by more than half. In Table 2 parentheses, GPT-6 Astra's step reduction is 67.2%, and GLM-5.3-Flash and DeepSeek-V4.1-Flash score gains are +35.2 and +29.8.
• Tokens and steps separated: Six models show decreased output tokens; Claude Opus 5 goes from 378.3K to 89.3K. GPT-6 Astra goes from 30.8K to 31.9K, and Gemini 3.8 Flash from 80.3K to 82.0K. The authors state that fewer steps need not reduce output tokens, and end-to-end latency must be measured separately; the main table does not report that latency.

Code Execution and GUI+CLI

• Four configurations: Figure 6 compares single-action GUI, batch GUI, batch GUI+CLI, and code execution on Qwen3.8-Max, DeepSeek-V4.1-Flash, and GLM-5.3-Flash. The authors state that in selected runs, code execution has higher scores and fewer steps. Relative to batch GUI, the authors report score increases of 15.5 to 33.6 and step reductions of 55.3% to 63.1%.
• Same interface plus control flow: The introduction states that relative to the GUI+CLI baseline, code execution increases the average partial score by 6.1 points and reduces steps by 14.8%. The main text gives Qwen3.8-Max steps going from 66.1 to 54.9 (stated as a 17.0% reduction) and GLM-5.3-Flash score from 55.8 to 63.2 (stated as +7.4). Section 3.1 also writes that Qwen3.8-Max's code execution is +5.2 points relative to the mixed baseline; the code-execution baseline score in Table 4 is 62.4.
• Full GCUA: Table 3 compares batch GUI+CLI and full OpenGCUA under the same GUI and CLI, adding code execution and model-initiated control flow. The authors state scores increase by 4.4 to 9.1 across four models and report fewer steps. Claude Opus 5 is 65.7 vs. 70.1, with steps 61.6 vs. 61.4.

Decision-Transfer Ablations

• Setup: Table 4 uses code execution as the baseline and adds screenshot, todo, intent, micro-compaction, and scrolling screenshot buffer to Qwen3.8-Max and DeepSeek-V4.1-Flash, then combines all. The authors state that except for todo, these mechanisms are mainly not intended to improve task scores.
• On-demand screenshots: cache hit goes from 75.6% to 86.1%, and from 61.9% to 76.3%; scores go from 62.4 to 64.3, and from 56.7 to 58.1. The authors state task scores did not decrease. Steps simultaneously go from 54.9 to 63.0, and from 74.5 to 90.0.
• Ledger and intent: todo brings scores to 64.0 and 58.5. intent brings scores to 61.5 and 53.3; the authors state a slight decrease, and its goal is to let real-machine users understand the program's purpose, not to improve benchmark scores.

Real-Machine Four Tasks

• Table 5: In the same selected run (run 1), cost and Complete both include unfinished tasks. GPT-6 Astra is 34.5 steps, 12.9K output tokens, estimated $18.290, 4/4. Qwen3.8-Max is 50.8 steps, 73.4K, $1.494, 3/4. The authors state the latter has 1.5 times the steps and 5.7 times the output tokens; reasoning averages 2.6K vs. 50.2K.
• Remaining three rows: Claude Opus 5 is 3/4, $17.117, 42.3K; Gemini 3.8 Flash is 2/4, $0.662, 31.9K; Qwen3.8-Flash is 2/4, $0.120, 75.7K. The authors state that the ordering of token usage and dollar cost need not be the same. DeepSeek lacks the hotel trajectory, and no summary is given.
• Within-task gaps and cases: In the workbook selected run, both satisfy the evaluation criteria; GPT-6 Astra is 22 steps and 8,783 output tokens (reasoning 1,487), Qwen3.8-Max is 37 steps and 120,831 (reasoning 92,974). PDF output tokens are 10,013 vs. 28,268. In Figure 11, the former fills 125 Level answer cells, the latter fills 85 and leaves 40 empty. In one screenshot step in Figure 10, reasoning tokens are 11 vs. 2,030. The authors attribute the gap to reasoning overhead on simple operations, real-machine observation targets, and using code to cover the whole task and check for omissions. The authors state that four tasks, one run per model, cannot establish a general ranking; the cases come from earlier low-effort trajectories, separate from Table 5.

Other Ablations and Analyses

• All mechanisms combined (last row of Table 4): Qwen3.8-Max is 64.0 score, 57.0 steps, cache hit 84.1%; DeepSeek-V4.1-Flash is 57.7 score, 84.9 steps, 70.5%.
• Scrolling screenshot buffer: cache hit reaches 91.6% and 75.3%; the authors state increases of 16.0 and 13.4 percentage points, and task scores did not decrease in these two comparisons.
• Micro-compaction: cache hit goes from 75.6% to 68.6%, and from 61.9% to 58.9%; the authors state task scores did not decrease.
• Single-action GUI vs. batch GUI (Figure 6 scores): 50.9 vs. 46.9, 29.5 vs. 28.0, 37.6 vs. 29.6, in the order Qwen3.8-Max, DeepSeek-V4.1-Flash, GLM-5.3-Flash. The paper does not explain why batch is lower.
• Figure 6 single-action steps: 175.0, 227.1, 187.1; batch GUI steps are 127.7, 167.3, 190.8. GLM-5.3-Flash's batch steps are higher than single-action.
• Spending and time (Figure 7): at about $0.38/task, Qwen3.8-Flash is 60.5; at about $15.26/task, GPT-6 Astra is 75.7. The authors state that as budget increases, the best-scoring model shifts from Qwen3.8-Flash to GLM-5.3-Flash, Gemini 3.8 Flash, and finally GPT-6 Astra. The time curve is called exploratory by the authors: bandwidth, number of concurrent environments, queuing, and tool latency are not controlled across models.

Limitations and future directions noted by the authors

• No newly trained model: Section 5.1 states the model-side contribution of this report is validating the design with existing models, not a newly trained model. Section 7 likewise states no new trained model is introduced.
• Harness and SDK are preliminary: Section 5.1 states the real-machine harness and cross-device SDK released alongside are preliminary versions.
• Next-version model: The authors plan to release GCUA models that unify computer and browser use, turning code-coordinated visual and programmatic operations, on-demand observation requests, and maintaining task state during execution into learned abilities, and strengthening real-machine scenarios (Section 5.1).
• Inference and execution efficiency: The authors plan to shorten inference for routine operations, mitigate overthinking, and improve code execution, context management, and infrastructure, while exploring other system-level approaches (Section 5.2).
• Work on user devices: The authors plan to extend the harness and SDK across computers, browsers, and possibly mobile devices. Priorities include target selection, recovery after execution failure, task continuity after user interruption or changed requests, and which workflow decisions should move from fixed rules to the model (Section 5.2).
• Real-machine benchmarks and timeline: The authors plan to build real-machine evaluation environments and benchmarks, especially macOS, covering more applications, user interactions, and changing device states, while measuring time and resources. Section 5.3 states these priorities have no fixed delivery timeline.

Experimental coverage

• Offline benchmark: The main comparison is eight models on the 82-task offline subset of OSWorld-v2 v2.1; this subset excludes 26 tasks requiring external network access (Section 3.3.1, Appendix A).
• Ablation scope: Code execution compares three models across four configurations (Figure 6); batched GUI+CLI vs. full GCUA compares four models (Table 3); decision transfer and context policies compare two models (Table 4).
• Real machine: Four macOS tasks, with task descriptions in Appendix B. The paper states seven configurations were studied, Table 5 summarizes five using run 1; each configuration has at most three recorded runs. There is also a low-effort case separate from Table 5.
• Judgment and repetition: Success is determined by whether the OSWorld evaluator's normalized score is greater than 0.999999. The paper does not report the number of independent repetitions for the main comparison, nor the reviewers for the real-machine Complete setting.
• Resource curves: Section 3.3.6 states the cost curves use public API prices, retrospectively impose a spending cap on each trajectory, and include unsuccessful attempts. API bandwidth, concurrent environment counts, provider queueing, and tool latency in the time comparison were not controlled across models.

OpenGCUA is an ongoing open computer-use project; this report tests the design with existing models and does not release a newly trained model.

• Problem: The authors want to expand computer use from "can the model click a GUI" to "can the system complete tasks on a real computer using visual and programmatic interfaces." GCUA is defined as the model deciding each step what to observe, how to act, and how to organize.
• Approach: The harness executes Python programs written by the model, so GUI operations and the shell share loops, branches, and intermediate results. The model decides whether a new screenshot is needed, how to track progress with todos, and writes an intent before execution; historical screenshots and old tool outputs are still managed in batches by the harness. The real-machine side is a five-layer harness and a cross-device SDK that binds operations to explicit devices, windows, or tabs.
• Offline main results: On the OSWorld-v2 offline 82 tasks with eight models, relative to batch GUI-only, the average partial score is +19.6 and steps are on average 53.0% fewer (Table 2). GPT-6 Astra scores 75.7 with 33.5 steps; GLM-5.3-Flash's score gain is +35.2. Success is higher for all eight models. Output tokens increase in two cases.
• Design decomposition: The authors state that with the same GUI and shell, writing programs with control flow still yields higher scores and fewer steps than batched GUI+CLI. On-demand screenshots raise cache hits for two models to 86.1% and 76.3%, while increasing steps. Intent decreases scores in Table 4.
• Real machine: Across four macOS tasks in run 1 of the same harness, GPT-6 Astra completes 4/4 and Qwen3.8-Max completes 3/4; the latter uses 50.2K reasoning tokens versus the former's 2.6K, which the authors give as 19.4 times. Estimated costs are $18.290 and $1.494 respectively.
• Authors' conclusion: The authors argue that execution style is not just a matter of whether an interface exists, and place future releases on GCUA models, shorter inference, and real-machine environments and benchmarks.