Qwen-CUA: Native Computer Use for (almost) Everything

TL;DR: The Qwen team trained a 397B-parameter (17B active) mixture-of-experts model to operate a computer the way a human does — screenshots in, mouse and keyboard events out, no DOM, no accessibility tree, no APIs — and it scores 86.2 on OSWorld-Verified, beating GPT-5.5 (78.7) and Claude Opus 4.8 (83.4) while using roughly 6× fewer output tokens than Opus. The interesting part isn't the score; it's the training factory behind it: ~100,000 vCPUs of cloud desktops, ~40,000 automatically verifiable tasks, terminal-reward RL over multi-hundred-step trajectories with a clever "trajectory slicing" trick, and an iterative loop that refreshes SFT data and re-calibrates RL task difficulty between runs. The main thing missing is any ablation that separates the recipe from the sheer scale.

Why screenshot-only is the hard road, and why it might be the right one

There are three ways an agent can act on software: write code, call APIs, or push pixels around like a human. The first two are where LLMs already shine, but they only cover software that exposes a programmatic surface. A huge fraction of digital work — legacy desktop apps, CAD tools, Blender, hospital EHR systems, whatever your bank's website did to itself last Tuesday — is only reachable through a GUI.

The "cheat" that most GUI agents use is to read the DOM or accessibility tree: a machine-readable skeleton of the interface. It makes grounding trivial but breaks the moment you leave the browser, and it silently leaks hidden state that no human user could see. Qwen-CUA refuses the cheat entirely. Every observation is a screenshot; every action is one of a small schema of native input events (left_click(x, y), type, scroll, key, plus control actions like wait and call_user).

This makes three problems brutal at once:

1. State tracking. GUI state is partially observed. Did the file save? Is that dropdown still open? The evidence lives in a history of screenshots, and screenshots are expensive — a long task can blow through any context budget.
2. Experience is costly. Every training rollout consumes a real, stateful virtual desktop for minutes at a time. You can't just sample tokens; you have to run computers.
3. Reward is sparse but verifiable. You usually only know whether the task worked by inspecting the final environment state. That's a single scalar at the end of a 100+ step episode.

The paper's contribution is a coherent answer to all three, executed at industrial scale.

The one context trick: chunked screenshot folding

The scaffold keeps up to 20 screenshots "live" in the context. When step 21 arrives, a naive sliding window would drop screenshot 1 — but that rewrites the front of the prompt on every step, destroying KV-cache reuse and discarding early evidence unpredictably. Instead, Qwen-CUA advances a fold boundary by 10 screenshots at once: screenshots 1–10 are replaced by a fixed text placeholder (<image collapsed>), while the reasoning and actions from those steps stay in the conversation verbatim. Steps 21–30 then extend a stable prefix; the boundary only jumps again at step 31.

The elegant move is reusing this exact fold operator for RL. A long trajectory that overflows the 144K-token training context is rendered as multiple slices by advancing the boundary; each slice inherits the parent episode's terminal reward, and only the model-generated reasoning/action tokens in the active window contribute to the loss. No learned summarizer, no hand-designed step-level rewards, and — crucially — training and inference see identical folded representations. Sparse episode-level feedback gets reused densely across the whole trajectory.

The training factory

The data and infrastructure section (§3) is where most of the paper's effort lives.

Environments. An Alibaba Cloud ECS fleet with access to ~100,000 vCPUs supports tens of thousands of concurrent isolated desktops. Environments span self-built mock web services (clones of popular products with programmatic state injection/inspection/reset, so one app spawns thousands of reproducible tasks) plus real desktop software across office, creative, scientific, and engineering domains.

Tasks: ~40,000, all verifiable. Three flavors: (1) environment operation tasks sampled from per-app feature taxonomies, each pairing a natural-language goal with a reproducible initial state and an executable evaluator; (2) user-interactive tasks where a simulated user holds information the agent must ask for — asking only pays off if it leads to a correct final state; (3) long-horizon workflows built by phase-state chaining: each phase has a verifiable completion state which, once validated, is serialized as the initial state of the next phase. That last idea is a nice engineering pattern — you can audit a 200-step workflow phase by phase, then chain the phases back into one end-to-end task.

Human data. For behaviors that RL won't discover (Blender, CAD), annotators complete realistic workflows on personalized desktops. Since annotators don't narrate their thinking, a model reconstructs step-level rationales conditioned on the observed action, filtered for consistency — action-grounded chain-of-thought at scale.

RL. Standard RLVR skeleton: for each task, sample G=16 valid trajectories (oversampling 20), score each terminal state with the executable evaluator r_i \in [0,1], compute a group-relative advantage \hat A_i = (r_i - \bar r)/(\sigma_r + \delta) shared by every token in the trajectory. The optimizer is SAPO: instead of PPO's hard clip on the importance ratio \rho, the ratio passes through a smooth sigmoid gate whose gradient weight is 1 exactly on-policy and decays continuously off-policy — a soft trust region. The gate temperature is asymmetric (\tau_{\text{neg}} = 1.05 > \tau_{\text{pos}} = 1.0): negative-advantage updates are attenuated faster, because pushing probability mass away from a token spreads it over the whole vocabulary and destabilizes long multimodal MoE training. The final RL run: 512 H200s, ~5 days, ~61,440 GPU-hours, validation score 0.734 → 0.770.

The iterative loop. This is the part I'd bet matters most and is least analyzed. Each trained model diagnoses its own frontier: failed SFT queries get re-demonstrated by updated teachers, weak domains get targeted data, and — my favorite detail — the RL task pool is re-calibrated by running 8 trial rollouts per candidate and keeping only tasks where 0 < c(q) < 8 successes. Tasks the policy never solves give no gradient; tasks it always solves give none either (mean-centering zeroes the advantage). This keeps RL compute concentrated exactly at the edge of competence, and it recomputes that edge every iteration. Notably, each SFT run restarts from the same mid-training checkpoint rather than fine-tuning the previous agent — improvements transfer through curated data, not inherited weights, avoiding optimization drift.

The evidence

The pattern across all eight benchmarks: enormous jumps over the Qwen3.7 base (OSWorld-Verified +12.9, ScienceBoard +29, OSWorld 2.0 binary completion 2.5 → 18.5), state-of-the-art on OSWorld-Verified and MacAgentBench, and rough parity with GPT-5.5/Opus elsewhere. Opus 4.8 still wins on OSWorld 2.0 (20.3/54.8 vs 18.5/48.4), MyPCBench, WebArena, and Gym-Anything by small margins. Scaling the identical recipe to a >1T-parameter backbone (Qwen-CUA-Max) lifts OSWorld-Verified to 87.6 and OSWorld 2.0 to 21.2/53.3 — the recipe hasn't saturated with model size.

The efficiency numbers rebut the "it just thinks longer" hypothesis: on OSWorld-Verified, Qwen-CUA hits 86.2 with 3,606 output tokens per task, while Opus 4.8 gets 80.0 at a similar budget and needs 21.8K tokens to reach 83.3. That's a genuine capability-per-token gap, not verbosity. (Turn counts go the other way — 218.9 turns/task on OSWorld 2.0 vs 83.5 for GPT-5.5 — but the comparison is confounded: proprietary interfaces batch multiple actions per turn while Qwen-CUA emits one native action per turn.)

The hybrid-tool experiment is a candid negative-ish result worth knowing:

Bash cuts turns by ~23% but drops success 3.6 points — the model hasn't learned when to switch modalities. The authors frame this as an optimization gap and pitch the endgame explicitly: native computer use as the universal grounding-and-fallback layer of a hybrid agent, with code and APIs handling whatever they can express directly. That framing — pixels as the fallback, not the whole strategy — is probably the paper's most durable idea. (There's also a prompt-injection robustness result on RedTeamCUA, ASR 36.6 → 16.4, if that's your thing.)

What to be skeptical about

No method ablations. This is the big one. The triage question — can you separate the recipe from 397B/1T-scale compute? — goes unanswered. There is no comparison of SAPO vs. GRPO/PPO-clip, no folding-vs-sliding-window ablation, no trajectory-slicing on/off, no controlled measure of what the iterative refresh buys over one big run. Figure 4(b) shows scores rising across development checkpoints, but the authors themselves warn the slopes aren't interpretable since data, teachers, and RL tasks all change between runs. The final RL stage's own contribution is measurable and modest (0.734 → 0.770 on their validation mix); most of the gain evidently comes from the SFT data flywheel plus scale, in unknown proportions.

Reproduction caveats. Some proprietary baselines were reproduced by the authors under settings they chose (GPT-5.5 at 8K token limit on Gym-Anything vs 32K for others). Gym-Anything covers 97 of ~200 environments, with 100 omitted for infrastructure reasons. On MacAgentBench, 12 clock tasks scored zero due to an apparent evaluator bug that the authors kept in the official score — honest, but it means several benchmark numbers carry noise in both directions.

Cost. The final RL run alone is ~61K H200 GPU-hours, on top of ~100K vCPUs of environment fleet and unspecified SFT compute across multiple iterations. This recipe is not reproducible outside a handful of labs, and "scalable verifiable interaction" here means scalable if you own a cloud.

If the result holds, the practical takeaway is that screenshot-only agents are no longer a handicapped academic setting — they're competitive with (and more token-efficient than) the best proprietary agents that grew up with richer interfaces, and the bottleneck has shifted from perception/grounding to verifiable task supply and tool routing. The sections worth your time: §3.2 (task synthesis, especially phase-state chaining) and Appendix C (the full SAPO objective and slicing mechanics — concrete enough to reimplement).