Long-WAM: Scaling the Context of World-Action Models

TL;DR: Long-WAM is a world-action model — a video-prediction model adapted into a robot controller — and its headline finding is a dependency most people would not have guessed: giving a robot policy a longer window of visual history only pays off if the underlying video model was pretrained autoregressively (next-chunk prediction) rather than bidirectionally. With an AR foundation, scaling context from 0 to 19.2 seconds lifts RoboCasa GR-1 success from 63.3% to 78.7%; with a bidirectional foundation (Wan2.2), the same context scaling does essentially nothing. Paired with an aggressive inference stack (107 ms per action chunk on an RTX 5090, including future-video prediction), the result is a policy that does things its baselines simply can't: 95% success on stacking a cup onto a moving cup, where both π₀.₅ and Fast-WAM go 0 for 20.

Why history is awkward for robot policies

A single camera frame tells you where things are but not where they're going. For dynamic manipulation — intercepting an object on a conveyor, tracking task progress across a 40-second multi-step task — the policy needs memory. The obvious fix is to feed in more frames.

But two problems bite. First, latency: more history means more tokens to encode, prefill, and attend over, and a controller that thinks for 400 ms about a cup moving at 7.5 cm/s misses the cup. Second, and this is the paper's central observation, giving a model access to history is not the same as the model having learned to use it.

That second point deserves unpacking, because the field has largely assumed it away. The dominant recipe for world-action models (WAMs) — systems like DreamZero and LingBot-VA that couple a video generator to an action head so the policy "imagines" the near future before acting — starts from a bidirectionally pretrained video generator. Bidirectional video diffusion models (Wan2.2, etc.) are trained to denoise all frames of a clip jointly, with every frame attending to every other frame, past and future. They learn superb visual priors. Then, during robot adaptation, a causal attention mask is bolted on so the model can run as a streaming controller: past frames predict future frames.

Long-WAM's claim: that bolt-on causality is where things quietly break. A bidirectional model never had to learn the temporal factorization — "given everything so far, what happens next?" — because during pretraining it could always peek at the answer. You can hand it 20 seconds of history at adaptation time, and it will dutifully attend over it, but it never developed the machinery to extract predictive signal from a past-only prefix. Access ≠ use.

The recipe

Long-WAM's pipeline has three stages, and the whole argument rests on keeping the causal structure intact end-to-end.

Stage 1 — Robot-domain AR video pretraining. They continue training LongLive-2.0 (an autoregressive video generator with a 16-second context) on ~10,000 "window-equivalent" hours of robot and egocentric video — RoVid-X, AgiBot World, EgoDex, EgoVerse, VITRA — with sequences up to 30 seconds. No action labels anywhere: supervision is purely next-video-chunk prediction via teacher forcing with block-causal attention, flow-matching loss \mathcal{L} = \mathbb{E}[w(\sigma)\|v_\theta(x^\sigma \mid h_{<i}, c) - (\bar\epsilon - z)\|^2] where h_{<i} is the clean history prefix and c the language prompt, plus an "error recycling" trick that injects buffered model errors into the context so the model learns to correct its own rollout drift. Because there are no actions, the data can mix embodiments freely — robot arms and human egocentric video in one pot. This cost ~30,720 H100-hours. (Note the accounting caveat the authors themselves flag: "10,000 hours" counts 16-second training windows, not unique footage.)

Stage 2 — Causal-to-causal action adaptation. An action expert is grafted on with an asymmetric interface: video tokens attend only to their own and earlier video blocks — never to actions — while action tokens read everything visual (history + partially denoised future). Training alternates two passes: video flow matching on future latents, then action flow matching conditioned on a detached visual cache built from ground-truth futures forward-noised to \sigma_\star = 0.9. The detachment means action gradients never touch the video expert, so the predictive prior from Stage 1 survives.

Stage 3 — Predict-then-act inference (IDM). At each decision, the video expert rolls out K_v future latent chunks from Gaussian noise but stops at \sigma_\star = 0.9 — i.e., futures are only ~10% denoised, and never decoded to pixels. Their KV cache is handed to the action expert, which denoises an H-step action chunk against it. The ablation (w/o V vs. CoD vs. IDM in Tables 1–2) shows this explicit imagine-first ordering matters: on LIBERO-Long, 99.5% (IDM) vs. 97.8% (co-denoising) vs. 94.5% (no future prediction at all). Prediction as an intermediate representation for control, at latent level, on the cheap.

The evidence: context scaling depends on pretraining

The key experiment (Section 5.3, Figure 6 of the paper) trains separate WAMs at each context length from 0 to 38.4 seconds, on three different video initializations, all under identical causal adaptation. The only variable is what the video backbone learned before it ever saw an action.

Both AR variants gain 11–12 points from 2.4 → 19.2 s of history; the bidirectional variant wobbles up and back down, ending where it started. The AR-vs-bidirectional gap grows with context: 3.3 points with no history, 17.1 points at 19.2 s. That's the signature you'd expect if the bidirectional model simply never learned to extract predictive structure from a past-only prefix — extra history is extra tokens, not extra information.

The decline at 38.4 s is worth noting and the authors' explanation is plausible: average training trajectories are 12.1 s, so at 38.4 s, 80.4% of sampled history frames are padding. This looks like a data-coverage ceiling, not a memory ceiling.

On absolute performance: Long-WAM at 19.2 s hits 78.7% on GR-1, 11.6 points over the best baseline (Cosmos Policy at 67.1%). On LIBERO-Long, 2.4 s of history already lifts 94.5% → 99.5%. RoboTwin 2.0 average is 94.4% (best among ~10 compared methods, though by only 0.3 points over ABot-M0.5 — that one's a wash). The standout benchmark number is DOMINO, which involves moving objects: 34.9% success vs. 19.9% for Fast-WAM — a 15-point jump where motion cues from recent history genuinely matter.

Real robots, real time

Imagining the future before every action is expensive, and the systems half of the paper is about not paying for it in wall-clock time. The pieces: W4A4 NVFP4 quantization of the video expert (actions stay BF16 — quantizing them hurt control precision for little latency gain), CUDA Graph replay to amortize the launch overhead that quantization kernels introduce, within-call KV reuse, a neat online-softmax trick that merges attention over separate video and action KV buffers without concatenation, a streaming causal VAE that encodes observation frames as they arrive instead of re-encoding the full window after the inference trigger, and asynchronous execution where inference overlaps robot motion. Net result: 107.4 ms per action chunk on RTX 5090 (3.3× over BF16 eager), 328 ms on DGX Spark, 379 ms on Jetson Thor — all including future-video prediction and the full VAE.

Two deployment results stand out. First, asynchronous robustness: when you switch from synchronous to asynchronous chunk handoffs, Fast-WAM loses 15.4 points and LingBot-VA loses 45.5, while Long-WAM loses 0.2 — with no blending or prefix guidance. The claimed mechanism is that because consecutive decisions are conditioned on temporally continuous AR forecasts, consecutive action chunks naturally agree in their overlap (overlap RMSE roughly a third of Fast-WAM's). That's a second, independent dividend of the causal foundation.

Second, the dynamic manipulation results on a Unitree G1:

Zero-vs-95% gaps are rare in robot learning papers and this one is well-documented (Appendix H shows the failure modes: baselines grasp the stationary blue cup fine, then miss the moving green cup). On a YAM bimanual arm, Long-WAM also sustains 40+ second tasks at 81.7% average — so it's not a pure speed demon.

There's also a hierarchical result that I suspect will travel: using the unchanged Long-WAM checkpoint as the executor under a GPT-6 Astra planner on RoboCasa365 lifts overall success from 31.4% to 54.4%, and unseen skill compositions from 6.1% to 35.0%. The planner gains more from a strong executor (+23.0 points) than from π₀.₅ (+13.6) — evidence for the "memory-informed executor + semantic planner" division of labor, where neither component alone comes close (planner alone: 25.2%).

What to believe, and what to poke at

The causal claim is confounded, and the authors half-know it. The clean reading — "AR pretraining causes long-context gains" — rests on comparing Wan2.2 vs. LongLive-2.0 vs. LongLive2.0-Robot. But these differ in training data, recipes, context lengths seen during pretraining, and likely scale, not just attention directionality. LongLive-2.0's pretraining explicitly includes 16-second windows and error recycling; Wan2.2 was built for short clips. A bidirectional model simply may never have seen long temporal dependencies, regardless of masking. The ideal control — same data, same compute, flip only the attention mask — doesn't exist here. The fact that generic AR (not just robot-domain AR) scales similarly is suggestive but not decisive. Treat the finding as a strong, actionable correlation: if you want long-context WAMs, start from a long-context causal video model.

Each context length is a separately trained model. There's no single policy that adapts its window; the scaling curve is a family of models. The authors flag this and propose test-time adaptive windows as follow-up.

Context is expensive. 19.2 s of history costs 341 ms per chunk even on an RTX 5090 (vs. 74.6 ms with none), and over a second on edge hardware before optimization. The real-robot results use 2.4 s contexts; the 19.2 s wins are simulation-only. The "scaling the context" title slightly outruns the deployed configuration.

Baseline fairness in the dynamic tasks. π₀.₅ and Fast-WAM going 0/20 is dramatic, but the paper doesn't detail whether all three got equivalent dynamic-task demonstration data on the G1 (for DOMINO they explicitly fine-tuned on dynamic data; for the real-world tasks the training setup for baselines isn't spelled out in the extracted text). The 15-point DOMINO margin over Fast-WAM — same fine-tuning regime — is probably the more trustworthy dynamic-manipulation signal.

Narrow margins elsewhere. RoboTwin 2.0 and LIBERO leads are ≤1 point over the strongest baselines — the method is competitive, not dominant, on short-context static benchmarks, which is actually consistent with its own thesis: history should matter most when something is moving or the task is long.

Where to spend your time

Section 5.3 and Figure 6 are the heart of the paper — the context-scaling curves across the three initializations are the finding. Section 3.2 plus Appendix B.2 give you the IDM mechanism in enough detail to reimplement (the \sigma_\star = 0.9 partial-denoising trick and detached visual cache are the non-obvious bits). If you care about deployment, Table 7 (async robustness) and the cumulative-optimization Table 8 are a compact engineering case study. Appendix E's smoothness-vs-decision-frequency analysis — where minimizing chunk discontinuity does not maximize success — is a small gem for anyone building asynchronous chunked controllers.

The broader takeaway, if the causal story holds up under cleaner controls: the field's default of adapting bidirectional video diffusion models into robot policies may be leaving the main benefit of video pretraining — learned temporal prediction — on the table. For anyone training video foundations intended for control, that's an argument worth the price of the ablation.