RoboTTT: Context Scaling for Robot Policies

TL;DR: RoboTTT takes the test-time-training (TTT) idea from long-context language modeling — replace attention over history with a small "fast" network whose weights are updated by gradient descent at every timestep — and grafts it into a vision-language-action robot policy (GR00T N1.7). The result is a policy that conditions on up to 8,000 timesteps (~4.5 minutes at 30 Hz) of its own visuomotor history at constant per-step inference cost. That long context is not decorative: it enables one-shot imitation from a single in-context human video, a "DAgger Distillation" trick that teaches the policy to correct its own mistakes online, and the first evidence that pretraining context length is a scaling axis for robot policies (8K-context pretraining beats 1K by 63% in closed-loop task score).

Why robot policies are amnesiacs

Nearly every state-of-the-art robot foundation model — the π-series, GR00T, OpenVLA, and friends — looks at the current camera frame (or the last 2–8 frames) and emits an action chunk. That's it. The policy has no memory of what it did thirty seconds ago.

For short pick-and-place tasks this is fine, and it's fine partly for a subtle reason: naively appending history often hurts. Past observations implicitly encode the robot's past actions, so a policy trained on demonstrations learns spurious shortcuts ("my gripper was moving left, so keep moving left") that break in closed loop. The paper reproduces this: GR00T with one history frame scores 39.5% on their car-assembly task versus 57% for the same model with no history.

But long-horizon manipulation genuinely needs memory. Consider a ten-stage toy assembly: install a gear on the left side, flip the chassis, install a gear on the right side. The two sides look nearly identical. A memoryless policy suffers state aliasing — it literally cannot tell which stage it's in from the current frame — so it skips stages or repeats them. Memory also matters for using in-context demonstrations, for recovering from perturbations, and for acting on objects that are currently occluded but were visible earlier.

The obvious fixes each fail on one axis. A Transformer over the full history has the capacity but its per-step cost grows with context — untenable for a 30 Hz controller running for five minutes (9,000 steps). A classic RNN (LSTM) has constant cost but compresses everything into a fixed vector, and empirically scales worse than attention. So the design problem is: a recurrent state that is expressive enough to compress dense, repetitive robot video streams, cheap enough for real-time control, and trainable to actually use what it stores.

The key idea: memory as a tiny model you keep training

Test-time training answers this by making the recurrent state a set of fast weights: the parameters of a small neural network (here, a two-layer MLP per DiT layer, ~10M params each) that is updated by gradient descent during inference, one step per timestep.

The mechanics are a reinterpretation of attention. Tokens are projected into queries, keys, and values (Q_t, K_t, V_t) as usual. But instead of storing all past keys and values and attending over them, you train a little function f_W to predict values from keys. At each timestep:

then read out with the query: O_t = f_{W_t}(Q_t). Update, then apply. The fast weights are an associative memory; the gradient step writes the current timestep into it; the query reads from it. All past context lives inside W, so per-step cost never grows.

The crucial twist is that everything about this inner loop — the projection matrices, the initialization W_0, even the inner learning rate \eta — is meta-learned by the outer imitation loss, via gradients through the inner gradient steps. The model doesn't just get a memory; it learns what to write into it and what to read out of it so that action prediction improves. This is the answer to the spurious-correlation problem above: rather than hand-designing which parts of history to keep, the fast-weight update learns to retain what's salient and discard what's redundant in a stream where 95% of frames look alike.

Wiring it into a pretrained VLA without breaking it

RoboTTT is instantiated on GR00T N1.7: a VLM backbone feeding a 16-layer Diffusion Transformer (DiT) action head trained with flow matching. Three architectural choices matter.

TTT goes across time, attention stays within time. A TTT layer is inserted after the attention layers in each DiT block. Attention operates only on the tokens of a single timestep (register tokens, proprioception, noised actions, cross-attending to that step's vision-language tokens). The concatenated per-step outputs then flow through TTT along the time axis. This keeps the expensive VL tokens out of the recurrence — instead, 16 learned register tokens per timestep act as a summary bus, attending to everything and carrying condensed VL information into the temporal channel.

Gated insertion. To avoid clobbering the pretrained policy, the TTT output is gated: O = \tanh(\alpha) \odot O_{\mathrm{TTT}} + O_{\mathrm{attn}}, with \alpha initialized near zero. At the start of training the model is exactly GR00T; TTT's contribution grows only as it proves useful. (Total parameter cost: 538M → 690M.)

Two training-recipe tricks make long sequences feasible. First, sequence action forcing: the flow-matching noise level \tau_t is sampled independently for each action chunk in the sequence, rather than one shared level. Sharing a noise level makes whole sequences uniformly easy or hard and destabilizes training — the ablation shows removing this cripples the policy. Second, truncated BPTT: gradients flow only within fixed-length segments, but the fast weights themselves carry across segment boundaries. GPU memory scales with segment length, not sequence length, so training context can grow arbitrarily. The subtle cost: W_0 and the update dynamics only receive gradients through the first segment / within-segment steps, yet the scaling results suggest this is enough.

Pretraining runs on bimanual robot data plus egocentric human video for 30K steps on 16 GB200s, with context curriculum up to 8K timesteps; post-training on each task uses 1K context. Notably, only the new TTT layers are tuned during pretraining. Deployment runs at 30 Hz on a single RTX 5090 — the per-step gradient update on a 10M-param MLP is cheap.

The evidence

The evaluation is three real bimanual assembly tasks on a YAM setup: Pup Go Car (toy car assembly with a drill, ~2 min episodes), Circuit (assemble 2–3 electronic components in a specified order; 80 configurations, train on 20, test on 60), and Gear Bot (ten stages, ~5 minutes). Baselines: GR00T N1.7 (single-step), GR00T with one history frame, and GDN — the same architecture with TTT layers swapped for Gated DeltaNet, a strong linear-recurrence memory that updates its state without gradient descent. This is the right control: it isolates the update rule while matching parameters, placement, and gating.

On rubric-based task completion, RoboTTT averages 79% versus 42% for single-step GR00T (+87% relative) and 56% for GDN. On Gear Bot, no baseline ever finishes; RoboTTT does 2/10 times. The qualitative failure modes of baselines are exactly the state-aliasing story: skipping visually similar stages, or proceeding as if a failed step had succeeded.

The scaling result is the headline for anyone thinking about robot foundation models. Pretraining the same model at context lengths from 128 to 8K timesteps, closed-loop performance climbs steadily with no saturation, while GDN shows no trend at all:

Why does TTT scale where GDN doesn't? The paper's hypothesis is that a gated-delta linear associative update has no meta-learnable inner loop: there's nothing for longer training sequences to shape. TTT's fast weights are updated by gradient descent, and the outer loss meta-learns W_0 and the update dynamics — longer training sequences mean more inner steps for that meta-learning to optimize over. If this interpretation holds, it's a genuinely useful finding about why expressive, gradient-based recurrent memories differ from linear ones, beyond language-modeling perplexity comparisons. Note also the intriguing detail that RoboTTT below 1K context underperforms — because rollouts are longer than training sequences, the fast weights drift into an update regime never seen in training. Context length must cover the deployment horizon.

Two capabilities that only exist with memory

One-shot imitation from human video. Because fast-weight updates are decoupled from the imitation loss (you can mask the loss on any span, making it pure context), the model can be trained on sequences of the form [human video of the task] + [robot trajectory], with loss only on the robot part. At test time, show it a single human video of an unseen Circuit configuration — the language prompt is identical across configurations, so the video is the only signal — reset the scene, and the robot reproduces the assembly. RoboTTT: 6/10 successes, 65% score. GDN: 0/10, 33% — it encodes the video into its state but can't retrieve and act on it. This gap between storing context and using it is the paper's most interesting empirical distinction.

DAgger Distillation. Standard DAgger collects rollouts where a human intervenes to correct the robot's mistakes, then fine-tunes on the corrections and throws away the robot's bad actions. The insight here: those bad actions are exactly what tells the model which failure each correction fixes. So: put the full interleaved trajectory (failures included) into the fast weights as context, but compute the imitation loss only on the human corrections.

From the same 100 DAgger trajectories, standard DAgger fine-tuning improves the sequence models by 13%; DAgger Distillation improves RoboTTT by 36% (GDN by 29%). And a clean control: fine-tuning GR00T on the full trajectories including robot actions performs identically to corrections-only (57% both ways) — the failures' value is purely as context. The authors frame this as Algorithm Distillation for robotics: the improvement process itself, not just the improved behavior, gets compressed into the fast-weight dynamics. At deployment, the robot misses a screw with the drill, raises its arm, re-aligns, misses again, adjusts, and succeeds — with no human in the loop. Perturbation recovery follows the same logic: when a human removes an installed part, RoboTTT recovers in 15/20 (roof) and 18/20 (tire) trials versus at most 10/20 for short-context baselines, though GDN is competitive here (13 and 18), suggesting within-episode robustness needs less memory expressiveness than cross-episode conditioning.

What to be skeptical about

The evidence is real-robot and multi-faceted, but narrow in ways worth naming. Everything is three assembly tasks, in one lab, on one bimanual platform, with 10–20 trials per condition and rubric scoring by the authors — no confidence intervals survive at these sample sizes (2/10 on Gear Bot is suggestive, not conclusive). The "scaling law" is one curve, averaged over three tasks, for one backbone; calling context length "a new scaling axis" is a hypothesis these data support, not establish. There's also no full-attention long-context baseline — understandable (latency would preclude deployment), but it means we can't see how close TTT's compression comes to the attention ceiling. GDN is a strong control for the update rule, but only one point in the linear-recurrence design space. The main-results figure includes DAgger training for Pup Go Car while the scaling figure predates it, which makes cross-figure comparisons slightly slippery. And training cost is nontrivial: sequence training over 8K-timestep trajectories on GB200s is not a hobbyist recipe, though TBPTT at least bounds memory.

None of this undermines the core claims; it just means the generality question flagged in the abstract — does this hold beyond dexterous tabletop assembly? — is genuinely open.

Why this matters

If the result holds, the takeaway is architectural: robot memory should be a learned, gradient-updated compressor, not a token buffer or a linear recurrence. The TTT framing also collapses several previously separate robotics problems — in-context imitation, online adaptation, failure recovery, long-horizon state tracking — into one mechanism: everything is just context flowing into fast weights, with loss masking deciding what's demonstration and what's target. DAgger Distillation in particular is a recipe anyone with intervention data can try on any sequence-model policy (it helped GDN too), and it extracts substantially more from the same data by treating failures as features rather than garbage.

Most worth your time: Section 3.3 and the DAgger Distillation results (Fig. 10–11) for the genuinely new training idea, and Fig. 8 for the context-scaling curve that motivates the whole enterprise. Section 2 is a compact, readable primer on TTT if the fast-weights formalism is new to you.