sii-research/tau-0-vla

TL;DR: τ₀-VLA is a hierarchical robot foundation model for long-horizon manipulation that borrows the "spend more inference compute when the problem is hard" playbook from LLMs: a memory-augmented high-level policy proposes subtasks and uses a learned world model to search over alternatives at test time, while a generalist flow-matching low-level policy executes across embodiments. The low-level stack (Qwen3.5 VLM + Mixture-of-Transformers action expert, trained on 40,115 hours of real robot data) is released with training, serving, and eval code — but the interesting part, the high-level world-model search, is not yet in the repo.

Why this matters

Test-time scaling has transformed LLM reasoning, but manipulation policies still mostly run a fixed forward pass per control step. Using a world model to score and search over candidate subtasks is a plausible way to buy robustness on long-horizon tasks without retraining — the robot "thinks harder" only when it needs to. Combined with a 40k-hour heterogeneous training corpus (in the same regime as π0-style generalist policies, and larger than most published figures), this is one of the more credible open attempts at an inference-compute axis for robotics.

How it works

Two levels:

• High-level policy: memory-augmented, emits the next subtask given task context and history. When "additional reasoning is needed," it generates alternative subtasks and uses a world model to evaluate them before committing — i.e., world-model-guided search rather than greedy decoding. The README gives no detail on the world model's architecture, the trigger for extra computation, or the search procedure; that's in the paper.
• Low-level policy: a Qwen3.5 vision-language backbone feeding a Mixture-of-Transformers action expert trained with conditional flow matching (the now-standard continuous-action recipe from π0). Actions and proprioception live in a unified 40-dimensional state/action space so one policy spans embodiments. Trained with multimodal co-training on 40,115 hours of heterogeneous real-world data.

What's actually released

• Code (Apache-2.0): post-training pipeline, LeRobot v3.0 data loading, embodiment adapters with a template for new robots, a policy server, and open-loop evaluation.
• Weights on Hugging Face (linked from the README), also Apache-2.0.
• Example data: a small AgiBot World subset with a matching fine-tuning recipe.
• Not released: the high-level policy and world model. An Aug 19 note says components will be released "progressively" — so today this repo is effectively a strong π0-class low-level VLA, not the full hierarchical system.

The README contains zero quantitative results — no benchmark numbers, no success rates, no ablation of the test-time search. The evidence for the headline claim lives entirely in the paper/project site. The "40,115 hours" figure is impressive if accurate, but data provenance and composition aren't described here.

Caveats and maturity

• Reference environment is Python 3.11 / CUDA 12.8 / PyTorch 2.7.1; expect to need a serious GPU for post-training (no hardware guidance given).
• Serving v1 supports joint-control checkpoints only — end-effector control works for training data but not deployment yet.
• Open-loop evaluation only in deploy/; no simulation benchmark harness bundled.
• The novelty claim rests on the unreleased high-level component. Until it ships with real-world ablations (search on vs. off), treat "world-model-guided test-time computation" as a paper claim, not a reproducible artifact.

Trying it

Fine-tune on the bundled AgiBot World example, then serve and eval:

git clone git@github.com:sii-research/tau-0-vla.git
cd tau-0-vla
bash scripts/setup.sh

bash scripts/train.sh configs/example_agibot_world_gong/train.yaml \
  --model_name_or_path /path/to/tau-0-vla-checkpoint

python -m deploy.server --model outputs/
python deploy/openloop.py --ckpt outputs/ --no-plot

Watch the repo for the high-level policy drop — that's when this becomes testable as a claim about inference-time scaling in robotics rather than another capable generalist executor.