Hugo-AGI/RoboDawn

TL;DR: RoboDawn drives bimanual robots with a frozen frontier VLM (no robot training, no fine-tuning) through a ~10-command discrete grammar (move, rotate, gripper, point…) in a closed perception-action loop, and beats VLA policies post-trained on the full benchmark: 73.6% on RoboTwin 2.0 with one in-context demonstration vs. 46.0% for π0.5 trained on all 50 tasks. The entire evidence chain — code, prompts, seeds, 128 demos, and all 710 evaluation episodes replayable turn-by-turn — is public, which makes this a rare checkable challenge to the task-specific VLA training paradigm.

Why it matters

The dominant recipe in robot learning right now is: take a VLM backbone, bolt on an action head, post-train on thousands of demonstrations per benchmark. RoboDawn's claim is that if you give a strong-enough frozen VLM the right interface — discrete, human-intuitive commands, each executed as one complete planned motion, with visual feedback after every command — the reasoning already in the model transfers to manipulation. One demonstration in context substitutes for the entire training set. If this holds up, it reframes VLA post-training as compensating for a bad interface rather than adding manipulation ability.

How it works

The loop is simple. Each turn the VLM sees camera views, robot state, the instruction, and its accumulated memory/feedback; it reasons (thinking mode required) and emits a few commands like left move z -5 or right gripper close. Each command is expanded by a motion planner into a full trajectory, executed, and the result (new images, execution report) goes into the next turn. A task-independent "command primer" (6 images) teaches the grammar; optionally one expert demonstration of the task (up to 48 frames) sits in context.

The evidence

RoboTwin 2.0 C2R (50 tasks × 10 seeded episodes, domain-randomized scenes):

On RoboDojo (42 tasks, 5 dimensions), GPT-6 Astra one-shot hits 47.17% success / 54.63 score vs. 19.34% for the best full-set-trained baseline (DM0.5) — and it even zero-shots the 8 "Open" tasks at 62.5%. Notably, the same GPT-6 Astra scored only 22.58% under RoboDojo's own eval harness: the interface design, not the model, accounts for the gap. Ablations back this up: removing reasoning drops Gemini zero-shot from 47.0% to 34.8%, removing grid localization to 32.4%. Demonstrations saturate quickly (1 demo: 62.2%, 8 demos: 62.7%), and success scales with the interaction budget (60 → 240 turns: 31.2% → 47.2%), a test-time-scaling result for manipulation.

Real robots: zero-shot with Gemini 3.8 Flash — 9/10 block-in-basket and 5/10 stacking on a Franka, but 0/10 cloth folding on a Piper. The discrete world-frame grammar clearly doesn't cover deformables, and Precision is the weakest sim dimension (25%). Real-robot evidence is thin; the repo ships only a skeleton (--mock dry run), not the lab setups.

Caveats

• It is slow. ~9.7 s inference per decision with Seed-2.1-Pro, inference/motion ratio 4.65 (vs. 0.037 for π0.5). Episodes take 10–30 minutes of wall clock. This is not a deployable controller; it's an existence proof.
• Frontier-model dependent, with huge variance: GPT-6 Astra 73.6% vs. GPT-5.6 Luna 14.4% on the identical harness. Your results depend entirely on which API you point it at, and API cost will be substantial (up to 58 images per request, 45–240 turns per episode).
• The headline comparison (one-shot in-context vs. full-set-trained VLAs) is not the official protocol; the README is upfront about this and reports zero-shot too (53.2%, still above π0.5's 46.0%). The replayable episode logs at robodawn.top/results, failures included, are the right way to sanity-check success criteria.
• Fresh (Sept 2026, 58 stars); simulator setup is heavy (SAPIEN+cuRobo for RoboTwin, Isaac Sim 5.1 for RoboDojo). MIT license.

Try it

git clone --recurse-submodules https://github.com/Hugo-AGI/RoboDawn.git
cd RoboDawn
# install the RoboTwin submodule env per its README, then:
conda activate RoboTwin
PYTHONPATH=$PWD python harness/run_robotwin_eval.py \
  --task place_empty_cup --episodes 10 --model gpt-6-astra \
  --api_base https://<endpoint>/v1 --api_key_file ~/.config/robodawn/key \
  --output results/rt2/gpt-6-astra/place_empty_cup/shard_0

Or skip the simulator entirely and browse the 710 replayable episodes at robodawn.top/results — the fastest way to judge whether the successes look legitimate.