real-stanford/transformer-transformer

TL;DR: Transformer Transformer (Ha, Liu, Song — Stanford/Columbia) is a diffusion transformer that takes a manipulation demonstration and generates a complete robot — links, joints, motors, inertial properties — plus the control to execute that motion. The co-design framing is genuinely new: instead of learning a policy for a fixed embodiment, the model treats the embodiment itself as part of the output. They fabricated one of its designs (a modified bimanual ALOHA for cloth flinging) and got a 73% reduction in tracking error and 30% lower peak joint speed versus stock ALOHA. The full stack is open: tokenization, procedural robot generation, data gen, training, baselines, checkpoints.

What it is and why it matters

The pitch is "not all robots are created equal — design one for the task at hand." Robot co-design (jointly optimizing morphology and control) has a long history dominated by evolutionary search per-task, which is slow and doesn't amortize. This work amortizes co-design into a single generative model: condition on a demonstrated motion, sample a robot optimized for it. The unifying trick is RoboTokens — a tokenization that puts robot embodiments (from MuJoCo models), states, and actions into one sequence vocabulary, so a single transformer can do cross-embodiment control, hardware generation, or both jointly. That's the same "everything is a token" move that unlocked omni-modal LLMs, applied to the morphology-control loop.

How it works

From the README (the paper has the details): robots, states, and actions are serialized into RoboTokens; a diffusion transformer (backbone modified from Meta's DiT) is trained on procedurally generated robots over parameterized design spaces, with demonstration data produced by differential-IK oracle controllers (Mink) and RL controllers (Brax PPO on MuJoCo MJX). At inference, you condition on a target motion and denoise either actions (cross-embodiment control), hardware parameters (design generation), or both. CMA-ES (pycma, evosax on GPU) serves as the classical co-design baseline. Real-world motion conditioning comes from UMI human demonstrations, including an iPhone-tracked variant (iPhUMI).

What's actually there

Unusually complete for a robotics paper repo: pretrained checkpoints and data (downloaded via Getting Started), evaluation scripts for control and co-design, RoboTokens extraction from MJCF, procedural design-space generation, Mink and RL data-gen pipelines with parallelization docs, Hydra-configured training for control-only / hardware-only / joint modes, CMA-ES baselines, UMI collection and processing, and even the Blender visualization pipelines. There's an explicit "Extending" section for adding new design spaces and new RoboToken fields, plus a stated Limitations section — good signs of a repo meant to be built on, not just to reproduce a figure.

Evidence

The headline result is real hardware: they fabricated a generated design for cloth flinging on a bimanual ALOHA platform.

Lower peak joint speed with better tracking is the interesting part — the design absorbs dynamic demands that the stock morphology had to buy with velocity. The README doesn't give sim benchmark tables or comparisons against CMA-ES numerically; those are in the paper.

Caveats

• One fabricated task. The hardware validation is a single task family (cloth flinging) on one platform. Whether generated designs are good across diverse motions, or how broad the procedural design spaces really are (parameterized morphology, not free-form), is the key open question — check the paper and the "Adding New Design Spaces" docs.
• License split. Most of the repo is MIT, but the model core (t2/model/core.py and positional-embedding utils) inherits DiT's Attribution-NonCommercial license. Commercial use of the model as-is is off the table without a rewrite of those files.
• Heavy dependency stack: MuJoCo/MJX, JAX (Brax PPO), Mink, evosax, Blender for viz. Expect GPU requirements for MJX-parallelized data gen and RL, though the README doesn't state hardware specs.
• Fabrication itself (CAD-to-part) presumably involved manual steps; the repo covers design generation, not manufacturing.

Try it

The README's Getting Started downloads checkpoints and data, then evaluates a pretrained model on both control and co-design:

git clone https://github.com/real-stanford/transformer-transformer
cd transformer-transformer
# follow README: ⚙️ Setup → 📍 Checkpoints & Data → 📊 Evaluation

Videos of the fabricated robot and diffusion-process visualizations are on the project page: https://transformer-transformer.github.io/