facebookresearch/robo_jepa

TL;DR: Meta FAIR open-sourced RoboJEPA: action-conditioned world models that predict in the latent space of a frozen V-JEPA 2.1 encoder, trained on robot data from 12 embodiments at predictor sizes from 22M to 8B. The headline claims are a clean power law — latent rollout error scales predictably with training compute, and downstream planning success tracks it — plus zero-shot goal-image planning via CEM/MPC on a real Franka. The release is unusually complete: every intermediate checkpoint of every size, training/eval/planning code, data converters, and a one-conda-env reproduction path.

What it is and why it matters

The idea is the JEPA thesis applied to robot control at scale: don't predict pixels, predict the future in a good pretrained representation space. A frozen V-JEPA 2.1 ViT-G/384 provides 1664-dim features; RoboJEPA trains only the action-conditioned predictor transformer on top, from 22M to 8B parameters. Training uses a 2-step autoregressive rollout loss (K=2), followed by a K=10 "cooldown" that is the long-horizon recipe used for planning.

Two things make this release interesting beyond yet-another-world-model. First, the scaling-law angle: imagination error (per-step L1 of latent rollouts on held-out DROID) follows a power law in compute, and they released every training checkpoint of every size — 70+ checkpoints total — with a script and FLOP accounting so you can reproduce the curve yourself. That's a rare artifact; most scaling-law papers release a final model at best. Second, planning is done entirely in latent space: a CEM/MPC planner rolls out candidate action sequences through the predictor and scores them against a single goal image's embedding. No reward model, no task-specific training, no policy head.

What's actually in the release

Code and weights for all eight sizes plus an 8B variant finetuned on DROID with three camera views at 720p. Third-party pieces you fetch separately: the V-JEPA 2.1 encoder and NVIDIA's Cosmos tokenizer (decoder only). Data converters exist for DROID and AgiBot World; converters for the rest of the 12-embodiment mixture "will follow," so you cannot yet reproduce the full pretraining mix. Four RoboCasa tasks (Reach, ObjectReach, ObstacleReach, ObjectPush) have one-command reproduction scripts, and there are env servers for real DROID/Franka deployment.

Notably absent from the README: any actual numbers. No planning success rates, no L1 values, no comparison to baselines — the evidence lives in the paper (arXiv:2610.10515) and in the scripts that regenerate it. The scaling law is the claim to check first since the repo makes that fully reproducible.

Caveats

• The real-robot and simulated tasks are reaching and pushing toward a single goal image. This is genuinely zero-shot planning, but it's positioning, not contact-rich or long-horizon manipulation — temper the "world model plans on a real robot" framing accordingly.
• CC BY-NC-SA 4.0 — non-commercial, for both code and weights.
• Heavy setup: a pinned conda env (PyTorch 2.7.1/CUDA 12.6), the frozen ViT-G encoder is always in the loop (so even the 22M predictor needs a giant encoder forward pass), RoboCasa assets are ~23 GB, and the repo is clearly built around SLURM clusters (submitit, S3 data staging). There's AOT-compiled/FP8 inference code for the big models.
• Repo is days old with the usual rough edges the troubleshooting section already admits (attention-backend incompatibilities, a frame-rate bug in older configs).

Trying it

Smallest path — load the 50M model and run a RoboCasa planning repro:

git clone git@github.com:facebookresearch/robo_jepa.git && cd robo_jepa
ROBOCASA_ASSETS=1 bash scripts/setup_env.sh && conda activate $SCRATCH/miniforge3/envs/robo_jepa

mkdir -p checkpoints/wm-50m
cp configs/released/wm-50m/config.json checkpoints/wm-50m/
wget -P checkpoints/wm-50m https://dl.fbaipublicfiles.com/robojepa/wm/wm-50m/cooldown_50m_120k.pth.tar
ln -s /path/to/vjepa2_1_vitG_384.pt checkpoints/wm-50m/encoder.pth.tar

bash scripts/deployment/examples/reproduce_reach.sh 50m 120k 0 1 2 3 4

If you care about the scaling-law result, scripts/evals/eval_droid_holdout.py plus the plotting snippet in the README regenerates the compute-vs-L1 curve from the released checkpoints — probably the most independently verifiable claim in the release.