mira-wm/mira

TL;DR: MIRA is a 5B-parameter latent diffusion world model that runs a full 2v2 Rocket League match entirely inside the network — four players, four synchronized first-person views, all conditioned on everyone's keyboard actions, at 20 FPS on a single GPU. It's a joint release from General Intuition, Kyutai, and Epic Games with training code, a large multimodal dataset, and a live playable demo at mira-wm.com. The interesting bits are (a) multiplayer, i.e. multi-agent, multi-view consistency in a neural game engine — everything before this (GameNGen, Oasis, Genie-class models) was single-player — and (b) diffusing in a frozen DINOv3 representation space instead of a VAE latent space.

What's actually new

Two things stand out relative to the interactive-world-model lineage:

Multiplayer. The model generates four players' views frame by frame from all four players' action streams simultaneously. That forces the model to maintain one implicit shared game state (ball position, car physics, score) that stays consistent across four rendered perspectives while four humans provide inputs in real time. That's a qualitatively harder problem than one-player Doom or Minecraft rollouts, and it's the thing to stress-test in the demo: do the views desynchronize? Does the ball behave the same for everyone?

Representation Autoencoders. Instead of the usual VAE codec, the tokenizer is built around a frozen DINOv3-L/16 encoder — diffusion happens in a semantic, pretrained representation space with a learned decoder mapping back to pixels. This is the "RAE" idea from recent image-diffusion work applied to a real-time interactive video model. The plausible payoff is better semantic coherence per latent dim and faster convergence; the README doesn't quantify it, so that argument lives in the tech report (arXiv:2607.05352).

What's in the release

• Code (Apache-2.0): full training pipeline — codec training, single-player world model, and a multi-player wrapper warm-started from the single-player checkpoint. Hydra configs, torchrun multi-GPU, plus an offline eval script.
• Dataset (kyutai/rocket-science on HF): 4-second windows of 2v2 matches with all four synchronized views, per-frame 9-key multi-hot keyboard actions, game events (goals, boost pickups), and ground-truth physics state (ball, cars, score). Ground-truth state alongside video+actions makes this useful well beyond MIRA — e.g., probing what world models learn about latent physics.
• Demo: live playable at mira-wm.com, plus a video in the README.
• Not clearly in the release: pretrained model weights. The README shows how to train and evaluate your own checkpoints and links a demo, but I see no HF model card or checkpoint download for the 5B model. If you want the actual weights, check the homepage/paper — otherwise budget for training from scratch.

Evidence-wise the README is thin on numbers: no FVD, no coherence horizons, no hardware spec for the "single GPU" claim (a 5B diffusion model at 20 FPS almost certainly means a distilled few-step sampler on an H100-class card). The playable demo is the strongest evidence — interactive latency and drift are things you can feel directly, and it's harder to cherry-pick a live system than a video reel.

Caveats

• Codec training requires Meta's gated DINOv3 weights (RS_DINO_WEIGHTS_DIR); inference and world-model training don't.
• Requires pixi, torch ≥ 2.8, NVIDIA GPU. Repo is a month old, 521 stars — expect rough edges.
• Single-domain: this is a Rocket League simulator, not a general world model. The transferable ideas are the multiplayer conditioning scheme and the RAE codec; generality is unproven here.
• Rocket League is a friendly case for multi-view consistency (one ball, one arena, symmetric views). Whether the approach scales to games with richer hidden state is open.

Try it

Fastest path is the browser demo at mira-wm.com — no install. To poke at the code and dataset:

git clone https://github.com/mira-wm/mira && cd mira
pixi run setup      # needs an NVIDIA GPU
pixi run explore    # interactive 4-player dataset viewer

from mira.data import RocketScienceDataset
ds = RocketScienceDataset.from_hub("kyutai/rocket-science", split="test", shards=1)
clip = ds.load_match(ds.match_ids()[0], clip_len=16, target_fps=20)[0]
# clip.frames: (4, 16, C, H, W); clip.actions: (4, 16, 9); plus events + physics

Worth 30 minutes in the demo with a friend on the other keyboard — multi-agent consistency is the claim, and it's directly testable.