Riemann-Dynamics/Matrix-Game-3.5

TL;DR: Matrix-Game 3.5 is a 5B-parameter interactive world model (built on Wan2.2-TI2V-5B) that generates 720p, camera-controllable video over long horizons, with runnable checkpoints for both bidirectional base inference and a distilled three-step causal generator. The interesting bit is a parameter-free geometric memory: past frames are lifted into 3D via metric depth and retrieved by visibility, with camera projection folded directly into RoPE — long-horizon scene consistency without touching the backbone. This is the current frontier for "playable video" world models, and it's actually released.

What it is

The Matrix-Game series (2.0 → 3.0 → 3.5) is a lineage of real-time streaming interactive world models: you give an anchor image, a camera trajectory, and a text prompt, and the model generates a video that respects the camera path — return to a viewpoint and the scene should still be there. Version 3.5's headline contribution is Patch Memory + Warped PRoPE for cross-view scene recall, plus a static/dynamic split so that a moving protagonist doesn't get smeared into the geometric memory.

How it works

Three pieces, per the README:

1. Patch Memory + Warped PRoPE. Past observations are unprojected into a 3D memory using metric depth from Depth-Anything-3 (a "Mosaic Memory" frustum reprojection engine is vendored in the repo). Retrieval is geometrically aligned and visibility-aware: when the camera revisits a region, the relevant memory patches are pulled back in. Warped PRoPE folds the camera projection matrices into the spatiotemporal RoPE, so attention jointly encodes temporal order and view geometry. Both are parameter-free — no new learned modules on the backbone, which is why the checkpoints are just DiT weights loaded on top of stock Wan2.2-TI2V-5B.

2. Static–dynamic decoupling. The static scene lives in Patch Memory; dynamic subjects (the third-person protagonist) are carried by lightweight multi-view reference tokens encoding identity and appearance, with motion-aware filtering so moving things don't pollute the geometric memory. This targets the classic failure modes of memory-augmented video models: ghosting of moved objects and identity drift over long rollouts.

3. Distillation to real-time. The base model is bidirectional and slow (25 denoising steps). They distill to a few-step causal generator in two stages: Flow Matching in a perceptual feature space to jointly learn autoregressive denoising and few-step generation, then curriculum-style self-rollout DMD that progressively distills CFG, camera control, and memory conditioning. Result: a three-step causal checkpoint for minute-long streaming generation. (Self-Forcing is acknowledged as the ancestor here.)

What's actually released

• Weights: two 5B base bidirectional models on HF (first-person; third-person with optional 0–4 protagonist reference crops), plus a distilled three-step first-person-only causal checkpoint.
• Code: infer.py and infer_distilled.py with sample scenes, plus a diagnostic memory_visualization.mp4 output showing generation alongside the mosaic memory — useful for actually inspecting recall behavior.
• No quantitative results in the README. No FPS numbers, no consistency metrics, no comparisons to 3.0 or to WorldMems/StreamDiT-style baselines. There's a technical report link, but the README itself gives only mechanisms. Treat "real-time" as a claim to verify — a 5B model at 704×1280 in 3 steps is plausible on an H100, but nothing here says what hardware achieves what frame rate.

Practicalities

Apache-2.0, Linux only, ≥40 GB VRAM (A/H series tested), 64 GB RAM. Dependencies are heavy but self-contained: DiffSynth pipeline, the frustum engine, and Depth-Anything-3 are vendored, and no flash-attention compilation is needed. Each block is 80 frames / 84 camera poses; camera input is a simple .npz of c2w extrinsics plus intrinsics. Note that "interactive" here means camera-trajectory-conditioned — there's no keyboard/action interface in this repo's inference scripts, so live gameplay-style control isn't demonstrated in what's released.

Try it

git clone https://github.com/Riemann-Dynamics/Matrix-Game-3.5
cd Matrix-Game-3.5
conda create -n mg35 python=3.10 -y && conda activate mg35
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt

hf download RiemannDynamics/Matrix-Game-3.5-Base --local-dir checkpoints/Matrix-Game-3.5-Base
hf download Wan-AI/Wan2.2-TI2V-5B --exclude "assets/*" "examples/*" --local-dir checkpoints/Wan2.2-TI2V-5B
hf download depth-anything/DA3NESTED-GIANT-LARGE-1.1 --local-dir checkpoints/DA3NESTED-GIANT-LARGE-1.1
ln -s Matrix-Game-3.5-Base/first-person.safetensors checkpoints/first-person.safetensors

python infer.py --person first \
  --image samples/first_person/case_7/input.png \
  --camera samples/first_person/case_7/camera.npz \
  --prompt-file samples/first_person/case_7/prompt.txt

Demos and the technical report are at matrix-game-v3-5.github.io. Watch the memory_visualization.mp4 output on a revisit-heavy camera path — that's where this either delivers or doesn't.