jd-opensource/JoyAI-Echo

TL;DR: JD.com's JoyAI-Echo is two related releases built on Lightricks' LTX-2: Echo-LongVideo, which generates multi-shot audio-visual stories up to ~5 minutes with a paired audio-video memory bank carrying continuity across shots, and Echo-WM, an "enterable" omnimodal world model where you navigate continuously while video, ambient sound, music, and speech evolve together, rolled out causally with a KV-cache and 4-step inference. Both ship code and checkpoints; the README is thin on numbers, so evidence lives in the papers and the demo page.

What it is

The repo hosts two independent projects with separate environments and checkpoints:

Echo-LongVideo (in echo_longvideo/): long-horizon, multi-shot generation up to roughly five minutes. The core claim is cross-shot continuity — characters, scenes, and the audio track stay coherent across cuts, via a "paired audio-video memory bank." There's also a Director Agent (presumably an LLM planner that breaks a story into shots — the README doesn't elaborate) and "consumer-GPU inference profiles."
Echo-WM (in echo_wm/): an interactive world model. Instead of prompt-to-clip, it responds to continuous navigation input while generating video plus environmental sound, music, and speech jointly. The public "Flash Preview / Causal" variant uses chunk-causal attention, KV-cache rollout, and 4-step (distilled) inference — i.e., the standard recipe for turning a bidirectional video DiT into a streaming, autoregressive one, but here applied to a joint audio-visual model.

Both are derivatives of LTX-2.3 (Lightricks' audio-visual DiT), with a roadmap to rebase onto LTX-2.5 (Gemma 4 text encoder, new VAE/DiT) and to attack rollout cost with sparse attention (SageAttention), FlashAttention/FlashInfer, paged KV-cache with RoPE/UCPE rebasing on eviction, and FP8/TensorRT. The roadmap mentions a "UCPE branch" alongside video and audio attention branches — likely their positional/control encoding for navigation, but the README doesn't define it.

Echo-WM (interactive, streaming) Navigation input Chunk-causal AV DiT KV-cache · 4-step Video + ambient sound + music + speech closed loop: user steers as world rolls out

Two distinct capabilities in one repo: offline long-form storytelling held together by an audio-video memory bank, and a real-time-oriented causal world model where all four audio streams are generated jointly with the video.

Why it matters

Two things stand out. First, audio is a first-class citizen of the long-horizon problem. Most long-video work carries visual identity across shots; keeping voices, music, and ambience consistent across a 5-minute multi-shot narrative is a harder and largely unaddressed continuity problem, and the paired memory bank is the interesting mechanism to inspect in Paper 1.5. Second, Echo-WM extends interactive world models (Genie-style navigation-conditioned generation) from silent video to joint omnimodal output — footsteps, environment sounds, and speech evolving with your movement. If it works even at ~10 s (the Base model's horizon), that's a genuine capability step toward enterable generative environments.

What's actually there

Code for both pipelines (inference.py, configs, prompts; Gradio demo for Echo-WM), each with a bundled copy of ltx-core/ltx-pipelines so they don't conflict.
Two Hugging Face releases (Long Video and World Model checkpoints, downloaded separately).
ComfyUI integration.
Three papers (Echo 1.0, 1.5, Echo-WM) and a project page with samples.
Echo 1.0 is preserved on an echo1.0 archive branch.

What's missing from the README: any benchmark numbers, GPU requirements, latency figures, or resolution/fps specs. "Consumer-GPU inference profiles" is promising but unquantified. The roadmap itself signals that long-rollout cost is currently a pain point (sparse attention, paged KV-cache, FP8 are all still unchecked). Treat the 5-minute claim skeptically until you've watched full uncut samples on the project page — long-horizon systems typically degrade well before their advertised limit, and the "up to ~5 minutes" phrasing leaves room.

License: academic/non-commercial only, inherited from the LTX-2 Community License; commercial use requires talking to Lightricks. That will limit adoption relative to Apache/MIT video stacks.

Try it

Echo-WM's Gradio demo is the fastest way to get a feel for the interactive claim:

git clone https://github.com/jd-opensource/JoyAI-Echo
cd JoyAI-Echo/echo_wm
conda create -n echo-wm python=3.11 -y && conda activate echo-wm
pip install -r requirements.txt
# download checkpoints per echo_wm/README, then run inference_wm.py or the Gradio demo

For long video: cd echo_longvideo && conda env create -f environment.yml. Before installing anything, watch the samples at the project page — full-length, uncut clips are the real test here.