Viggle/Viggle-Animate

TL;DR: Viggle-Animate replaces the character in a video using nothing but the video and one of its own frames repainted in an image editor — no pose skeleton, no segmentation mask, no face tracker, no text encoder. It's a 33.1B full finetune of MiniMax-H3's ref2va transformer, DMD-distilled to three forward passes: 124 frames in 26 s on a B200, a claimed 6.1× faster than Wan2.2-Animate-14B with a far simpler interface. This is the most interesting reframing of controllable video editing this week.

The idea

The standard character-replacement stack conditions on a standalone character photo plus a pile of extracted intermediates (pose tracks, masks, face crops, background plates) whose entire job is to reconcile a photograph with footage it was never part of. Viggle's move is to hand that reconciliation to an image editor: grab one frame of the clip, repaint the character in gpt-image or Photoshop, and now your reference already agrees with the footage on pose, camera, framing, and lighting. The video model's only remaining task is propagating an edit that is already correct on one frame across the shot.

Because nothing in the loop encodes "person" — no skeleton with a neck, no person-shaped mask hole — the model animates whatever you paint: animals, a clay figure, a robot, even an airliner whose wings the paint binds to the actor's arms. Identity is never named; conditioning is a single frozen prompt embedding shipped with the weights. The text encoder is never loaded.

Viggle-Animate driving video one repainted frame 33.1B transformer 3 forward passes 26 s / clip (124 frames, B200)

How it works

Two artifacts ship: a full bf16 finetune of MiniMax-H3's ref2va transformer on the replacement objective (14 shards), and a rank-128 LoRA (2.5 GB) that is the DMD2 distillation delta — that's what collapses sampling from ~30 steps to three passes. The LoRA only makes sense on the finetuned transformer, not stock H3.

The distillation detail is genuinely clever: two teachers split by noise level. The finetune supervises the high-noise end (where the swap is decided), the original H3 supervises the low-noise end (where texture is decided), so the student inherits the finetune's replacement ability without its visual regressions. Worth noting for anyone distilling finetuned video models.

There's also a nice retraction-turned-feature in the limitations: lip-sync looked weak, but it turned out the model was syncing the mouth to its own generated audio track, which the fixed prompt asks to be silence. Pinning the driving clip's audio into the target audio rows as a clean latent throughout the denoise — an inference-only change — makes the mouth track real speech. That's an interesting empirical fact about conditioning omni-modal generators via clamped latent rows.

Evidence and caveats

The headline comparison (same machine, B200, same source videos and output geometry) against Wan2.2-Animate-14B at its documented replacement settings:

Caveats: this is a self-run comparison, and the quality claims (sharpness under fast motion, generalization to non-humans) rest on cherry-pickable video panels, though the README says generalization clips are single renders at --seed 42, no best-of-N. Known failure modes are stated honestly: shape reverts to the driving pose (a LEGO minifigure kept its palette but regained human anatomy), multi-character and cut-heavy shots degrade, and everything depends on the quality of your image edit. The team says a better model is training.

Practical friction is real. bf16 needs ≥96 GB (62 GiB resident, 80.1 GiB peak — one 80 GB card is not enough) or slow CPU offload. Their own RTX 5090 NVFP4 deployment path is not shipped. It also depends on downloading ~11 GB of the 269 GB MiniMax-H3 base (VAEs, schedulers) and a pinned diffusers commit. License: the weights fall under the MiniMax H3 Community License (read before shipping a product); the inference code is Apache 2.0. ComfyUI nodes and a hosted demo exist.

Try it

hf download MiniMaxAI/MiniMax-H3 --local-dir ./MiniMax-H3 \
  --include "modular_model_index.json" "vae/*" "audio_vae/*" \
  "scheduler/*" "audio_scheduler/*" "assets/ref2va.mp4"
hf download Viggle/Viggle-Animate --local-dir ./Viggle-Animate
pip install torch "git+https://github.com/huggingface/diffusers@d6726f3" av

# pull a frame, repaint it in your editor of choice, then:
python Viggle-Animate/inference/sample.py --model-dir ./MiniMax-H3 \
  --cond driving.mp4 --ref ref.png --out swapped.mp4

examples/demo.sh runs end-to-end on a bundled clip (good LoRA sanity check); pass --audio driving.mp4 to get lip-sync; add --offload if you're under 96 GB. Model card: https://huggingface.co/Viggle/Viggle-Animate