MiniMaxAI/MiniMax-H3

TL;DR: MiniMax H3 is an open-weights (with caveats) 33B omni-modal generator that turns arbitrary mixes of text, images, video clips, and audio clips into up to 15 seconds of 24fps video with natively synchronized 32kHz stereo audio — dialogue in 11 languages, voice-timbre cloning from a reference clip, video editing with lip re-animation, the works. It's the most capable open audio-video release so far, but two of the three system modules (the Context-IR prompt-preprocessing pipeline and the 2K regeneration stage) are hosted-API-only, so the local story is 768p and hand-rolled prompting.

What it is

H3 treats audio-video generation as one joint prediction problem rather than video-then-foley. A single dense single-stream transformer (H3-Omni-Transformer, 33B params) predicts video latents and stereo audio latents together in one packed sequence. Two checkpoints are released, both CFG-distilled BF16:

• FL2VA — text-to-AV, plus first-frame / last-frame / both-frame conditioning.
• Ref2VA — the interesting one: up to 9 reference images, 3 video clips, and 3 audio clips (max 12 files, ≤15s total per media type) mixed freely. The README's Ref2VA example takes a source video, a music track to reuse, and a separate voice-timbre reference, then re-animates the subject's mouth to speak new dialogue in that voice while keeping the original footage and score — that's video editing, voice cloning, and dubbing as one generative call.

How it works

Text goes through the H3-Encoder, which is literally the full pretrained Qwen3-VL-32B with hidden states tapped from layer 50 — an off-the-shelf VLM as conditioning encoder rather than a bespoke text tower. Visual inputs are encoded both by the VLM (semantics) and by H3-VisualVAE (a temporally causal f16t4d24 autoencoder; after 1×2×2 patchification tokens are 32× spatially / 4× temporally downsampled). Audio uses a shared-weights per-channel VAE at 40Hz latent rate per channel, which is how stereo comes for free. Everything is packed into one sequence with 3D MM-RoPE over (t, h, w).

The architecture notes are worth reading: attention and FFN layers have no modality-specific structure — modality specialization lives only in I/O layers and per-modality AdaLN branches. About 13B of the 33B parameters are AdaLN, and since modulation outputs can be precomputed and cached, inference effectively runs a ~20B model. Sparse attention was used in late-stage training but the open release ships full-attention inference only.

The cleverest design choice is H3-Regenerate-2K: instead of a super-resolution module, the 768p output plus the original multimodal context are fed back into the base model, which regenerates at 2K in-context. That lets it recover details a blind upscaler must hallucinate (small text, fine reference details). It's also a nice existence proof that a general AV model can subsume its own SR stage as just another conditioning task.

What's actually there, and the catches

Weights for both checkpoints (original + diffusers layouts), full weights including the AdaLN branches to enable fine-tuning, tokenizer/encoder/VAEs, prompting guides, a GitHub "skills" repo, and day-one serving in SGLang, vLLM, diffusers (ModularPipeline), and ComfyUI. Reproducible example scripts for T2VA/FL2VA/Ref2VA with reference outputs.

Catches: no benchmarks or quantitative evals anywhere in the card — evidence is example videos only. Context-IR expands short prompts into elaborate multi-thousand-token structured descriptions (the examples show 3k–10k completion tokens), and the model was clearly trained against that format, so raw prompts will underperform without your own preprocessing LLM. Sparse attention isn't released, so long sequences are expensive; the reference deployment uses 4 GPUs with Ulysses sequence parallelism — expect a multi-GPU node for the 33B in BF16. License is a custom "Community License" with an application form required for USA/EU/UK/South Korea users, which will complicate commercial adoption in exactly those markets.

Try it

hf download MiniMaxAI/MiniMax-H3 --include "model_index.json" "FL2VA/*" --local-dir MiniMax-H3

sglang serve \
 --model-path MiniMaxAI/MiniMax-H3 \
 --num-gpus 4 --ulysses-degree 4 \
 --performance-mode speed \
 --host 0.0.0.0 --port 30010 \
 --model-variant fl2va

Or skip local entirely: the full system (Context-IR + 2K) is live at hailuoai.video and via the platform.minimax.io API. If you deploy locally, read VIDEO_PROMPT_WRITING_GUIDE_base_en.md first — output quality lives or dies on the prompt format.