kandinskylab/kandinsky-6

TL;DR: Kandinsky 6.0 is an MIT-licensed family of diffusion models (3B "Lite" and 29B "Pro") that generate 5-second video clips with synchronized 44 kHz audio — including lip-sync — from text or an image, plus a plug-in super-resolution model to 1080p. Open joint audio-video generation is still rare (the capability that made Veo 3 notable is almost entirely closed), and this ships with weights, a distilled fast variant, and day-one integrations into Diffusers, ComfyUI, vLLM-omni, SGLang, and FastVideo. The catch: the README is all packaging and speed numbers, with essentially no architecture, training, or quality details.

Why it matters

The open video-generation ecosystem (Wan, HunyuanVideo, CogVideoX, Mochi) produces silent video; audio is bolted on afterward with a separate foley/TTS pipeline, and lip-sync in particular is hard to retrofit. A model that generates the audio and video tracks jointly — so that a prompt like "a cat opens its mouth and says Hello" produces matching mouth movement and speech — is a genuine capability gap in open models. If the lip-sync claim holds up on real prompts, this is the first well-packaged open option for it, and the Kandinsky lineage (Sber's image/video line) has shipped working models before.

The second notable thing is accessibility: the published timings include an RTX 5060 Ti and RTX 4090 with block offload, meaning the 29B Pro model is expected to run (slowly) on consumer cards, not just 80 GB datacenter GPUs.

What's actually in the repo

Weights for Lite (3B), Pro (29B), a Pro Distill variant, and the video super-resolution model; quick-start guides for four serving paths (native pipeline, vLLM-Omni, SGLang, FastVideo); and a just-based recipe system. Modes are T2AV and image-conditioned TI2AV. Attention backends are tuned per card: SageAttention2++ on consumer GPUs and A100, FlashAttention 3 on H100.

The only quantitative evidence is a wall-clock table for the non-distilled models (5-second clip, after warmup, excluding weight load and encoding):

So: ~7 minutes for Pro Full HD on an H100, ~21 minutes on a 4090. Oddly, Pro HD is faster than Pro SD on most cards (e.g., 292 vs 356 s on H100) — likely a resolution/kernel-shape artifact, but unexplained. No numbers for the Distill variant, which is presumably much faster and is what the Hugging Face Space serves.

What's missing and what to watch

The README gives zero information on architecture (how audio and video latents are coupled, what the audio codec is, text encoder, diffusion formulation), training data, or quality — no VBench, no AV-sync metrics, no human evals, no comparisons to Veo 3 or Wan+foley pipelines. No paper is linked. The lip-sync claim rests entirely on the demo prompt. Clips are capped at 5 seconds. VRAM requirements aren't stated explicitly; the offloading configurations imply Pro doesn't fit resident on anything below 96 GB. Treat quality as unverified until you run your own prompts — the Space is the cheapest way to do that.

Trying it

git clone https://github.com/kandinskylab/kandinsky-6.git
cd kandinsky-6
just setup
just download pro-distill
just generate 'A cat is lying on a mat, turns toward the camera, opens its mouth and says Hello'

Requires an NVIDIA GPU, uv, and just; see docs/quick_start.md (or the vLLM/SGLang/FastVideo variants), or the Kandinsky 6.0 Pro Distill Space on Hugging Face for a zero-install test of the lip-sync claim.