catnip-ai-tech/MaineCoon

TL;DR: MaineCoon is a 22B-parameter autoregressive model from the Catnip AI Team that streams synchronized audio+video chunk-by-chunk, claiming sub-second interaction latency and up to 47.5 FPS at 480p on a single H100 — roughly 7× faster than other streaming audio-visual generators. It's framed as the first "social world model" — a generator built natively for real-time human interaction rather than an offline diffusion model retrofitted for streaming. The catch: this repo is a technical report plus links, with no weights and no code, so for now the evidence is the paper, hosted demos, and a self-defined benchmark.

What it is and why it matters

Most audio-visual generation today is offline bidirectional diffusion: you prompt, you wait, you get a clip. MaineCoon's pitch is that interactive social video — talking-head conversation, reactive avatars, high-tempo social-media pacing — is fundamentally a streaming problem, and the whole stack (data pipeline, attention pattern, KV-cache usage, training recipe, inference framework) should be designed for deployment-time streaming end-to-end. That's the framing worth paying attention to: it's the video analogue of the shift from turn-based to full-duplex speech models. If the numbers hold, a 22B model doing joint audio+video at 47.5 FPS with sub-second reaction and sub-$0.001/second cost changes what interactive video products are feasible.

How it works (per the README)

The generative core is an autoregressive audio-visual model built on Lightricks' open LTX-2.3 backbone, made streaming-native via a multi-stage training recipe the team calls "forcing-free streaming training": self-resampling, cross-modal representation alignment, domain-aware preference optimization, and ROPD (reinforced online-policy distillation) — an extension of the DMD/DMD2 distribution-matching distillation line, which is presumably where most of the speed comes from.

On top of that sits an agentic streaming inference layer for thousand-second-scale generation: agentic cache management, chunk commitment, long-context rollout, and prompt planning to fight the drift that kills long autoregressive video rollouts. The README doesn't detail how these components actually work — that's in the arXiv report (2606.17800).

Evidence

The headline result is speed. On 480p/20s generation, single H100:

On quality, they introduce SocialVideo-Bench (9 metrics: visual/motion/audio quality, ImageBind alignments, audio-visual harmony) and report a best average of 0.934 vs 0.895 for the next-best streaming baseline (SoulX-FlashTalk) and 0.848 for LTX-2.3. They win most metrics but notably lose motion (1.62 vs JavisDiT++'s 2.22) and IB-TV. Standard caveat: authors evaluating on a benchmark they designed, with metrics ("AVH", "JAVIS") defined in their own paper.

What's actually there

Be clear-eyed: the GitHub repo explicitly states it contains no model weights or source code — it's the README, embedded demo videos, and links. There is a Hugging Face org link and an arXiv link, but from the README alone you cannot verify whether the HF page has weights or is a placeholder. The runnable artifact is their hosted experience platform at mainecoon.tech, where you can interact with it live — which is real evidence of a working system, but not reproducible evidence. No license information, no hardware/inference code, no training data details in the README. The "first real-time audio-visual autoregressive model" and "record-breaking" claims should be read as marketing until independently benchmarked; the FPS table also mixes bidirectional and streaming models measured under one setting, which flatters the comparison.

The team is upfront that this builds on LTX-2.3 and DMD2, which makes the contribution more legible: streaming-native training + aggressive distillation + inference-time orchestration on a strong open backbone, rather than a new architecture from scratch.

Try it

No install path exists yet. The shortest routes:

Live demo:   https://mainecoon.tech/experience-platform
Paper:       https://arxiv.org/abs/2606.17800
HF (check for weights): https://huggingface.co/catnip-ai-tech/MaineCoon

Worth 15 minutes on the experience platform to judge latency and drift yourself, and worth watching the HF page — if 22B streaming audio-visual weights actually land under a permissive license, this becomes a foundation people build interactive avatar and social-video systems on. If they don't, it's a well-produced tech report with a compelling demo.