hugging-apps/fireredaudio-tts-demo

TL;DR: FireRedAudio is a 9B audio language model that handles both directions — understanding (ASR, audio QA) and generation (zero-shot cloning, prompt-driven voice design, speech editing) — from a single autoregressive backbone, and this Space is a runnable four-tab Gradio demo of it. The interesting bit is the architecture: instead of discrete audio tokens, it uses two different continuous representations, a Whisper-style encoder for input and a VAE-latent-plus-flow-matching pipeline for output, which is a clean articulation of the "continuous everywhere, decoupled per direction" design that several labs have been converging on.

Why this matters

Most unified speech LLMs quantize audio into discrete codec tokens so the LLM can treat everything as next-token prediction, and pay for it in fidelity and awkward tokenizer coupling. FireRedAudio's bet is that the LLM should predict hidden states and let a small diffusion-style head do the acoustics — the same move as recent continuous-token generation work (e.g., MAR-style image generation, and CosyVoice-adjacent TTS designs), but here applied symmetrically across understanding and generation in one 9B model. If the FireRed lineage holds (this is presumably the same team behind FireRedTTS/FireRedASR, i.e. Xiaohongshu's audio group), it's a serious open effort rather than a weekend fine-tune.

The other draw is speech editing as a first-class task: insert/substitute/delete words in an existing recording, or change pitch/speed, while keeping the voice. That's still rare in open models and is where a unified text+audio model genuinely pays off — the model rewrites the transcript in text before switching modality to synthesize the patch.

How it works

• Understanding path: a 32-layer Whisper-style encoder (128-mel, 16 kHz) produces features at 12.5 Hz that are spliced directly into the LLM's token stream as continuous embeddings. No discrete ASR tokens.
• Generation path: a continuous audio VAE ("RedAE") encodes 24 kHz audio into 25 Hz, 64-dim latents. The LLM predicts one hidden state per group of four latents (so ~6.25 Hz autoregressive steps — cheap), and an 11-layer flow-matching head ("RedDiT", 10 denoising steps, CFG 2.0) expands each hidden state into the actual latents. A separate RedAE decoder (Qwen3 + ISTFT head) renders the waveform.
• Because text and audio share one backbone, generation tasks can interleave text: voice design first writes its own structured timbre tags from your natural-language description, then speaks; semantic editing writes the rewritten transcript, then synthesizes.

What's actually here

• A working ZeroGPU Gradio demo with four tabs (cloning, voice design, editing, ASR/QA), also exposed as an MCP server per the Space tags.
• The Space wraps the upstream fireredaudio package and inference.py verbatim (Apache-2.0), so what you hear should match the reference implementation — modulo one honest caveat: the optional kernels (flash-attn, liger-kernel, etc.) aren't installed, so bf16 reduction order differs and outputs are numerically (not qualitatively) different from the paper's setup.
• The README links a paper, code repo, and samples page, but contains no benchmark numbers — no WER, speaker-similarity, or MOS figures. Take the capability claims as demo-verified, not measured.
• Weights are implied (the Space loads a ~21 GB bf16 backbone) but the README doesn't state a model-card link or weight license; the code is Apache-2.0. Worth verifying on the upstream repo before building on it.

What to expect

Cold starts pay a weight-streaming penalty on ZeroGPU. Voice cloning requires an exact transcript of the reference clip — a mismatch degrades the clone, which is a real usability constraint versus transcript-free cloners like some recent systems. For local use you'll want a GPU with ≥24 GB for the bf16 backbone plus the fp32 RedAE decoder. Open questions: how cloning/editing quality stacks up against specialized systems (CosyVoice 3, VoiceCraft-style editors), and whether the understanding side is competitive with dedicated ASR — the README makes no comparative claims either way.

Try it

Fastest path is the hosted demo — upload a few-second clip plus its exact transcript in the cloning tab, or just type a voice description in the design tab:

https://huggingface.co/spaces/hugging-apps/fireredaudio-tts-demo

For local inference, follow the linked upstream repo (fireredaudio package, Apache-2.0); the Space uses its inference.py unmodified, so its README is the authoritative setup guide.