FireRedTeam/FireRedAudio

TL;DR: FireRedAudio (from the FireRed team, of FireRedTTS lineage — likely Xiaohongshu) is a 9B audio language model that does the whole audio stack in one checkpoint: ASR, audio QA/reasoning, hour-long temporal grounding, zero-shot voice cloning, instruction-driven TTS, voice design from text descriptions, and free-form speech editing. The interesting bet is architectural: instead of one shared discrete token space for both listening and speaking, it keeps two decoupled continuous representations — a Whisper-initialized encoder for understanding and a "RedAE" continuous-latent pathway for generation — hanging off a shared Qwen3.5 backbone. Code and weights are out under Apache-2.0.

The idea

Most unified audio models (Kimi-Audio, MiMo-Audio, Qwen-Omni variants) push toward a single tokenization that serves both understanding and generation, and accept the tension: semantic tokens are lossy for synthesis, acoustic tokens are noisy for reasoning. FireRedAudio's claim — "the first publicly disclosed design of its kind" — is to stop trying. Understanding flows through a continuous audio encoder; generation flows through a continuous autoencoder latent (RedAE, decoded by a flow-matching model to 24 kHz waveforms); only the LLM backbone is shared. Purpose-built representations, shared reasoning.

What it claims

The README's numbers are broad and, if they hold, strong:

• Understanding: MMAU test-mini 82.0 / test 80.9, MMSU 83.3 — reported above Gemini 3.1 Pro (80.7/78.8/82.7) and Qwen3.5-Omni-Plus. Note those comparison numbers are marked "our own evaluation."
• ASR: LibriSpeech clean 0.67 WER (unusually low — worth verifying), AISHELL-1 0.71, FLEURS-102 avg 14.94. Competitive rather than dominant across Chinese test sets.
• Zero-shot TTS: Seed-TTS-Eval avg 1.20 CER/WER with 0.71 SIM, edging out Qwen3-Omni-30B (1.23) and Seed-TTS itself (1.69).
• Editing: consistently beats Ming-UniAudio-Edit on semantic deletion/insertion/substitution and by wide margins on acoustic edits (volume relative-amplitude error 2.4% vs 14.9%).

The hour-long temporal grounding — timestamped structuring, grounded summaries, retrieval by time or content — is the most novel-sounding capability but is only shown as a demo in the README, with no benchmark table.

What's actually there

PyTorch code, pretrained weights on Hugging Face and ModelScope (backbone plus a separate RedAE decoder needed only for generation tasks), a Python API and CLI covering all six task modes, and example audio. No arXiv paper yet — the citation is a placeholder — so architecture details beyond the README (RedAE-Patch specifics, training data, RL involvement) are unknown. HF downloads are near zero at this point; nobody has independently kicked the tires.

Caveats to know before trying:

• Acoustic edits are templated, not free-form: the model was trained on exact strings like "shift the pitch by 3 steps" and "adjust the speed to 1.5". Semantic edits (rewrite/insert/delete content) are more flexible.
• Generation and understanding are Chinese/English only; only ASR is multilingual (FLEURS-102).
• TTS output is stochastic (flow-matching noise); seed it for reproducibility, and quality varies across seeds.
• Needs CUDA 12.8 by default, compiles flash-attn and causal-conv1d, so expect a real GPU box and a slow first uv sync. A 9B model plus decoder wants ~20+ GB VRAM in bf16.
• Voice cloning is flagged "solely for academic research purposes," though the license itself is plain Apache-2.0.

Bottom line

The decoupled-continuous-representations framing is a clean articulation of a position several groups have been converging on (Ming-UniAudio also uses continuous latents), and having it fully open at 9B with editing and hour-long grounding in one model makes it a useful reference system regardless of whether every benchmark number survives scrutiny. Treat the leadership claims as provisional until the paper lands; the editing tables against Ming-UniAudio-Edit are the most checkable and the most impressive.

Try it

git clone https://github.com/FireRedTeam/FireRedAudio && cd FireRedAudio
uv sync --extra accel --extra accel-build --group tools
uv run hf download FireRedTeam/FireRedAudio --local-dir pretrained_models/

# clone a voice from a reference clip
uv run inference.py --task tts --model pretrained_models/FireRedAudio \
  --vae-decoder pretrained_models/RedAE_decoder/model.pt \
  --prompt-audio assets/examples/tts_zh_prompt.wav \
  --prompt-text "同时，他强调微调要科学有序。" \
  --target-text "安徽淮南秦师傅发现，停在小区的爱车右前驾驶窗玻璃被砸。" \
  --language zh --output tts.wav