XXH333/WordVoice-main

TL;DR: WordVoice is an LLM-TTS system built on CosyVoice3 that gives you explicit, independent, per-word knobs for duration, pause boundary, energy, pitch, and intonation shape — the AR model literally "plans" each word's acoustics as tokens before emitting its speech tokens, and you can override any of those plans by hand. The release is unusually complete (training + inference code, a 0.5B checkpoint, the WordVoice-5A dataset, annotation pipeline, and a HF Spaces demo), but the README contains zero quantitative evidence, so the demo is where you should form your opinion.

What it is and why it's interesting

Most LLM-based TTS (CosyVoice, ChatTTS, Fish-Speech, etc.) controls style implicitly — via a reference clip or a text instruction — which is coarse and non-deterministic. If you want this specific word stretched, quieter, and falling in pitch, you're stuck regenerating and praying. WordVoice's pitch (pun intended) is decoupled, deterministic, word-level control: five acoustic attributes per word, each independently settable or left to the model. That's the editing primitive audiobook and dubbing pipelines actually need, and it's a plausible foundation for "prosody editing" UIs, which basically don't exist today for zero-shot TTS.

How it works

Two ideas do the work:

1. "Acoustic thinking" via bound-tokens. In the autoregressive LM, each word's speech tokens are preceded by a bound-token block where the model explicitly predicts that word's attributes — duration, boundary class (b0–b4 pause levels), energy (0–1), pitch (−1 to 1), and tone shape (7 prosodic morphologies: flat, rise, strong rise, fall, strong fall, peak, valley). It's chain-of-thought for prosody: plan the word's acoustics, then generate its sound conditioned on the plan. Because the plan is explicit tokens in the sequence, user intervention is trivial — you just replace the model's predicted attribute tokens with your own (Control Mode), or let the model fill them in (Free Mode).

2. Word-level conditioning in the Flow Matching stage. Discrete speech tokens lose fine acoustic detail, so the FM decoder gets an LLM-derived word-level style token upsampling + fine-grained conditional modulation module, intended to make the waveform actually honor the word-level plan rather than washing it out.

What's actually released

• Code: full inference (wordvoice_infer.py, both modes) and training (train_code/wordvoice/run_wordvoice.sh), Apache-2.0.
• Weights: WordVoice-base-0.5B on HF (Qwen2.5-family backbone via CosyVoice3), Apache-2.0. download_models.sh pulls it plus CosyVoice3 and MMS-FA (forced alignment) into checkpoints/.
• Data: WordVoice-5A dataset + the annotation pipeline that produces the five word-level attribute labels (its own separate license — check before commercial data use).
• Demo: HF Spaces demo built by the HF team, plus a project demo page.
• Paper: arXiv 2607.06461.

Caveats

The README gives no numbers at all — no MOS, no control-accuracy metrics, no comparisons to Free Mode CosyVoice3 or other controllable TTS (e.g., prompt-based style control, FastSpeech-style variance adaptors, which did explicit pitch/duration/energy prediction years ago at phoneme level, though not in an LLM-token, user-editable form). Whether the five controls are truly decoupled — change pitch without dragging energy and duration along — is exactly the claim to stress-test in the demo. Languages are Mandarin and English only. At 0.5B the model is small, so expect zero-shot voice-cloning quality below the big CosyVoice3 checkpoints. No hardware requirements are stated; a 0.5B LLM + FM decoder should run on a single consumer GPU. Repo is ~2 months old, 70 stars, so expect rough edges.

Try it

Fastest path: the HF Spaces demo. Locally:

conda create -n wordvoice python=3.10 -y && conda activate wordvoice
git clone https://github.com/XXH333/WordVoice-main.git && cd WordVoice-main
pip install -e .
pip install num2words==0.5.14 x_transformers==2.11.24
bash download_models.sh
python wordvoice_infer.py   # edit text / ref audio / per-word controls in the script