MuyeHuang/DuplexOmni

TL;DR: DuplexOmni is a full open-source stack for a real-time full-duplex voice+video agent: a Qwen3-Omni-based model that listens, watches, and speaks simultaneously (handles barge-in, overlap, backchannels), with a pluggable "System-2" channel that injects results from a slower reasoning model mid-conversation. The interesting part is the completeness of the release — synthetic data pipeline, Megatron training fork, modified vLLM serving, weights, and reproducible metadata for a 9TB dataset — and the concrete tag-based scheme for teaching duplex behavior. The README contains zero evaluation numbers, so the claims rest on the paper and your own testing.

What it is

Most omni models are half-duplex: they wait for you to stop talking, then respond. DuplexOmni targets the harder regime where the model keeps consuming streaming audio/video while generating speech — handling interruptions, waiting, and overlapping talk. The architectural bet is a two-speed split: a fast "S1" interaction model (Qwen3-Omni thinker/talker, Mimi codec output) that handles low-latency turn-taking, and a pluggable "S2" layer — any OpenAI-compatible reasoning model, retrieval, or tool — that runs in the background and injects its output (marked 「...」 in training data) before the agent states facts or conclusions requiring reasoning. This is the fast/slow split people keep proposing for agents, but here it's baked into the training data format and serving stack rather than bolted on.

How the data pipeline works

The central engineering contribution is a Writer-Director synthesis pipeline that manufactures duplex conversations, since no natural corpus contains them at scale with the right annotations:

1. Seed configs generate scenario/style/interaction constraints; public dialogue corpora (UltraChat-style) and video event tags (NExT-QA, LLaVA-video) provide content.
2. A Writer LLM turns content + constraints into natural scripts; cleaning stages classify S1 vs. S2 turns, filter and repair hallucinations.
3. A Director LLM converts scripts into tag-rich duplex transcripts: ^ and [CUT] for user interruptions, [WAIT]/[PENDXS] for holding, [THINK] to trigger the System-2 channel, 「...」 for injected S2 messages.
4. TTS (Qwen3-TTS + forced aligner) synthesizes and time-aligns both speakers, sessions are cut into chunks, Mimi codec features extracted, user-side noise injected (MUSAN/FSD50K).

What's actually released

• Code (Apache-2.0): full data pipeline, Qwen3-Omni E2E training framework (ms-swift/Megatron fork, bundled Megatron-LM v0.15), and realtime serving stack with a modified vLLM fork plus WebSocket server and a Mac CLI client.
• Weights: HF repo MuyeHuang/DuplexOmni (thinker/talker checkpoints).
• Data: full dataset is ~9TB so they ship reproducible metadata (MuyeHuang/DuplexOmni-Data) plus one complete parquet shard (train_e2e_codec_000000.parquet) as a schema reference. Recreating the rest means running the pipeline yourself — which requires substantial LLM API throughput (writer/director calls DeepSeek/Qwen-class models per sample) and TTS compute.

What's missing: the README gives no latency numbers, no duplex benchmark results, no demo video or hosted demo, and no comparison to Moshi, GLM-Voice, or other full-duplex systems. Evidence is deferred entirely to the arXiv paper. The selection note is right: test barge-in and turn-taking yourself before believing it.

Practical expectations

Maturity is "research release, honestly labeled": placeholders everywhere for paths and endpoints, legacy script defaults, and a warning that argument names may drift from the docs. Running inference needs CUDA GPUs, the modified vLLM fork, and both thinker and talker checkpoints; reproducing training data needs an OpenAI-compatible LLM farm plus Qwen3-TTS, a forced aligner, Mimi, and MUSAN/FSD50K. The for_agent.md file for coding agents is a nice touch that may itself become a pattern.

Even if the model underwhelms, the Director tag vocabulary (^, [CUT], [WAIT], [PENDXS], [THINK], 「...」) and the pipeline for manufacturing time-aligned duplex sessions with an S2 injection channel is reusable machinery — arguably the most valuable artifact here, since duplex training data is the bottleneck for everyone.

Try it

Fastest route: grab the weights from the HF model card (MuyeHuang/DuplexOmni), then:

git clone https://github.com/MuyeHuang/DuplexOmni && cd DuplexOmni/open_source
pip install -r requirements.txt

cd inference_framework/realtime_serving
export VLLM_ROOT=../vllm_qwen3_omni
export THINKER_MODEL=/path/to/thinker/checkpoint
export TALKER_MODEL=/path/to/talker/checkpoint
./start_thinker_talker_stack.sh

# then, live mic/speaker test from a Mac:
python3 omni_realtime_server.py --host 0.0.0.0 --port 8765
python3 omni_realtime_mac_client.py --server ws://127.0.0.1:8765

simulate_v8.py / simulate_video_v8.py offer a no-microphone sanity check first. Paper: arXiv:2606.09186.