jd-opensource/JoyAI-VL-Interaction

TL;DR: JD's Joy Future Academy has released an 8B vision-language model trained to be proactive — it watches a live video stream continuously and decides every second whether to speak, stay silent, or delegate a hard subtask to a background agent, rather than waiting to be prompted. What makes it worth your attention is the completeness of the release: weights (plus INT4/INT8/FP8/NVFP4 quants), the training recipe, 4M time-aligned interaction samples, and a full WebRTC streaming deployment stack, all Apache-2.0.

What it is and why it's interesting

Every current video-language product is turn-based: it answers when addressed. JoyAI-VL-Interaction inverts this — the model is "present" in a stream, and interactivity is trained into the model rather than bolted on by a harness of heuristics (VAD, keyword triggers, polling loops). The core learned behavior is a per-second three-way decision: speak, stay silent, or delegate. The team's claim, backed by their evals, is that once you train this decision natively, capabilities like acting at the right moment, sensing elapsed time, and long-stream memory come along for free.

The other notable claim: a compact 8B open model beats Doubao and Gemini's video-call products in its "advantage zone" (proactive, event-driven interaction), while the README is refreshingly upfront that those products win at open-ended chat and general knowledge.

How it works

Built on JoyAI-VL-8B. Speech is deliberately pluggable I/O (Qwen3-ASR in, Qwen3-TTS out), so the core model only does vision + timing judgment. Behavior is learned via SFT on 4M+ time-aligned clips — data where the when of a response is supervised, not just the what — then refined with RL (LLaMA-Factory + EasyVideoR1). A predictive video codec, AdaCodec, spends few tokens on predictable frames and full detail on scene changes, keeping the token budget viable over long streams — though note the AdaCodec model variant is the one unchecked TODO; the currently shipped model doesn't include it.

The deployed system is five pluggable services on standard vLLM infrastructure (Day-0 vLLM-Omni support): inference, browser WebUI with WebRTC, ASR, TTS, and a background delegation agent. A LiveKit branch exists to reduce exposed ports, but they flag it may not be maintained.

Evidence

Evaluation is 58 real, event-driven interaction settings, pairwise human-judged on quality and timing:

Caveats: this is a self-run eval, and per-category sample sizes look tiny (win rates like 77.8% imply n≈9). The comparison is also deliberately in the model's advantage zone; the README concedes larger products win elsewhere. On offline video understanding, the unified model averages 57.53 across 26 standard benchmarks vs 54.16 for Qwen3-VL-8B-Instruct — respectable, showing the interaction training didn't cost offline capability, but not the headline.

The more interesting soft evidence: emergent behaviors from small interaction data (e.g., guiding a shopper through changing app screens without explicit training), and the claim of "clear gains from scaling further" on the 4M samples.

What to expect

• Maturity: ~2.5 months old, actively updated, 1.8k stars. Full stack is released: weights, quants, data, recipe, system code, tech report PDF. Apache-2.0 on the repo; check the dataset card separately if you want to train derivatives.
• Hardware: tuned inference configs for RTX 3090/5090 exist, and INT4/NVFP4 quants target lighter deployment — a single consumer GPU looks plausible.
• Gaps: AdaCodec variant not yet shipped, so long-stream token cost with the current model is unverified; latency ("under a second when needed") is a claim you should measure on your own streams; the eval is small-n and self-judged.
• Dependencies: vLLM, Qwen3-ASR/TTS adapters, browser WebRTC frontend (built on NVIDIA's Live VLM WebUI).

Try it

git clone https://github.com/jd-opensource/JoyAI-VL-Interaction.git
cd JoyAI-VL-Interaction
./install/install.sh --with-all
./install/download-models.sh --all
./services/scripts/run.sh minimal
# open https://127.0.0.1:8099

Weights at jdopensource/JoyAI-VL-Interaction on Hugging Face; there's also a hosted API via JD Cloud if you'd rather not deploy. There's an RTSP streaming guide in doc/ for pointing it at arbitrary video sources — the fastest way to stress-test the speak/silence judgment yourself.