Grounding Spoken LLMs in Multi-Speaker Audio via Diarization Conditioning

TL;DR: To make a spoken LLM work on messy far-field meeting audio with multiple overlapping speakers, the authors don't teach the language model anything new. Instead they hand the acoustic encoder a soft "who to listen to" signal — frame-level diarization masks blended into every encoder layer — so the frozen decoder always receives what looks like clean single-speaker audio. The resulting system, Dixtral (a DiCoW encoder grafted onto Voxtral Mini 3B), cuts speaker-attributed word error rates roughly in half versus Gemini 3.0 Flash and Voxtral's own multi-talker model, while keeping the LLM's QA and summarization abilities intact because the LLM was never touched.

The problem: LLM decoders hate cocktail parties

Spoken LLMs — a speech encoder bolted onto an instruction-tuned language model — are getting good at transcribing, summarizing, and answering questions about audio. But they were pretrained almost exclusively on the fiction that audio contains one speaker. Real meeting recordings from a distant microphone contain three to eight people interrupting each other, reverberating, and mumbling.

The dominant fix is Serialized Output Training (SOT): concatenate everyone's transcripts into one token stream, ordered by speech onset, with special speaker-change tokens marking the boundaries. This works fine for conventional encoder-decoder ASR. But applied to an LLM decoder it creates two problems the paper argues are fundamental:

1. Vocabulary surgery. Speaker-change tokens don't exist in the LLM's vocabulary, so you must expand it and retrain at least part of the decoder.
2. Distribution shift. Interleaved multi-speaker token streams look nothing like the tidy single-speaker text the LLM was instruction-tuned on. Closing that gap requires heavy fine-tuning on multi-speaker data — which catastrophically forgets the reasoning, summarization, and QA abilities that were the whole reason to use an LLM instead of a plain ASR model.

There's also a computational argument that's easy to miss but genuinely nice. If S speakers each say N tokens, SOT decodes one joint sequence of length SN, costing \mathcal{O}((SN)^2) in the attention of the autoregressive decoder. Extracting each speaker separately costs S decoding passes of length N: \mathcal{O}(S \cdot N^2) — a factor of S cheaper. Since decoding on a multi-billion-parameter LLM dominates the compute, re-running the small acoustic encoder S times is a bargain.

The idea: steer the encoder, freeze the brain

The reframing is: multi-speaker handling is a perception problem, not a language problem. So solve it in the encoder. Condition the acoustic encoder on a diarization mask — "here is when speaker 3 is talking" — so that it emits representations of only that speaker, and the frozen LLM sees exactly the kind of single-speaker input it was trained on. Want a different speaker? Run the encoder again with a different mask. Want global understanding of the whole room? Set the mask to "everything is target."

This is target-speaker ASR, an established idea, but the paper's contribution is recognizing that it is the LLM-compatible formulation of the multi-talker problem: no new tokens, no decoder fine-tuning, no forgetting.

The mechanism: STNO masks and layer-wise blending

The conditioning signal comes from an off-the-shelf diarizer (DiariZen), which outputs per-frame speaker activity probabilities d(s,t) — the probability that speaker s is talking at time t. From these, for a chosen target speaker s_k, the model computes four probabilities per frame, the STNO mask:

• Silence: nobody is talking — p_{\mathcal{S}}^t = \prod_s (1 - d(s,t))
• Target: only the target speaker is talking
• Non-target: someone else is talking, not the target
• Overlap: the target is talking and someone else is too

These four numbers sum to 1 per frame and give the encoder a richer signal than a binary "target on/off" — critically distinguishing clean target speech from target speech buried under an interrupter.

Rather than masking the audio itself (which would destroy context and mangle overlapped regions), the masks modulate the encoder's internal computation through Frame-Level Diarization-Dependent Transformations (FDDT). Each encoder layer l gets four learnable diagonal affine transforms (\mathbf{W}_i^l, \mathbf{b}_i^l), one per STNO category, and the hidden state at each frame is replaced by a probability-weighted blend:

Intuitively: at frames labeled "non-target," the transform learns to suppress or re-tag content so the decoder ignores it; at "overlap" frames, it learns to emphasize the target's contribution; and because this happens at every layer, the encoder can progressively disentangle speakers rather than making one hard gating decision at the input. It's a soft, learned, per-frame FiLM-style conditioning — cheap (diagonal matrices) and surgical.

One elegant consequence: setting p_{\mathcal{T}}^t = 1 everywhere reduces the model to a normal global encoder, so the same weights handle single-speaker audio, whole-room reasoning, and target-speaker extraction with a one-line change to the mask.

The assembly is opportunistic in a good way. Voxtral trained its modality adapter against a frozen Whisper large-v3 encoder; DiCoW trained its FDDT parameters against a frozen Whisper decoder. Both are built on the same Whisper architecture, so the FDDT modules can be injected into Voxtral's encoder with minimal friction. Training then updates only the encoder and FDDT parameters (LLM and adapter frozen), on eight 24GB A5000s — deliberately academic-scale compute.

The evidence

Speaker-attributed transcription is measured by cpWER (concatenated minimum-permutation WER — find the speaker assignment that minimizes error, then score; it penalizes both transcription and attribution mistakes). Across NOTSOFAR-1, AMI, LibriSpeechMix, and Mixer6:

The macro-average tells the story: 15.4% for Dixtral versus 44.4% for Gemini 3.0 Flash, 35.2% for VibeVoice, and 31.4% for Voxtral's own multi-talker transcription model. Gemini's collapse on far-field AMI (56.3%) and Mixer6 (58.3%) is a useful reminder of how brittle frontier omni-models still are on distant-microphone audio. Dixtral nearly matches the dedicated DiCoW acoustic model (15.4 vs 14.0 average) — the price of the LLM is small — and holds up on Mixer6, which was excluded from training.

The ablations (Table 3, oracle diarization) contain a subtle finding: fully swapping in DiCoW's pretrained encoder converges fastest but ends up worse than keeping Voxtral's original encoder weights and injecting only the FDDT parameters, because replacing the encoder shifts the embedding distribution the frozen adapter and LLM were aligned to. Adding LoRA to the decoder gives the best pure ASR (21.3% NSF-1) but, as we'll see, damages instruction following — a miniature demonstration of exactly the forgetting problem the whole design exists to avoid.

QA and summarization are evaluated on NSF-QA, a new benchmark the authors built on NOTSOFAR-1 and released openly: content questions (entity/topic/yes-no/detail about what a target speaker said), paralinguistic questions (emotion, gender — unanswerable by any cascade that discards audio), and per-speaker summarization. Questions carry no timestamps, so systems must localize the target speaker's contributions across an entire meeting themselves.

Two readings of this table matter. The honest zero-shot comparison: Dixtral on raw far-field mixtures matches Gemini on far-field content QA (54.6 vs 55.1) and matches its parent Voxtral — which was handed close-talk, per-speaker oracle audio — on emotion QA and summarization (ROUGE-L 24.4 vs 24.1). That's the key evidence for the central claim: the frozen decoder's abilities survived intact, and diarization conditioning on far-field audio buys back most of what oracle close-talk audio provides. The flashier numbers — 73.0% content, 95.5% gender after fine-tuning, beating even close-talk Gemini — are real but come from in-domain fine-tuning on NSF-QA's training split while every baseline is zero-shot. They demonstrate headroom, not a like-for-like win.

What to make of it, and where to push back

The framing is the contribution more than any single component. DiCoW existed; Voxtral existed. The insight is that target-speaker extraction is the right interface between multi-talker audio and an instruction-tuned LLM — it converts a decoder-side problem (which forces forgetting) into an encoder-side one (which doesn't) — plus the practical recipe for gluing two frozen-compatible pretrained systems together on modest compute. That recipe is general: any SLM with a swappable encoder could adopt it, and the \mathcal{O}(S \cdot N^2) vs \mathcal{O}((SN)^2) decoding argument gets stronger as backbones grow.

Reasons for healthy skepticism:

• It's not really end-to-end. Dixtral depends on an external diarizer (DiariZen), so it inherits a modular pipeline's failure mode: diarization errors propagate. The ablation table uses oracle diarization and the numbers are notably better there (e.g., AMI 19.8→17.1 for the FDDT-swap variant), quantifying that dependence. The paper cites prior DiCoW work on diarizer sensitivity rather than re-examining it here.
• Baseline fairness is mixed. Gemini gets no diarization signal — it's prompted with awkward positional descriptions like "the speaker who starts speaking third" — so part of the gap on transcription and gender QA reflects interface asymmetry, not just capability. Conversely, the comparison against Voxtral MTv2 and VibeVoice on transcription seems fair, and beating Voxtral MTv2 by 16 points average is Mistral's own model losing to a modified Mistral model, which is telling.
• Judge circularity. NSF-QA references and judging both use Gemini 2.5 Flash. The authors note this should if anything favor Gemini's outputs, making Dixtral's parity conservative — a reasonable argument, but self-generated benchmarks deserve caution regardless.
• The forgetting problem isn't fully banished, just relocated. LoRA on the decoder improves ASR but breaks summarization (ROUGE-L 24.4→15.4, sometimes emitting transcripts instead of summaries), and QA fine-tuning degrades ASR badly on out-of-domain Mixer6 (14.4→26.1). The frozen-decoder configuration is clean, but every deviation from it re-introduces trade-offs. Joint multi-task training is left to future work.
• Scale. This is a 3B model, English-only evaluation, 2-minute training utterances. Whether the conditioning idea holds up at frontier scale, multilingually, and against a Gemini given a proper diarization interface is open.

If the result holds, the practical takeaway is that "who is speaking" belongs in the perception stack as an explicit conditioning signal, not in the token stream as a serialization format — and that far-field meeting understanding (a genuinely valuable, genuinely unsolved product surface) may be reachable by composing existing pretrained pieces rather than retraining an omni-model. The most worthwhile sections are 2.1, for the STNO/FDDT mechanism (five minutes, fully specified by two equations), and Table 3's ablations, which contain the least obvious lesson: when grafting onto a frozen LLM, preserve the encoder the adapter was aligned to and inject only the conditioning parameters.