microsoft/VibeASR.cpp

TL;DR: VibeASR.cpp is Microsoft's official CPU inference runtime for VibeVoice-ASR-BitNet — an LLM-decoder ASR model (acoustic VAE tokenizer + Qwen2.5-1.5B ternary-quantized à la BitNet) squeezed to 1.58 GB that runs real-time multilingual transcription on ordinary CPUs, reportedly 1.6–2.3× faster than Whisper.cpp at comparable size. It's interesting as the first serious demonstration that BitNet-style 1.58-bit quantization transfers to speech recognition, and as a plausible new default for offline/edge ASR.

What it is and why it matters

The recipe here is heterogeneous quantization matched to component sensitivity: the acoustic VAE tokenizer gets 8-bit (I8_S), the Qwen2.5-1.5B language-model decoder gets ternary I2_S weights (with Q6_K embeddings), all in GGUF running on custom SIMD kernels with operator fusion in ggml. That takes the FP16 model from 4.62 GB to 1.58 GB (2.9×), with the LM compressed 3.6×. The 7B→1.5B swap plus quantization together cost a claimed 1–4% absolute WER versus the FP16 7B — mostly true in their tables, with exceptions noted below.

The architectural point worth registering: this is LLM-decoder ASR, not an encoder-decoder transducer like Whisper. That shows up in the accuracy profile — it wins big on hard multilingual and far-field meeting data, and loses on clean read English.

The evidence

Real-time factors (lower is better, <1 is real-time on streaming input):

Accuracy is where you should read carefully. Versus the FP16 7B parent, the BitNet 1.5B loses 0.2–2.5 WER points on most sets — genuinely modest. Versus baselines, the story is bifurcated:

• Wins: meeting/far-field and multilingual — AMI-sdm 25.87 vs Whisper's 36.92; AMI-ihm 21.36 vs 27.07; MLC-EN 8.25 vs Whisper 13.57; VoxPopuli 5.18 vs 7.19.
• Losses: clean English read speech — LibriSpeech test-other 6.27 vs Parakeet's 3.13 and Whisper's 3.60. Fleurs-en 5.21 vs Whisper 3.99.
• Red flag: AISHELL4 jumps from 19.83 (7B FP16) to 27.45 — a 7.6-point degradation, well outside the "1–4% absolute" claim, and worse than SenseVoice and FunASR. Chinese meeting speech seems to be where the compression hurts most. The README itself warns performance on accented/dialectal speech "may degrade more significantly."

Also note the Whisper.cpp speed comparison says "at comparable model sizes" without naming which Whisper checkpoint or quantization — check the tech report before quoting the 2.3×.

What's actually in the repo

C/ggml runtime with a CLI (asr_infer), a Gradio web demo, conversion scripts (SafeTensors → F32 GGUF → I8_S/I2_S via a bundled llama-quantize), and pre-quantized weights on HuggingFace (microsoft/VibeVoice-ASR-BitNet). MIT licensed. There's a tech report on arXiv. Maturity caveats: the repo is two weeks old, MSVC is explicitly unsupported on Windows (MinGW required, with the Windows demo needing hand-edited MODEL_CONFIGS), and the CLI appears file-based — no streaming API is described in the README despite the "real-time" framing, so RTF < 1 here means "faster than audio duration," not a live mic pipeline out of the box.

Try it

git clone --recursive https://github.com/microsoft/VibeASR.cpp.git
cd VibeASR.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j

huggingface-cli download microsoft/VibeVoice-ASR-BitNet --local-dir models/vibeasr

./build/bin/asr_infer \
  --vae-model models/vibeasr/vibeasr-vae-encoder-i8_s.gguf \
  --lm-model  models/vibeasr/vibeasr-lm-i2_s-embed-q6_k.gguf \
  --audio input.wav -t 4

~2 GB disk, any AVX2 or NEON CPU. If your workload is multilingual, noisy, or meeting-style audio on CPU-only hardware, this looks like a real step up from Whisper.cpp; if it's clean English, Parakeet or Whisper still win on accuracy. Verify on Chinese and accented speech before committing — that's where the ternary compression visibly bites.