Oído: Open-Vocabulary Speech Recognition on a $5 Microcontroller without a Neural Accelerator

TL;DR: The authors run a 13M-parameter Conformer-CTC speech recognizer — genuinely open-vocabulary, not keyword spotting — entirely on a $5 ESP32-S3 microcontroller with no neural accelerator, hitting 3.70%/8.23% WER on LibriSpeech test-clean/test-other (2.99%/6.67% with a transducer head). That beats Whisper tiny.en and Moonshine tiny running in full precision on a laptop, and makes 2.3–2.8× fewer errors than the vendor's own recognizer on the same chip. The catch: it runs at ~2× real time on one core, so a 4-second command is transcribed about 8.7 seconds after it ends. The interesting engineering is in how they make a 14 MB model run from slow flash through a 64 KB cache, and the paper includes an unusually honest debugging saga about a two-core mode that silently corrupts data on real silicon.

The problem: the model isn't the bottleneck, the memory is

Small, accurate ASR models already exist. NVIDIA's Conformer-CTC Small (13M params) gets 3.7%/8.1% on LibriSpeech greedy — better than Whisper tiny with 3× the parameters. So why hasn't anyone run one on a commodity microcontroller?

Because the arithmetic isn't the hard part. The ESP32-S3 has 512 KB of on-chip SRAM, 8 MB of external PSRAM, and 16 MB of quad-SPI flash, all of which (the external ones) are accessed through a single shared 64 KB data cache with 64-byte lines. Sustained streaming from flash is maybe 30–40 MB/s. The int8 model is 14 MB: it doesn't fit in SRAM at all, only partially in PSRAM (which also needs to hold working buffers), and it fits in flash — but flash is slow. Meanwhile, existing microcontroller runtimes like TFLite Micro are built for static graphs with kilobyte-scale activations, not an attention model running over whole utterances. Vendor solutions on this chip (Espressif's MultiNet7) sidestep the problem by decoding against a fixed command list.

So the design problem is: how do you schedule a transformer-style model so that every weight byte fetched from slow memory does as much work as possible before it's evicted?

The mechanism: three ideas that compose

1. Per-frame dynamic int8 quantization, no retraining. Weights are symmetric int8 with one scale per output channel — standard. The twist is on activations: instead of a static per-tensor scale (the usual edge-deployment recipe), every frame (row) gets its own scale computed at runtime just before the matrix product: s_x = \frac{1}{127}\max_k |x_k|, where x_k are that frame's activation values. This matters because activation magnitudes swing wildly between speech and silence frames; a static scale wastes dynamic range on quiet frames. Accumulation is exact in int32, with rescaling in fp32 on the FPU. Attention is also integer: both terms of Transformer-XL-style relative-position attention are computed as int8 dot products with per-(frame, head) scales, with values stored transposed so the probability–value product is also a SIMD dot product. The pointwise stuff (LayerNorm, SiLU, GLU, depthwise conv) stays fp32 — it's cheap.

Result: post-training int8 loses 0.02 / 0.12 absolute WER points versus full precision. Essentially lossless, no quantization-aware training needed. (For int4 they do need QAT — plain int4 PTQ craters to 14.1%/22.0%; after 8,000 QAT steps it recovers to 4.61%/9.98%.)

2. The bandwidth-aware schedule. Each layer processes the utterance in blocks of 64 frames. A weight row is loaded from flash/PSRAM once per block and reused across all 64 frames, whose int8 activations stay in SRAM. Two cases break this naively: layers with long reductions (K > 1024, where K is the inner dimension of the matrix product) would force shrinking the block and multiplying weight reads — so they split K into ≤512-element chunks and keep int32 partial sums for 16 outputs × 64 frames in SRAM, so every weight byte is still fetched exactly once per block. And the front-end convolution batches three frames per GEMM. Net effect per their traffic model: weight streaming drops from ~18 MB/s to 7.5 MB/s, at a cost of 2.9% more instructions. They deliberately trade compute for bandwidth — the right trade on this hardware.

The SIMD kernels are built around one instruction, EE.VMULAS.S8.ACCX, which does 16 int8 MACs into a 40-bit accumulator and loads the next operand in the same instruction — 16 MACs per instruction in steady state when activations are kept register-resident.

3. Batched greedy transducer search. This is a neat algorithmic trick worth knowing. An RNN-T's joint network must be evaluated at least once per frame, which would stream its 328 KB output matrix 25 times per second. But the prediction-network state only changes when a non-blank token is emitted, and most frames emit blank. So: evaluate the joint for up to B{=}8 frames at once with the current prediction state; if a token appears at frame i, discard the rows after it and recompute with the updated state. Each row's arithmetic is unchanged, so the output is provably identical to sequential greedy search (they verified: identical word errors, substitutions, deletions, insertions on the full test sets), but the joint output matrix is read once per B blank frames instead of once per frame. This is essentially speculative decoding logic applied to transducer blanks, and it's exact.

The numbers

The headline comparison holds up beyond LibriSpeech. On Common Voice, VoxPopuli, and AMI, the on-chip int8 models beat all laptop baselines; they trail Whisper tiny.en slightly only on Earnings-22 (20.8–22.4% vs 20.3%). More convincingly, robustness under real noise and reverb (DEMAND recordings, babble, simulated rooms) also favors the microcontroller model:

To be clear about attribution: the accuracy comes from NVIDIA's openly-licensed Conformer models (trained on ~16,000 hours). The paper's contribution is making them run on this hardware with almost no loss — and the measurement rigor is notable. They run the unmodified firmware in an instruction-counting QEMU emulator; the physical board produces bit-identical activations to the emulator on traced clips, and identical transcripts on all 27 clips tested. Dividing board cycles by emulator instructions gives cycles-per-instruction per pipeline stage (Table 8): a fairly flat 1.3–2.4 cyc/instr everywhere. That flat profile is itself a finding — after the scheduling work, single-core performance is instruction-issue bound, not memory bound. The next factor of two lives in tighter kernels and the second core.

There's also a Spanish model (fine-tuned from the English weights on 2,492 h of open data) that makes 34–60% fewer errors than multilingual Whisper tiny on four Spanish test sets — though the comparison favors them, since they trained on those corpora's training splits and Whisper is zero-shot.

The two-core ghost story

Section 8.1 is the most interesting debugging writeup I've seen in an ASR paper, and worth reading in full. The two-core parallel schedule is correct in the emulator and projected a real-time factor of 0.7–0.95 — below real time. On silicon at 240 MHz it produces randomly corrupted activations (always first in layers 1–2, almost always the feed-forward block) and crashes about one run in eight with corrupted backtraces. At 160 MHz, two cores are perfectly correct — but no faster than one core at 240 MHz.

They ruled out an impressive list: PSRAM speed, cache line size, weight location, semaphore implementation, interrupts, tile placement and sharing, instruction fetch location, and the parallelization split itself. Only the K{=}704 matrix products — the ones moving the most weight data per cycle — corrupt. A supply/timing margin problem fits the symptoms, except a separate synthetic two-core stress of the same shape on the same board ran clean for 21 minutes. They conclude, honestly: we do not know the cause. And they add the sobering note that even a correct two-core build wouldn't reach real time: the one build they could time ran at 0.78× the one-core time, not the 0.55× instruction counts promised, because the cores contend for memory.

Streaming, and what to be skeptical about

The streaming variant fine-tunes the model for chunked attention (dynamic chunk sizes, cache-based int8 KV and conv state per layer, ~2 MB fixed-size caches) so that when the speaker stops, only the last partial chunk remains to compute. It shows partial text while the speaker talks and roughly halves the final-text delay (4.1–11.1 s vs 7.2–22.4 s after clips of 3.6–10.6 s), at a real accuracy cost: 4.86%/11.03% at chunk size 32 vs 3.33%/7.15% in utterance mode. One nice empirical nugget: noise augmentation during streaming fine-tuning mattered more than anything else — a clean-speech fine-tune destroyed robustness (12.0% mean over noise conditions); adding MUSAN noise and simulated reverb brought it to 9.0%, more robust than the original model.

Caveats to hold onto:

• Not real time. RTF ≈ 1.97 on one core. A 4 s command → text ~8.7 s after it ends (utterance mode), ~5 s streaming. The paper is admirably blunt that "neither is conversational." This limits it today to short commands where several seconds of delay is tolerable.
• One board. All silicon measurements come from a single ESP32-S3 module, with 10–27 clips per configuration, max 10.6 s. The two-core failure might be board-specific.
• Accuracy numbers come from the host build of the same C code; the board agrees with the emulator exactly, but the host's libm rounds log-mel features differently, changing a word on some hard clips (22/27 identical; the Spanish board result is notably worse, 9.3% vs 7.3% on 12 clips).
• Baseline hygiene is mixed: laptop baselines on 500-utterance subsets, MultiNet7 numbers are vendor-reported with an unspecified protocol, and some baselines (Moonshine, NVIDIA's models) may have seen the "held-out" sets in training. No comparison against TFLite Micro or ESP-DL deployments of the same model, so the engine's advantage over existing runtimes is argued, not measured.
• The closest prior microcontroller result — Arm's int8 Conformer-S — reports 6.4% test-clean with an Ethos-U85 neural accelerator. Oído's 3.70% without one is, per the authors, the lowest published microcontroller LibriSpeech WER with or without an accelerator.

Why it matters

If this holds up (code, models, and board measurements are all released — GPLv3/CC-BY), the practical implication is that the "keyword spotting ceiling" on commodity microcontrollers was never a model-capacity limit, just a systems problem: per-frame dynamic quantization is essentially free for Conformers, and a bandwidth-first schedule makes a 14 MB attention model viable behind a 64 KB cache. For anyone building voice interfaces into toys, appliances, or wearables, full-vocabulary recognition at $3.41 per unit in volume, with zero cloud cost and no audio leaving the device, is a different design space than "pick your 20 commands" — provided a few seconds of latency is acceptable, or until someone closes the remaining 2× (the paper's own analysis says faster kernels plus a working second core should get there).

What to read: Section 5.3 (the bandwidth-aware schedule) and Section 5.4 (the batched transducer search — an exact, reusable trick) are the technical core. Section 8.1 (the two-core failure) is the most memorable five minutes, and a useful cautionary tale about trusting emulators for parallel code on cheap silicon.