lokutor-ai/oido

TL;DR: Oído runs open-vocabulary English and Spanish speech recognition — NVIDIA's Conformer-CTC Small, int8-quantized, with optional on-chip language-model beam search — entirely on a $5 ESP32-S3 microcontroller with no NPU and no cloud, claiming 3.7% LibriSpeech test-clean WER (3.3% with the LM). That's better than Whisper tiny.en and Moonshine tiny running on a laptop, in 14 MB of flash and 2.4 MB of PSRAM. The big caveat: accuracy is verified via a bit-identical host build and QEMU, but real-time speed on physical silicon is still an estimate.

Why this matters

Microcontroller speech has been synonymous with fixed command lists (Espressif's MultiNet7 scores 8.5/21.3 on LibriSpeech and takes a command vocabulary). Oído is full open-vocabulary transcription on the same chip, at WERs that were laptop-class two years ago. Arm has shown Conformers on Cortex-M55 — but with an Ethos-U NPU attached. This is plain dual-core Xtensa at 240 MHz, using the ESP32-S3's PIE vector unit (16 int8 MACs per instruction). If the speed numbers hold on hardware, always-on private transcription becomes a commodity-BOM feature for wearables, appliances, and voice agents.

How it works

The model is stock: log-mel → 2× strided conv subsampling to 25 Hz → 16 Conformer layers (d=176, 4 heads, relative-position attention) → CTC over 1024 BPE tokens, plus an optional 1.3M-param GRU LM for prefix beam search (beam 4) and an RNN-T head (you convert NVIDIA's transducer weights yourself; it gets 3.0/6.7).

The engineering is in the C engine (esp32/components/tinyasr): int8 matmul kernels on EE.VMULAS.S8.ACCX, int4 outer-product kernels, a lookup-table softmax for quantized attention, dual-core scheduling, and — the key move on a flash-bandwidth-bound chip — tiling so each weight is streamed from flash once per 64 frames, cutting weight traffic from 18 to 7.5 MB/s. int8 quantization costs essentially nothing: 3.70/8.23 on chip vs 3.68/8.11 fp32. The int4 model (4.6/10.0) is a quantization-aware fine-tune that fits in 8.3 MB, leaving a 6 MB app partition for your own code.

There's also a streaming mode (oido_stream.tnm): the encoder processes 1.28 s chunks with 5 s left context while you speak, so final text lands ~1.1–1.4 s after you stop instead of ~3 s, at the cost of accuracy (4.9/11.0 with LM). The same file runs in full-context mode at 3.4/7.8. Robustness was benchmarked too: 7.5% mean WER across 14 noise/reverb conditions vs 12.1 for Whisper tiny.en.

Spanish is a 2,492-hour fine-tune (Common Voice, MLS, VoxPopuli, FLEURS) with its own vocabulary and LM: 13.3/10.6/15.5/11.3 with LM. The README is unusually honest that this comparison flatters them — Oído trained on these corpora's training splits, Whisper is zero-shot.

What's actually there, and what isn't

Everything is in the repo or on Hugging Face: C engine, ESP-IDF firmware, host build of the same engine, PyTorch training/export code (including int4 QAT and the streaming fine-tune), eval scripts, raw benchmark JSONs, audio samples with per-clip compute, and all model files (int8, int4, streaming, Spanish). No weights withheld except NVIDIA's transducer (NGC terms).

The honest gap, flagged prominently: no physical-board measurements yet. WER comes from a host build compiling the same C with the same int8 arithmetic; QEMU runs the real firmware and matched the host transcripts on 22/27 clips (last-bit libm rounding differences on hard noisy audio). Instruction counts are exact from QEMU -icount, but real-time factor (estimated 0.7–0.95 utterance, 0.80–1.00 streaming) depends on flash/PSRAM stall behavior that only silicon reveals. Streaming at 16-frame chunks may not keep up. The demo video is engine output, not board footage. Board numbers are promised "in the next days."

Licensing: code is GPL-3.0 with a commercial option (classic dual-license play by Lokutor, who also sell other languages and on-chip TTS). Weights are CC-BY-4.0 or CC-BY-SA-4.0. Requires the N16R8 variant (16 MB flash, 8 MB octal PSRAM) — still a few dollars.

Try it

The host build runs the exact chip engine, so you can evaluate accuracy without hardware:

git clone https://github.com/lokutor-ai/oido && cd oido/esp32/host && make
./tasr_cli ../../models/nemo8.tnm recording.wav   # 16 kHz mono PCM16
python live_demo.py                   # mic → firmware VAD + engine, with ESP32 time estimates
python live_demo.py --model stream    # words appear while you speak
python live_demo.py --model es        # Spanish

On hardware: an ESP32-S3-DevKitC-1 N16R8 plus an INMP441 I2S mic, ESP-IDF v5.5, then esp32/tools/flash.sh /dev/ttyUSB0 models/nemo8.tnm. The TASR_MODE=file flash variant prints measured RTF — the first thing worth checking before trusting the latency story.