Helldez/BigMoeOnEdge

TL;DR: BigMoeOnEdge runs MoE models several times larger than device RAM on plain CPU — a 284B DeepSeek V4 Flash (~91 GB on disk) generates on a 12 GB Android phone — by streaming only the experts each token routes to directly from flash storage, losslessly (byte-identical to fully-resident inference). It sits on top of stock llama.cpp's public API rather than forking it, which makes it unusually maintainable, and it ships an APK, a CLI, and honest benchmarks with committed per-token CSVs. The author is explicit that it's "an engineering package of ideas from AirLLM, LLM in a flash, FlexGen, PowerInfer and EdgeMoE, not a novel technique" — the interesting part is how well the package works and how cleanly it's built.

The core idea

Per token, a MoE model touches only its dense trunk plus a handful of routed experts. So the engine loads the model file-backed, hooks llama.cpp's public evaluation callback, and when a layer's router picks its experts, it fetches exactly those weight slices from flash just in time for the matmul. An LRU expert cache (typically 2–4 GB) holds the hot experts; direct I/O (O_DIRECT) avoids the OS keeping a second copy; --overlap issues the next reads while the current layer computes. Nothing about the math changes, so streamed output is byte-identical — enforced by ctest gates on a synthetic tiny MoE, across cache eviction, prefetch, multi-shard reads, and multi-turn sessions.

Two design choices stand out. First, it's a library on llama.cpp's public API — every quant format, tokenizer and chat template works for free, and a new MoE architecture is one registry row (expert layouts are discovered from the GGUF at runtime). The Qwen4-preview model ran the day the weights dropped, before upstream support. The one exception is a ~25-line optional hook for --overlap, dropped once upstream ships an equivalent. Second, the lossless/lossy line is drawn sharply: streaming knobs never change the math; the speed knobs that do (dropping cold experts on cache miss, cache-aware routing à la Skliar et al., reduced top-k) are flagged as lossy, quality-checked on GSM8K, and each documented with its own measured trade-off — including recorded negative results (predictive prefetch lost its on-device A/B by 30% after a simulation predicted a win).

Evidence

All measured on one 12 GB / UFS 4.x phone, Q4_K_M, 256-token greedy decode, best-of per config:

Headline cases: DeepSeek V4 Flash 284B at 0.94 tok/s and Qwen3.8-Flash-Next 125B at 3.5 tok/s, both streamed from multi-shard files with no merge step. The "just past RAM" regime is arguably the more practical win: an 18–22 GB model that mmap turns into a fault storm (0.1–2 tok/s, unstable, kills other apps) runs stably at ~5 tok/s. A desktop snapshot flips the bottleneck: on a 16 GB laptop with NVMe, decode is DRAM-bandwidth-bound (~9 tok/s ceiling), not I/O-bound.

The benchmarking hygiene is unusually good for a hobbyist-scale repo: raw per-token CSVs committed, bootstrapped CIs, caveats stated per table (the Qwen3.6 numbers are a single 96-token run and flagged as such), and telemetry that marks unmeasured quantities as unmeasured rather than zero.

Caveats

• 0.94 tok/s is real-time only in the loose sense; the sweet spot is models 1.5–2× RAM at ~5 tok/s.
• All numbers come from one phone and one laptop; throughput depends heavily on flash read bandwidth and routing locality. A community-benchmark script exists precisely because of this.
• Flash wear from ~100–1800 MiB read per token is unaddressed in the README (reads don't wear NAND like writes, but sustained bandwidth heats phones and thermals visibly move the numbers).
• Best-of protocol numbers; the app reads ~13% lower in real chat.
• Prompt processing / prefill speed isn't reported — likely painful at these sizes.

Apache-2.0, C++, CI on Linux with compile checks on Windows/macOS, prebuilt debug APK per release.

Try it

Easiest: install the APK from the latest release, tap a catalog model (Qwen3-30B-A3B, ~18.6 GB), chat with live telemetry. On desktop:

git clone --recursive https://github.com/Helldez/BigMoeOnEdge.git
cd BigMoeOnEdge && scripts/build-host.sh

build/cli/bmoe-cli -m Qwen3-30B-A3B-Q4_K_M.gguf --moe-stream \
  --cache-mb auto --cache-ceil-mb 4000 --io-threads 4 -t 4 -n 48 \
  --chatml -p "Explain MoE routing."

Omit --moe-stream to see the mmap baseline suffer for comparison; add --overlap --dense-weights anon for models far past RAM.