Nehanth/swarmllm

TL;DR: SwarmLLM runs a 27B LLM split across the browser tabs of whatever devices are in the room — laptop, phone, PC — with zero installs: a from-scratch WebGPU engine (~50 WGSL kernels) plus a WebRTC runtime that passes a 10 KB activation vector per token between peers. The headline demo is Qwen 3.8 27B (15 GB Q4_0) at 10.7 tok/s across a MacBook and an iPhone, and the engine itself measures faster than native llama.cpp on at least one GPU, which is worth your attention on its own.

What it is and why it's interesting

Distributed layer-sharding of LLMs is not new (Petals, exo, llama.cpp's rpc-server). What's new is the deployment substrate: nothing to install, any OS with a WebGPU browser, join by opening a URL. Each device downloads only its layer slice (cached), signaling goes through PeerJS, and inference traffic is direct WebRTC — no server ever touches tokens or activations. That collapses the entire "get a local LLM running on a heterogeneous fleet" problem into a room code.

The second surprise is the engine. This is not a WebLLM/MLC wrapper — it's hand-written WGSL that hits the memory roofline. On an NVIDIA GB10 (headless Deno/Vulkan), it decodes at 9.0 tok/s plain and 16.1 tok/s with speculative decoding, versus 8.0 tok/s for a CUDA llama.cpp build on the same GGUF file, same machine. The README's explanation is credible: decode is memory-bandwidth-bound, their WebGPU buffer reads reach 183 of 184 GB/s measured, and the CUDA path doesn't. WebGPU beating native on a real workload quietly overturns the "browser inference = toy" assumption.

How it works

The host embeds the last token into a 5,120-float hidden state (10 KB as f16), ships it over WebRTC through peers that each run a contiguous slice of layers on their own GPU, gets it back, applies the final norm and LM head, and samples. A multi-token-prediction draft head guesses ahead; drafts are verified in a single batched pass with exact rollback of the recurrent state, so each network lap carries several tokens instead of one. Prefill batches 16 tokens per round trip.

The model is Qwen 3.8 27B, a hybrid Gated-DeltaNet + attention architecture — notable because the DeltaNet recurrence makes prefill serial, which is the system's known weak spot (44 tok/s prefill vs. 377 for llama.cpp on the GB10). Correctness discipline is a highlight: every optimization is gated on golden tests, and the speculative path is verified bit-identical to plain greedy decoding.

Numbers

Degradation over real links is graceful rather than free: 10.8 tok/s solo → 7.7 over same-room Wi-Fi → 3.5–6 across the internet. The speculative decoding is doing a lot of the work in keeping multi-hop viable.

What's there, and caveats

MIT-licensed JavaScript, no build step — the repo is a static site. Real engine code (GGUF parsing, quant repacking, DeltaNet WGSL kernels, autotuning), a test suite with GPU golden tests, benchmark harnesses, a public bench log with commits, plus docs on architecture/kernels/protocol and a per-item roadmap. A hosted room at swarmllm.ai/room.

Caveats: it's a week-old project by one author; Chrome-on-macOS is the only tested host (Safari reloads under memory pressure with the big slice; Firefox/Linux untested); the model roster is essentially Qwen-only; prefill is slow by design of the DeltaNet recurrence. And note the honest privacy framing — a room is a shared conversation, and mid-model activations are explicitly not private against a determined peer. No obvious overclaiming; the README states its gaps plainly, which is rare.

Try it

Fastest path: open swarmllm.ai/room, create a room, share the code, pick a model. Or serve it yourself:

git clone https://github.com/Nehanth/swarmllm && cd swarmllm
npx -y serve -l 8080 .   # then open http://localhost:8080/room

If you have unusual hardware, they explicitly want benchmark reports — npm run bench:q38 with model files under models/ (needs Deno 2.x and a WebGPU GPU).