prism-ml/Ternary-Bonsai-2-27B-gguf

TL;DR: Prism ML has shipped a 27B reasoning model (derived from Qwen3.8-27B, hybrid linear/full attention, 262K context) with every language weight — embeddings, attention, MLP, LM head — in ternary {−1, 0, +1} at a true 1.72 bits/weight, fitting in 5.95 GB and claiming 98.2% of the FP16 benchmark average. If the numbers hold up under independent eval, this is the strongest evidence yet that sub-2-bit weights don't have to destroy reasoning, and it makes genuine 27B-class local inference a laptop-scale problem.

What it is and why it matters

The interesting claim isn't "2-bit quantization" — it's that conventional sub-4-bit builds collapse selectively on long-chain reasoning while looking fine on knowledge benchmarks, and that this build avoids the collapse. The README's own comparison: IQ2_XXS (a "2-bit" build that's really 2.8 bpw at 9.4 GB) still scores 88.9 on MMLU-Redux but falls to 57.5 on AIME26 and 56.4 on LiveCodeBench. Bonsai 2, at 1.72 bpw and 5.95 GB, scores 95.8 and 90.1 on the same — essentially matching FP16 (94.6 / 90.1). They report the same pattern on a second model family (Gemma-4-31B) in the previous release, arguing the collapse is a property of the quantization methods, not the base model. That's a useful, falsifiable insight regardless of this particular model.

How it works

Weights are ternary with one FP16 scale per group of 128 (≈1.71 bits/weight amortized). The trick that presumably makes this viable is a blockwise Hadamard rotation (block 1024, fixed ±1 signs) applied to each weight matrix before ternarization — the QuIP#/QuaRot family of ideas — with the rotation folded into the stored weights offline and the matching transform applied to activations at runtime. GGUF metadata declares the rotation so a runtime either applies it or refuses to load. Only 26.2M parameters (0.098% — linear-attention recurrent state paths and norms) stay in higher precision; the 0.46B vision tower ships as a separate Q8_0 mmproj loaded only for images.

Two packings, and the tradeoff is real: PTQ1_0 packs trits densely (1.75 bpw, 5.95 GB) and wins on bandwidth-starved hardware (4090, Ada, L4); PQ2_0 puts each trit in a 2-bit slot (2.13 bpw, 7.21 GB), cheaper to unpack, and wins on H100/A100/Blackwell where batch-1 decode is instruction-bound, plus prompt processing everywhere.

What the README doesn't say: how the ternary weights are obtained. There's no mention of whether this is pure PTQ, QAT, or distillation/continued training — the whitepaper presumably covers it, but you cannot get these results from naive rounding, so assume significant training compute went in. That also means "derived from Qwen3.8-27B (architecture unchanged)" may understate how much the weights diverged.

Evidence and caveats

Numbers (all self-reported): 84.78 avg across 14 benchmarks vs 86.32 FP16; math 96.57 vs 97.06; coding 89.42 (level); the losses concentrate in MuSR (−9), MMLU-Redux (−2.4), and vision (−5.2). Throughput: 129.9 tok/s decode on an RTX 5090, ~28 tok/s on an M5 Pro laptop at 27.5 W GPU rail, ~47 tok/s claimed on M5 Max (older pre-rotation build, pending re-measurement).

Things to watch:

• Eval/runtime mismatch: benchmarks were run with EvalScope + vLLM on H100, not the shipped GGUF kernels. The GGUF packs you download were not, as far as the card says, the artifacts that were benchmarked.
• Fork lock-in: stock llama.cpp rejects these files — or worse, loads one variant and silently produces garbage because it lacks the Hadamard activation runtime. You need their llama.cpp fork (CUDA/Metal), MLX fork, or mlx-swift fork. Whether these kernels get upstreamed will determine adoption.
• No independent verification yet, and "98.2% of FP16 intelligence" is marketing framing for a benchmark-average ratio. The per-benchmark table is refreshingly complete, though, and the card is unusually honest about true bit-widths (calling out that competing "2-bit" builds are really 2.8 bpw).
• Their "intelligence density" metric (−log₂(1−score/100)/GB) is homemade; ignore it and read the raw table.

Apache 2.0, ~2.2M downloads, active demo repo they call the source of truth for setup.

Trying it

# Their llama.cpp fork is required — stock llama.cpp will not run these files
git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j   # drop the flag on macOS

hf download prism-ml/Ternary-Bonsai-2-27B-gguf Ternary-Bonsai-2-27B-PQ2_0.gguf --local-dir .

./build/bin/llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
  -ngl 99 -fa on -c 32768 \
  --temp 1.0 --top-p 0.95 --top-k 20 \
  -p "Explain quantum computing in simple terms." -n 256

Pick PTQ1_0 instead if you're on a 4090/Ada/L4 or memory-tight. The fastest independent sanity check: run AIME-style problems and LiveCodeBench samples, since that's exactly where low-bit builds historically fake it.