IST-DASLab/disaggregated-quantization

TL;DR: IST-DASLab (Alistarh's group, with Blankevoort) proposes disaggregated quantization: use different quantization formats for LLM prefill and decode, because prefill is compute-bound (wants hardware-native low-precision matmuls like NVFP4) while decode is memory-bound (wants the smallest possible weight encoding). The headline trick: train a small NVFP4 "prefiller" around an existing frozen 1–2 bit GGUF decoder, stream it from SSD only during prompt processing, and more than double accuracy of Unsloth's IQ1_S Qwen3.8-27B (29.0% → 61.5% MMLU-Pro) while nearly halving TTFT at 8K context — with zero change to the decoder.

The idea

Every quantization scheme today picks one format and lives with it for both phases of inference, even though the phases reward opposite choices. Prefill throughput comes from native low-precision arithmetic (NVFP4 tensor cores); decode throughput comes from minimizing weight bytes read per token (LUT2/LUT3, GGUF sub-2-bit codes), and quantizing decode activations buys nothing. DQ decouples the two: format, weights, and even storage location are specialized per phase, and both pathways are trained toward one response objective.

Three variants, in increasing cost: format disaggregation (shared weights, NVFP4 compute in prefill, weight-only in decode — free, and improves 11 of 13 model–benchmark PTQ comparisons on models up to 2.8T params without retraining); full disaggregation (separate prefill and decode checkpoints); and ODP, which recovers full-disaggregation accuracy without extra GPU weight residency by streaming prefill weights from SSD.

Training is QADD: quantization-aware distillation where the SFT token mask selects the phase-specific linear pathway and the response targets, so gradients flow to prefill weights through the representations decode consumes — both pathways trained in one forward-backward pass. It also supports a frozen external decoder, which is what makes retrofitting existing GGUF releases possible.

Evidence

The frozen-decoder result is the striking one:

TTFT at 8K context drops 12.27s → 6.90s in their custom llama.cpp integration (1.38–1.78× over native llama.cpp across 4K–32K); short prompts are slower because SSD loading dominates. In the core Qwen3/Gemma3 experiments (0.6B–12B), full disaggregation at 2-bit decode beats trained weight-only baselines by 7.1/4.5 points on decode-heavy tasks (GSM8K, MATH-500, MMLU-Pro) and 12.6/8.9 on prefill-heavy RULER. ODP overhead is under 5–8% versus resident prefill weights at ≥16K context.

What's actually released

Real, but research-grade: QADD training code, phase-specific quantizers and export, disaggregated vLLM serving, LUT2/LUT3 CUDA+Triton kernels, NVFP4 prefill and SSD-offload benchmarks, a custom llama.cpp integration, plotting notebooks with recorded results, and Qwen3.8-27B NVFP4 prefiller checkpoints on HF (ISTA-DASLab/Qwen3.8-27B-NVFP4-prefiller). Launch scripts assume SLURM + containers (2 nodes × 4 GPUs default for training); no license is stated in the README.

Honest caveats, mostly flagged by the authors themselves: accuracy evals dequantize GGUF to BF16 in vLLM (the size axis is encoded weights, not what the eval backend allocated); TTFT was measured on a DGX Spark; error bars reflect checkpoint variation, not seeds; batched serving, multi-turn cache rebuilds, and agentic workloads are unevaluated — and batched serving is exactly where ODP's SSD streaming and buffer carve-outs could get awkward. Some higher-precision formats lose a little MMMU-Pro accuracy after training.

Try it

Easiest entry: grab the prefiller checkpoints and pair with Unsloth GGUF decoders. To train on your own model (cluster required, paths need adapting):

cd qad
# Shared-weight format-disaggregated NVFP4 — the "free" variant
MODEL=Qwen/Qwen3-4B ./bin/run_qad.sh --quantizer nvfp4pdshared
# Frozen-decoder prefiller: --quantizer nvfp4frozendec with
# decode_model passed via --quantizer-params

Worth watching: the framing (phase-specialized formats + storage placement) is orthogonal to nearly every existing quantization method, which means it composes with them — and the no-retraining PTQ variant could plausibly land in vLLM proper.