Qwen/Qwen3.8-Flash-Next

TL;DR: Qwen3.8-Flash-Next is an open-weight, image/video-capable multimodal LLM that serves as a public preview of the Qwen4 architecture: 125B total parameters with only 6B activated, mixing Gated DeltaNet linear attention with block-sparse full attention, plus a 51B-parameter n-gram embedding table pitched as a cheap, offload-friendly axis for parameter scaling. It's worth your attention less for the scores (which are good) than as a concrete statement of where one frontier lab thinks efficient long-context architecture is going.

What's actually new here

Four things, per the card:

Hybrid attention with QSA. The layer stack is 48 layers arranged as 12 × (3 × (GatedDeltaNet → MoE) → 1 × (QSA → MoE)) — three linear-attention layers for every one "real" attention layer. The real attention is now Qwen Sparse Attention (QSA), which selects at the micro-block level rather than per-token, using a lightweight MQA indexer (4 query heads, 1 shared key head, dim 128) with a budget of 512 blocks / 2048 attended tokens. This is in the same family as DeepSeek's sparse-attention indexer, but with coarser block granularity, and the 3:1 ratio keeps the KV cache tiny (only 2 KV heads at dim 256 in the QSA layers).

N-gram embedding as a scaling axis. A 20M-entry bigram/trigram embedding table (51B params) injected at layer 2. The framing is the interesting part: embeddings are lookup-only, so this is parameter scaling that costs almost no FLOPs and — unlike MoE experts — can be offloaded to host memory with predictable access patterns. It's the "memory layers" idea taken seriously at scale in a production-track model.

Gated Residual. The residual stream is widened into 4 branches (bottleneck rank 320), with an element-wise data-dependent read gate and a per-branch scalar write gate. Claimed benefit: finer per-layer control over what flows through the residual without destabilizing deep training.

Training recipe. Muon + AdamW split by weight category, no batch-size warmup (start directly at target batch size, guided by refitted scaling laws), enabling larger learning rates and fewer optimizer steps.

Evidence

All numbers are vendor-reported. The agentic results are the standout claim — a 6B-active model beating much larger dense/MoE baselines:

Also strong on general reasoning (GPQA Diamond 91.7, LiveCodeBench v6 91.9) and vision (ERQA 72.3, RealWorldQA 88.5, LVBench 76.6). Grains of salt: several benchmarks are in-house (CoWorkBench, RecreationBench), some public benchmarks were "corrected" and baselines re-evaluated by Qwen themselves, and the headline claim — long-context latency reduction from QSA — has no numbers in the card at all. The architecture-ablation evidence presumably lives in the technical report.

What's there and what to expect

Weights in safetensors, config, and a thorough deployment guide (Transformers, vLLM, SGLang, TokenSpeed) are in the repo. No training data, and the license is "other" — check it before commercial use. This is explicitly an experimental preview; the production version (Qwen3.8-Flash) is API-only.

Practical notes: ~180B params total means multi-GPU or aggressive offload (the n-gram table is the natural offload target — that's the point). Native context is 262K, extended to 1M via static YaRN with documented rope_parameters overrides, with the usual caveat that static YaRN hurts short-context performance. Thinking mode is on by default with a reasoning_effort knob (low/medium/xhigh) and a preserve_thinking option that keeps reasoning traces across turns for agentic KV-cache reuse — a nice API-level detail worth stealing.

Try it

vllm serve Qwen/Qwen3.8-Flash-Next
pip install -U openai
export OPENAI_BASE_URL="http://localhost:8000/v1" OPENAI_API_KEY="EMPTY"

from openai import OpenAI
client = OpenAI()
r = client.chat.completions.create(
    model="Qwen/Qwen3.8-Flash-Next",
    messages=[{"role": "user", "content": "Write a Python function to merge two sorted linked lists."}],
    reasoning_effort="xhigh",
)
print(r.choices[0].message.content)

Model card: https://huggingface.co/Qwen/Qwen3.8-Flash-Next