Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gains

This paper takes the most common sentence in ML—"our method improves benchmark accuracy by X points"—and asks a question almost nobody asks: did the model gain the ability to answer those questions, or did it just start emitting answers it could already produce? The authors build a question-level audit that separates the two, and find that in the cases they check, the celebrated gains from RL-with-verifiable-rewards come almost entirely from the second category. In the most striking case, DAPO training raises deployed accuracy by 14.7 points while the set of questions the model can reach at all under a fixed sampling budget shrinks by 13.3 points.

Two numbers hiding inside one score

Here's the setup. A benchmark score is a single number: fraction of questions answered correctly by the default deployment procedure (greedy decoding, one sample, the standard prompt). Call a question realized if that default procedure gets it right. Formally, for question i with grading function g_i(a) \in \{0,1\} and default output \hat{y}_i^{\mathrm{dep}}:

Now give the model more chances. Run a specified probe—say, sample K completions at temperature 0.8, or run the model with K different layer configurations—and check whether any candidate is correct:

A question is reachable if o_i(K) = 1. Averaging over questions gives deployed performance D and the "reachable ceiling" O_K. If you've followed the pass@k literature, O_K is essentially pass@K measured under a rigidly fixed protocol (same prompts, temperature, format, grading, budget on both sides of every comparison), and the paper's contribution is to track d_i and o_i(K) per question across interventions, rather than comparing aggregates.

The point of the decomposition: a benchmark gain of, say, +10 points is consistent with two completely different stories. Story one: the model's reachable set expanded—it can now find answers it couldn't before. Story two: the reachable set didn't move (or shrank), and training just redistributed probability mass so that already-reachable answers now come out on the first try. Aggregate scores cannot tell these apart. The paper's audit can, because it joins the same questions across checkpoints.

One honest caveat the authors state up front: reachability is defined relative to a protocol—question set, format, temperature, probe, budget K. An answer no probe finds within budget is "unobserved under that protocol, not impossible for the model." So O_K is a measured ceiling, not the model's platonic capability. The audit's power comes from holding the protocol rigidly fixed while varying only the intervention.

Warm-up: routing gains were selection, not search

Before getting to training, the paper does a clever calibration exercise using layer routing—running inference with a modified sequence of transformer layers (skipping, reordering) without touching weights. Several recent works search for input-specific layer paths and report gains. Are those searches discovering structure?

The control: compare structured search (MCTS, hill climbing, genetic search) against purely random layer paths at a matched budget. Result: across all 43 clean model–task cells, the random-path oracle matched or exceeded the structured-search ceiling. On Llama-3.2-3B / ARC-Challenge, random paths reach 100% oracle accuracy at K=200. The structured search saturates earlier and lower.

But note the word "oracle." Identifying which random path produced the correct answer requires knowing the answer. When the authors test answer-blind selectors—majority voting, learned routers, and four other channels that see candidates and model scores but not gold answers—the gains evaporate:

At K=32, five of six answer-blind channels had negative median recovery of the oracle–greedy gap; the best was +2.2%. And the same pattern shows up with ordinary temperature sampling, no routing at all: Llama-3.1-8B on GSM8K scores 13.3 deployed, 18.7 with majority-vote@8, but 44.7 with oracle@8—a 26-point gap between "a correct answer exists in the candidate pool" and "we can find it blind."

The lesson from this section is methodological: any evaluation that implicitly uses answer-aware selection (including some search-based routing results) is measuring reachability, not deployable capability, and random baselines under matched budgets are mandatory. The gap between the two is enormous and doesn't close with clever blind selectors.

An interlude: why reachable answers fail to appear

If a correct answer is in the model but doesn't come out, where does it get blocked? The paper predefines a specific failure mode: the model ranks the correct option first under multiple-choice likelihood (recognition) but generates a wrong answer directly. Scanning 284 checkpoint–task cells, 27 exhibit this pattern—bounded, but recurring.

For six such cases spanning 0.5B to 31B parameters, they do causal surgery: silence (zero out) one identified MLP block. This repairs 67.6–92.0% of the predefined failure set, while breaking only 0–12% of a separate healthy panel. Silencing attention in the same layer does much less in all six cases. And the intervention is reciprocal: pushing the same block's activations in the opposite direction induces the failure in a healthy model (up to 97.5% induction). They can even manufacture the failure mode from scratch via magnitude pruning—Llama-3.1-8B develops a sharply localized failure with a 96.6% repair peak at layer 20.

The authors are careful not to overclaim a universal circuit: a directional top-32-unit subset recovers ≥87.5% of the whole-block effect in four cases but only 5.6% and 34.3% in two others. Still, this is a clean demonstration that "the answer is in there but a specific component vetoes it" is a real, causally localizable phenomenon—one concrete mechanism by which reachability can exceed realization. It connects to a growing interpretability literature on late-layer suppression of probe-visible correct answers.

The main event: RLVR realizes, it doesn't expand

Now the training question. RLVR (reinforcement learning with verifiable rewards—the recipe behind DAPO, SimpleRL-Zero, and much of the current reasoning-model wave) rewards correct final answers. Does it expand the reachable set, or just concentrate probability on it?

The audit joins base and trained checkpoints question-by-question on DART-Math and GSM8K, with matched prompts, temperature 0.8, n = 300 questions, K = 200 samples per question per checkpoint. The headline numbers:

Three findings, in increasing order of consequence:

Every newly realized question was already reachable in the base model. Across the four clean matched cells, all 11–68 questions per cell that flipped from wrong-to-right after training were questions the base model could already answer within K=200 samples. Zero came from outside the base reachable set. Meanwhile 6–24 previously-correct questions per cell were lost.

The gain concentrates on high-propensity questions. Among base-reachable questions the base model didn't produce by default, 85–92% of those the base sampled correctly more than 35% of the time became realized after training; low-propensity questions converted far less often. RLVR is sharpening a distribution, and it sharpens most where the mass already was. This is exactly what you'd expect if the objective is "make the modal sample correct"—it's a rich-get-richer dynamic over the base model's answer distribution.

The reachable ceiling can actively contract. For DAPO, O_{200} drops from 96.0% to 82.7%. Doubling the budget to K=400 doesn't recover it (trained ceiling stays at 85.3% in that control while the base rises slightly to 96.0%), so this isn't a low-budget sampling artifact—those 40 questions appear genuinely gone from the trained model's accessible distribution, at least under this protocol. SimpleRL, by contrast, holds its ceiling roughly flat (-0.7 pp). So contraction is not inevitable; it depends on the recipe.

The recipe-dependence gets its own control, and it's one of the most interesting tables in the paper. One OLMo lineage, four training variants, same evaluation:

Math-only RLVR: deployed up (12.7 → 25.3), ceiling down (74.0 → 60.7). Both SFT-containing recipes: deployed and ceiling up. The pattern is coherent with a simple picture: SFT injects new behavior from external traces (genuinely expanding what the model can emit), while pure on-policy RLVR can only reweight what the model already samples—and aggressive reweighting can crush the tails where rarely-sampled correct answers live.

One more honest control: prompt format. Trained models often use different answer formats, and format alone could inflate realization. Using a shared base prompt retains 41–50% of the native-format realization gain—so formatting contributes, but doesn't account for the checkpoint difference.

What this changes, and what to doubt

If this holds, the practical upshot is a reporting norm: capability claims should come with two numbers, deployed accuracy D and reachable ceiling O_K, under matched protocols. A method that raises D while flattening O_K is doing something real and useful—reliable elicitation is what deployment needs—but it is not "learning to solve new problems," and calling it that misleads downstream decisions (e.g., whether to keep the base checkpoint around as a sampling substrate for distillation or search, which this paper suggests you very much should).

It also sharpens a live debate rather than opening a new one. Yue et al. (2025) and others have already shown that RLVR gains at low sampling budgets can coexist with pass@k crossover at high budgets—the "RLVR doesn't expand reasoning capacity" argument. This paper's contribution over that line is granularity and controls: question-level fate tracking (every new realization traced to a base-reachable question), the propensity analysis, budget-doubling controls, a recipe ablation isolating RLVR from SFT, and a prompt-format control. The framing of "realization vs. reachability" as the general decomposition—unifying the routing-selection audit, the MLP veto mechanism, and the training analysis—is the genuinely useful conceptual export.

Reasons for skepticism, in rough order of importance:

• The dramatic contraction is essentially one data point. DAPO-32B on DART is the −13.3 case; SimpleRL is roughly flat; the OLMo RL-zero control supports the pattern but at small scale (n=150). Whether ceiling contraction is common across RLVR recipes, model families, and tasks is exactly what the paper's own "settings we audit" hedging acknowledges. The paper's selection commentary flags this too: watch for replication.
• O_K is protocol-bound. Reachability at temperature 0.8 with K \le 400 and a fixed prompt is a narrow window. A trained model whose distribution shifted might have its "lost" answers recoverable at different temperatures or prompts. The K=400 control helps, but a temperature sweep on the trained checkpoint would help more.
• Fairness of probes across checkpoints. Trained models produce longer reasoning traces; matched decode lengths and formats could differentially truncate them. The prompt-format control addresses part of this, but candidate-generation fairness between a base and an RL-trained model is subtle.
• Model coverage is Qwen-heavy for the RLVR analysis (both main pairs are Qwen2.5 derivatives), and math-only for tasks. The OLMo lineage control is welcome but singular.
• The three analyses (routing, MLP, RLVR) share a lens rather than a mechanism; the paper is upfront about this, but it means the MLP section, while elegant, doesn't actually explain the RLVR results.

If you read one section, read "RLVR and Reachability" (Tables 5–8 and Figures 6–7)—it's where the audit machinery pays off and where the controls are densest. The routing section is worth skimming for the methodological warning about oracle selection masquerading as search, and Table 2 is a two-minute read that will change how you look at majority-voting numbers.

The one-line takeaway: benchmark deltas conflate "can now" with "now does." This paper builds the tooling to un-conflate them, and the first things it finds under the hood are not what the leaderboards imply.