Base Models Can Reason By Taking a Cue From Training Data

TL;DR: If you prefill just two tokens — e.g. .\n\nOkay — at the start of a base language model's response, its math accuracy can jump from 42% to 78% on MATH-500, essentially matching its RL-trained sibling. The authors then show that RL-Zero training mostly works by making the model likely to emit those same opening tokens, and — the real payoff — they causally trace the cue's power to training data by retraining Olmo on edited corpora: replace every "okay" with "chicken" in mid-training and .\n\nChicken becomes a reasoning trigger; rewire the data differently and "okay" loses its effect. It's a Pavlovian conditioning account of "reasoning elicitation," with the bell swapped out mid-experiment.

The question behind the question

There has been a running debate since DeepSeek-R1: when you do RL with verifiable rewards on a base model and it starts producing long, self-checking chains of thought ("Wait, let me double-check…"), did RL teach that behavior, or merely surface it? Prior work (Yue et al., Zhao et al.) argued from pass@k statistics that RL mostly reweights trajectories the base model could already produce. But that's a distributional argument. This paper makes it mechanical and concrete, and then goes one level deeper: it asks where in the training data the switch that RL flips was installed, and whether you can install or remove it yourself.

The paper's starting observation is almost comically simple. Looking at Olmo-3-7B's uncued responses to MATH-500, 50% of correct answers begin with the token .\n\n (period, blank line), versus 14% of incorrect ones. The model's own first tokens predict whether it's about to reason.

The mechanism: a fork in the first two tokens

Think of a base model's response as a tree. The prompt puts the model at the root, and the first one or two sampled tokens select a branch. Because the model was trained on a mixture of document types — short Q&A pairs, expository math, synthetic reasoning traces — different openings are statistically associated with different document types, so each branch locks the model into a different mode of continuation. The authors formalize this as a factorization: the probability of opening c then continuation y given prompt x is p_\theta(c,y\mid x)=p_\theta(c\mid x)\,p_\theta(y\mid x,c). Prefilling studies the second factor while holding the first fixed.

To find effective cues without reference answers, they run beam search for high-probability two-token openings (width 20, depth 2, averaged over 30 MATH training problems), then select the opening whose 16 sampled answers per problem have the lowest entropy — self-consistency as a label-free proxy for accuracy. The selected cue lands within 1.1 points of the best among 500 tested openings. Note what the cue is not: it is not an instruction, not a demonstration, not "think step by step." It's punctuation plus a filler word.

The cues transfer: the same fixed opening lifts Olmo on GSM8K (45→86%), AMC 23 (31→60%), AIME 2024 (12→23%), and even HumanEval code (50→70%). Controls rule out the boring explanation that cues just buy more test-time tokens: some openings produce responses twice as long as the reasoning cue yet score below no-cue, and under truncation the cue beats no-cue from 1k tokens onward, reaching RL-level accuracy at roughly half RL's token cap. Majority-voting with no cue needs 3.6–12× the tokens to match a single cued response.

What RL actually changes

If the cue alone recovers RL-level accuracy, what does RL do? The paper's answer: mostly, RL turns up the prior on the cue.

Three pieces of evidence, each sharp. First, per-position KL divergence between base and RL policies, conditioned on identical token histories, peaks at the first two output positions and collapses afterward — given the same prefix, the two models mostly agree about what comes next. Second, RL raises the probability of .\n\nOkay from 0.14 to 0.65 in Olmo and of Alright, from 0.04 to 0.58 in Qwen. Third, the swap experiment: let the base model continue an RL-generated opening, and it scores 77.7% — above RL's own 75.1%. Let the RL model continue a base-generated opening, and it drops to 43.7%, barely above the uncued base's 41.7%. The RL model's advantage travels with the opening, not the weights that produce the continuation.

They push this to an almost absurd endpoint: a closed-form, single-layer weight edit. Compute the gradient G of the cue's log-likelihood, estimate the input second-moment matrix C on ordinary text, and apply \Delta W \propto G C^{-1} to the final MLP projection (a ROME-flavored whitened update) so the model emits the cue on its own with probability 0.95. This recovers most of the RL gain with a +0.04% change in C4 perplexity — no RL, no gradient descent on rewards, one matrix edit.

To be fair to RL: standard GRPO does eventually catch up to and slightly exceed the cued base on some models, and prefilling the cue during RL rollouts is worth roughly the first 50–100 GRPO steps. The claim is not that RL does literally nothing beyond the opening, but that in the R1-Zero setting, the dominant effect — most of the accuracy delta — is cue elicitation.

The Pavlov experiment: editing the bell

Correlation was already suggestive: in Olmo's mid-training data, "Okay" opens 83% of the synthetic reasoning traces. But the paper's most distinctive contribution is the counterfactual retraining. Olmo-3 releases its mid-training mixture, checkpoints, and recipe, so the authors restart mid-training from the final pretraining checkpoint on three versions of the same 10B-token mix, identical in order and seed:

• Base mix: unedited.
• Rename mix: every whole-word "okay"→"chicken" throughout the corpus.
• Redirect mix: rename, plus every paragraph-initial "Question:" label in short-QA documents becomes "Okay," — pairing "Okay" with question-asking instead of reasoning.

Read the chart column by column and the logic clicks. On the base mix, Chicken is useless (2.4%) — the model invents a problem about chicken nuggets. On the rename mix, Chicken becomes a full-strength reasoning cue (37.2%, vs. 38.3% for Okay), and after the cue the model now predicts ", let" and ", so" — the grammar of a reasoning trace. Crucially, "chicken" retains its ordinary meaning everywhere else: "She roasted the chicken and made a salad." The association is context-specific, exactly like Pavlov's bell only triggering salivation in the feeding context. On the redirect mix, .\n\nOkay collapses to 0.2% — the model responds to the cue by generating a new question ("Okay, what is the first step to solve…?") — while Chicken keeps working. A full 100B-token rename run replicates the effect at the released model's accuracy level (Chicken-cued: 76.9%), and parallel edits on SmolLM3-3B and with "Alright"→"Duck" reproduce the pattern.

One subtle result in the rename condition deserves attention: after scrubbing "okay" from mid-training, prefilling .\n\nOkay still elicits reasoning (38.3%) — the data edit only made the token unlikely to be generated (0.15 \to 5\times10^{-7}), not ineffective when forced. The authors note the cue already helps before mid-training, so the association partly predates the edited stage. Killing the effect required actively redirecting the token toward a competing behavior. Associations, once formed, are robust to mere absence.

The same logic applies to prompt instructions. Replace "step by step" with "duck duck goose" in training, and at evaluation "Think duck duck goose" (6.4%→16.5%) matches "Think step by step" (12.2%→14.9%). This is a direct shot at semantic readings of prompt engineering: the instruction's power came from its training-data pairing, not from the model "understanding" a described procedure.

Where the modes live: hidden states

The representational analysis closes the loop. Comparing layer-24 hidden states of generated responses against a bank of ~48k states from training documents (nearest neighbors by cosine similarity), different cues shift the response's neighbors toward different training sources: Okay toward synthetic reasoning traces (checking phrases in 97% of responses), To toward expository math (5% checking), Answer toward short Q&A, Problem toward "meta-reasoning" documents — which, notably, scores only 6.3%: the model restates and plans but never computes, a nice reminder that reasoning-flavored text is not reasoning. Cue position matters too: inserting Okay at paragraph 1–2 produces a sustained shift toward reasoning-trace neighbors and nearly doubles accuracy; at paragraphs 3–6 the shift is transient and accuracy drops below baseline. Context gates whether the bell rings.

The safety case study shows the same machinery steering refusal. On XSTest, I'm sorry makes Olmo-3-7B refuse everything including benign requests (89% safe-prompt refusal); Okay, is a compliance cue (42.4% harmful-response rate on unsafe prompts); but .\n\nOkay — differing only in whitespace — yields selective refusal (81.9% unsafe refusal, 4.2% safe refusal, 0.2% harmful), approaching the post-trained Instruct model's behavior, because the paragraph-break form routes into reasoning-trace mode where the model deliberates about the request first. Whitespace as a safety mechanism is the kind of result that should make anyone training on synthetic data pause.

What to make of it, and where to be careful

If this holds broadly, several things follow. The "aha moment" narrative around RL-Zero deflates further: much of the gain is a prior shift onto a two-token switch that mid-training data installed, which you can replicate with a prefill or a single whitened weight edit. It sharpens the distinction between what a model can do and what it is likely to do — and suggests post-training budgets are partly paying for something a conditioning trick gets free. It also reframes synthetic data design: since 83% of Olmo's reasoning traces begin with "Okay," the cue is an artifact of a stylistic monoculture in trace generation, and one could deliberately engineer cue–behavior pairings (or accidentally inherit harmful ones) as synthetic data floods pretraining mixes.

Now the caveats, which the authors state more plainly than most:

• Model dependence is strong. No effective cue was found for Llama-3.1-8B (all tested openings leave it near zero), and Qwen2.5-Math-7B gains almost nothing — it already reasons uncued. The mechanism requires training data that repeatedly pairs a stereotyped opening with reasoning, i.e. synthetic traces in the mix. This is a feature of the account, but it means "base models can reason via cues" is really "base models mid-trained on reasoning traces can."
• R1-Zero setting only. RL applied directly to a base model, mostly on MATH with binary rewards and LoRA. Multi-stage post-training pipelines, harder domains, or longer RL runs may add genuine capability beyond cue amplification; the paper explicitly does not claim otherwise. On a few cells (e.g., Qwen's RL-Zero-prompt GSM8K), cueing the RL model even hurts.
• The cue search touches the task distribution. Cues are selected on 30 MATH training problems (label-free, but in-domain), and transfer to code is uneven — Qwen's math cue slightly hurts HumanEval under the boxed prompt.
• For Olmo, the punctuation does most of the work: .\n\n alone scores 76.6% vs. 76.9% with "Okay" appended. The headline cue is really "start a new paragraph in the way reasoning traces do."
• The data-intervention runs are mostly 10B mid-training tokens at absolute accuracies far below the released model (38% vs. 78%), though the 100B replication substantially de-risks this.

The most worthwhile read is Section 3.1 (the data edits) together with Appendix F — that's where the causal claim lives and where the replications across models, cues, and token budgets either convince you or don't. Section 2.3's swap experiment (Appendix D.3) is the second must-read: four rows of a table that compress the entire "what does RL add?" debate into one clean 2×2.