How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents

TL;DR: The community's working assumption is that architecture changes shift the constant of the pretraining scaling law (a fixed compute discount at every scale) but never its exponent (a discount that compounds as budgets grow). This paper claims to break that rule twice, with careful controls: (1) a "boundary operator" that renormalizes the residual stream and re-injects an early representation between blocks, and (2) model growth — training shallow, then adding depth mid-run, implemented via looped transformers. Both make the compute-efficiency multiplier over a well-tuned vanilla transformer grow with scale, reaching 1.25× and 1.55× respectively at 10^{20} FLOPs. A held-out 7.4B growth model lands on the extrapolated curve and matches GPT-3 13B on CORE at ~20× less compute (with real caveats). The unifying story is "computational depth": deep pre-norm transformers waste a scale-growing fraction of their depth, and fixing that waste bends the scaling law rather than translating it.

Constants versus exponents

Compute-optimal pretraining loss follows a power law:

where C is training FLOPs, E an irreducible floor, A a constant, and \gamma the exponent. On a log-log plot, changing A slides the curve sideways; changing \gamma tilts it. The practical difference is enormous. A constant improvement — like Muon's ~40% gain over AdamW — saves you the same factor of compute at every scale. An exponent improvement gives a compute multiplier \pi(C) (how much extra compute vanilla needs to match you) that grows without bound as budgets increase. Mixture-of-experts is one of the very few interventions credibly claimed to do this; nearly everything else in a decade of architecture papers has been a constant shift, and several careful studies (Kaplan et al. on depth; recent controlled looping studies) explicitly found no exponent effect from the kinds of interventions studied here. That's the conventional wisdom this paper attacks.

The exponent gaps involved are numerically tiny — on FineWeb, the best variant fits \gamma = 0.1168 against vanilla's 0.1113 — but that 5% relative difference in slope is exactly what compounds: it's a 1.55× multiplier at 10^{20} FLOPs and a projected 2.7× at 10^{25}.

One family, three switches

The clever experimental design is that vanilla transformers, looped transformers, and model growth are all expressed as one architecture with three independent switches, so each effect gets an exactly matched control. Every model is a prelude–core–coda network (following Geiping et al.'s recursive-depth formulation): a prelude of transformer blocks computes an embedding e, a core of blocks is applied K times to a state h, and a coda produces the output.

The three switches:

Boundary operator (BO). Between core passes and before the coda, apply \mathrm{BO}(h,e) = \mathrm{Norm}(h) + \alpha e: RMS-normalize the residual stream and add back the prelude output scaled by a tuned \alpha. A vanilla transformer is K=1 with identity boundaries. "Operator-1" is vanilla plus BO — identical parameters and FLOPs. This targets the curse of depth: in pre-norm transformers the residual stream's norm grows with depth, so each block's additive update becomes a vanishing fraction of the stream and deep blocks drift toward doing nothing. Normalizing lets every block write at full weight; re-injecting e keeps every pass conditioned on the input. (One detail they show matters: apply the full normalize-and-inject before the coda too, which prior looped transformers skipped — ablating the coda injection costs a chunk of the gain.)

Tied vs. untied. Tied core passes give a looped transformer (stores one core, executes it K times); untied passes make it a plain deep transformer with BO. Same computation graph, same FLOPs — only stored parameters differ.

Growth. Start training with K=2, and partway through jump to K=4. Tied: just loop more (zero new parameters). Untied: copy the trained cores and let the copies diverge — this is depthwise stacking, but used here not to recycle checkpoints but to shape when depth exists during a single compute-optimal run.

The result

Each architecture gets its own compute-optimal scaling ladder from 10^{18} to 10^{20} FLOPs on FineWeb (Muon for matrices, per-architecture recipes — more on that below), and the fitted laws are read out as compute multipliers over vanilla. The signature of an exponent change is a multiplier that rises with scale:

The matched comparisons decompose this cleanly:

• BO alone changes the exponent. Operator-1 vs. vanilla is parameter- and FLOP-identical, yet its multiplier climbs from 1.12× to 1.25×. Same story from Deep Vanilla to Untied-2.
• Growth changes the exponent, independently of BO. Untied-Grow's advantage over Untied-2 widens from 1.09× to 1.16×; Deep Vanilla Grow shows the same shift over Deep Vanilla without the operator, so the two effects add.
• Untying only changes the constant. Untied variants sit a fixed ~1.06–1.16× above their tied twins at every scale. So weight sharing costs a flat compute tax — meaning Loop-Grow keeps the exponent improvement of growth at vanilla's parameter count (1.36× at 10^{20}).
• Depth alone is a constant. Deep Vanilla's flat 1.08× is just a better width–depth ratio.

Why nobody saw this before

This is arguably the most important methodological point. Prior controlled studies of looping in pretraining found neutral or negative results at compute-optimal settings. The authors' explanation: the exponent effect is invisible unless every architecture gets its own fully tuned, correctly scaled recipe. They fit a four-stage recipe per architecture — chain-tuned base hyperparameters at small scale, tokens-per-parameter (5 for vanilla, 6 with BO, 7–8 with growth: grown models want to spend more budget shallow), growth timing, and a learning-rate-vs-size scaling rule (whose exponent differs by architecture: -0.8 for vanilla vs. -0.5 to -0.7 for variants). Transferring vanilla's recipe to Operator-1 costs 7.8\times10^{-3} loss at the base size and turns the exponent improvement into a constant one (Figure 16b). This rhymes with the Muon story, where hyperparameter scaling determined whether the gain survived scale — and it's a warning shot for every architecture-comparison paper that reuses baseline hyperparameters.

The 7.4B extrapolation

The ladders span only two decades of compute, so the extrapolation test matters. They rerun vanilla and Untied-Grow ladders on FineWeb-Edu, fit through 10^{20} FLOPs, then train a single held-out Untied-Grow d26 (7.4B parameters, starting at 5B before growth) at 1.23\times10^{21} FLOPs — 8× beyond the fit range. It lands 0.03 below the predicted loss, within 0.002 on answer-NLL, and hits CORE 0.3865 vs. a forecast of 0.384. That CORE score matches the GPT-3 13B reference of 0.3852, which cost 2.31\times10^{22} FLOPs — the headline ~20×.

Read that 20× correctly: GPT-3 was trained in 2020 on far worse data with a far worse recipe, and the reference score comes from a separate evaluation pipeline. The paper says so plainly ("indicative rather than a controlled comparison"). The controlled number is the 1.8× multiplier over their own well-tuned vanilla at that budget — data quality, Muon, and modern recipe hygiene account for the other ~10×. The interesting part is not the 20× but that (a) the exponent gap survived an 8× extrapolation and a corpus change, and (b) the fitted laws project 2.7× at 10^{25} FLOPs, which — if the power law really holds five more decades — is a big deal.

When data runs out, loop more

The single-epoch story says weight sharing is never compute-optimal on fresh data (optimal K sits between 1 and 2). The multi-epoch story inverts this. Fix 100M unique FineWeb tokens, train 10 epochs: now the compute-optimal loop count rises with budget, from ~1.4 to ~6.7, and scaling loops at fixed model size and fixed weight decay matches the best loss of size-scaling with weight decay retuned at every size — using 2.2× less compute and essentially no tuning.

The proposed mechanism is crisp and testable: models overfit with the parameters they store, not the blocks they execute. Two pieces of evidence: the optimal weight decay rises with stored depth but is nearly flat in loop count; and the untied control (same executed depth, more parameters) does not beat the baseline under repetition. Tied looping adds capacity with nothing new to overfit. Given that compute is outgrowing high-quality text and multi-epoch pretraining is coming, this may be the most immediately actionable result in the paper.

The unifying lens: computational depth

Define computational depth as the number of blocks that actually influence the predictive distribution. The claim is that vanilla transformers waste depth in two scale-growing ways, and each intervention removes one:

1. Blocks that execute but don't matter (curse of depth). Fixed by BO.
2. Depth paid for before it's needed. Networks learn simple structure early; a fixed-depth model burns FLOPs on blocks it can't yet use. Fixed by growth.

Crucially, both wastes plausibly consume a larger fraction of depth at larger depths, and compute-optimal models get deeper with budget — which is exactly the condition under which fixing them changes an exponent rather than a constant. They probe this with KL effective depth (logit-lens: decode the residual stream after each block, find where predictions stop changing):

Two corroborating details: width-only scaling (holding depth fixed) collapses Operator-1's exponent gain to a constant, exactly as the hypothesis predicts; and tied/untied curves coincide in effective depth, so weight sharing doesn't change what depth gets used — matching its constant-only effect on loss.

How much to believe

The experimental hygiene here is well above field norms — per-architecture recipes, matched controls on every axis, a held-out extrapolation run, wall-clock as well as FLOP accounting, downstream metrics tracking loss. Still:

• Two decades of fitted compute is thin for an exponent claim. A slowly drifting constant (or a power law with mild curvature) can impersonate an exponent difference over 10^{18}–10^{20}. The 8× extrapolation helps, but it's one run, and the 10^{25} projection is five decades beyond any data.
• The exponent gap is only as real as the vanilla baseline's tuning at scale. They tuned vanilla hard, but any residual mis-scaling of vanilla's hyperparameters at larger sizes would inflate the variants' apparent exponent — and their own Figure 15c shows recipe choices can swing which architecture "wins" the scaling race. Their Appendix B.5 finding that Muon's advantage over Adam shrinks under joint width/depth scaling is a reminder that optimizer–architecture–scale interactions are live and poorly understood.
• The mechanism should saturate. If BO works by reclaiming a wasted fraction of depth, that fraction can't grow forever; the "exponent improvement" may be a transient regime before both curves re-converge in slope. Nothing in two decades of data rules this out.
• The literature is genuinely split: Schwethelm et al. found recurrence strictly worse at compute-optimal settings; concurrent work on a looped sparse MoE found growing gains. This paper's answer — it depends entirely on block allocation, loop count, and per-architecture tuning — is plausible but means the result is fragile to implementation details.

Where to spend your time

Section 4.2 with Figure 3 is the core evidence and takes ten minutes: the three matched comparisons (BO, tying, growth) and the exponent/constant decomposition in the right panel. Section 6 is the conceptual payoff — the two-wastes framing plus the KL-effective-depth measurements. If you'd actually train one of these, Appendix A.4 (the four-stage recipe) and Table 9 are the practical artifact, and Figure 16b is the cautionary tale: reuse your baseline's hyperparameters and you will conclude, wrongly, that architecture can't change the exponent.