Issue 27 · Pick 03 AI / ML ✓ read
Dictionaries, Not Darwin: Set-Level Selection Beats LLM Evolution in Scientific Equation Discovery
TL;DR: A large fraction of "LLM-driven scientific discovery" systems are built on an evolutionary loop — generate candidates, keep winners, feed them back as parents, repeat. This paper runs the control experiment almost nobody runs: does the loop actually beat spending the same LLM calls on independent samples? In equation discovery, it doesn't (median OOD NMSE 0.045 vs 0.049; identical solve counts; instructed crossover is worse). The author then extracts what the loop was implicitly doing well — accumulating candidate terms — and replaces the whole apparatus with a one-shot dictionary plus classical set-level sparse regression. Result: 73.2% numeric accuracy on LLM-SRBench with Llama-3.1-8B versus 49.2% for the best baseline, at one tenth the call budget. The deeper conceptual claim is about identifiability: underdetermined data can only score sets of terms jointly, so any method that assigns credit to individual candidates is throwing away the signal.
The premise being audited
Since FunSearch, a standard architecture has spread through LLM-based discovery: the LLM proposes candidate solutions (formulas, programs, heuristics), a scorer ranks them, and top candidates go back into the prompt as "parents" for the next generation of "mutations." The vocabulary — populations, crossover, generations — carries an implicit empirical claim: that the loop compounds, so generation 10 finds things that 10× more independent samples would not.
That claim has a natural null hypothesis: matched-budget independent sampling. If you have 100 LLM calls, does routing them through a lineage of parent-conditioned prompts beat just firing off 100 fresh, independent proposals? LLM proposals are already strong one-shot priors, so this null is not a strawman. Yet most papers in this line never test against it.
The chosen battleground is scientific equation discovery (LLM-SRBench): given finite samples of an unknown physical law, produce a symbolic formula. This is a good test case for a specific reason — finite data makes the structure underdetermined. Many expressions interpolate the training grid; only some encode the reusable structure that survives out-of-distribution evaluation. If iterative refinement genuinely accumulates scientific structure, this is exactly where it should show up.
The audit: the loop is resampling with extra steps
Everything is matched: same proposer, same prompt format, same budget of 100 LLM calls per problem. Four findings, labeled D1–D4 in the paper:
D1. Parent-conditioned evolution matches fresh sampling: median OOD NMSE 4.47\times10^{-2} vs 4.91\times10^{-2}, strict solves 16 vs 16 on the development split. Explicitly instructing the model to combine the best components of two strong parents makes things worse (2.00\times10^{-1}) — and proposal validity stays at 98.7%, so parsing failures don't explain it. The LLM, even when asked directly, does not recombine.
D2. Whether a run eventually succeeds is predicted by the quality of its round-0 independent proposals with AUC 0.959. Later parent-conditioned calls add no measurable lift. The outcome is decided before the "evolution" begins.
D3. Lineage tracking shows useful terms are only weakly heritable across generations; the dominant dynamic is accumulation of low-value terms. More on this below — it foreshadows the paper's main principle.
D4. Three distinct iteration schemes (naive re-extraction, credit-filtered recombination, guarded population-level recombination) run for four generations add zero solved problems. The guarded scheme goes 50, 47, 51, 51, 51. The naive scheme is actively destructive: 50 → 38 strict solves, with median formula length inflating from 270 to 523 characters.
So the loop is not a ratchet. It's a random sampler wearing a Darwin costume. But the audit also reveals what the loop was quietly producing that has value: across a problem's 100 independent proposals, the correct terms appear far more often than any single proposal is correct. The loop's real output, operationally, is a dictionary of candidate terms — and the open question the loop never answers is how to select from it.
The method: dictionaries, then classical statistics
PTB-Search is deliberately simple, and every component is justified by a "collapse ablation" (remove it, watch performance collapse):
- Variation (LLM, once). 100 independent proposals with a fixed prompt. No parents, no generations.
- Inheritance (extraction). Parse each proposal, canonicalize its additive terms into a per-problem dictionary D=\{\phi_1,\dots,\phi_p\} of basis functions, pooled with generic primitives (powers, exponentials, etc.). This is where heritability actually lives — terms outlive the formulas that carried them.
- Recombination + selection (external, set-level). Fit sparse linear combinations \hat{y}=\beta_0+\sum_{j\in S}\beta_j\phi_j(\bm{x}) with support size |S|\le 6; coefficients by least squares given a support; supports found by beam search scored on internal train splits only. This is best-subset selection — the SINDy-style estimator family sparse regression has used for a decade.
- Stop. No second generation.
The ablations are what make this convincing rather than merely convenient. Extract terms from a single proposal instead of pooling across all 100, and the development-split median OOD collapses from 4.7\times10^{-9} to 2.2\times10^{-2} — right back to the fresh-sampling level. Fit the whole dictionary densely without subset selection: worse everywhere. Transplant terms from other problem families: total failure (0 solves). So the recipe is genuinely cross-proposal recombination plus selection, not any single ingredient.
One honest calibration the paper states rather than hides: a dictionary of generic primitives alone is already a strong baseline (on the 717-cell Llama grid it actually gets more strict solves than the hybrid, 160 vs 135, though the hybrid wins on median OOD NMSE). LLM terms are complementary, problem-compatible priors on top of a strong classical method — not the sole source of the magic. That's worth remembering when reading the headline.
The aha: credit can only be assigned to sets
Why does per-term credit — the natural instinct, and the mechanism of concurrent work (IGSR) — fail? The answer is classical statistics wearing new clothes. A hundred LLM proposals for one problem share structure by construction, so the dictionary columns are highly correlated, and the training grid is small, so the design matrix is underdetermined. This is exactly the regime where sparse-recovery theory (Zhao & Yu's irrepresentability condition, Fan & Lv on marginal screening) says individual-predictor statistics are unreliable while the joint fit of a candidate support remains well-defined.
The paper measures this directly. A carefully built per-term credit (selection frequency × standardized coefficient × drop-one loss increase) does correlate with a term's functional contribution inside a fitted set (Spearman 0.26–0.63), yet it identifies ground-truth terms at chance level: AUC 0.44 on the dev split, 0.42 on the blind split. The credit measures "how much does this term matter inside this particular set" — not "is this term true."
The performance consequence is the paper's biggest experiment: on 717 problem-seed cells with identical dictionaries and zero additional LLM calls, three set-level joint selectors solve 165–169 problems while two very different single-term reductions — marginal-influence credit and stability-selection frequency — solve 74 and 78. Every cross-family comparison lands at p<10^{-73}. Crucially, both reductions fall significantly below the frozen set-level baseline they were meant to refine. The failure axis isn't the choice of statistic; it's per-term reduction itself. And the case study makes it constructive: the winning formula for one solved problem combines terms from three different LLM proposals plus two generic primitives — no single proposal ever contained the full answer. Recombination is the discovery; the LLM just never did it.
The benchmark numbers
On the official 239-problem LLM-SRBench split, evaluated with official code and aggregation, against baselines standardized to 1,000 LLM calls per problem, PTB-Search uses 100 calls:
The most interesting number here is the four-point gap between backbones. An 8B open model and a frontier API model, differing by orders of magnitude in capability, land at 73.2% and 77.0% under the identical frozen method. Under a "proposer strength is everything" account, that spread is surprising; under the dictionary-and-selector account, it's expected — both proposers surface enough correct terms, and the external selector does the rest. On LSR-Synth (the 128 problems with an OOD split), the Llama run hits 95.3% in-distribution and 71.9% OOD accuracy; material science is essentially solved (100% ID, 88% OOD). The weak spot is LSR-Transform (rearranged Feynman equations, 47.8% ID on Llama), and a post-hoc autopsy attributes remaining failures mostly to availability — required terms the proposer never wrote down — not to selector defects. That's a proposer-capability boundary, honestly labeled.
The scoped boundary: programs
The obvious objection: equations are unusually friendly to one-shot recombination because additive terms merge by least squares. So the paper runs a heterogeneous stress test on FunSearch-style bin-packing (LLM-proposed Python priority functions), explicitly not claiming PTB-Search applies there. On easy instances, evolution again ties fresh sampling (p=0.18–0.75). On harder heavy-tailed instances, something does separate — but not what the evolutionary story predicts: recombination over a retained external population beats fresh sampling (7/0 non-tie wins, p=0.016), and an instructed crossover probe wins 8/0 (p=0.008) at a quarter of the budget, while the parent-conditioned multi-generation loop stays indistinguishable from fresh (p=0.73). Even where iteration seems to help, the active ingredient is retained external state, not generation count. This is a small grid (3 distributions × 3 seeds, with two seeds excluded for API errors), so treat it as a scoped boundary readout, which is how the paper treats it.
What to make of it
If this holds, it reframes a rapidly growing methodology. Systems like FunSearch, LLM-SR, and ShinkaEvolve should be described by their actual division of labor — proposal material, archive state, verifier feedback, external selection — rather than as evolution, because the evolutionary framing encourages the wrong ablations and wastes 10× the budget on lineage-conditioned rewrites that behave like resampling. The practical recipe is portable to any domain where candidates decompose into reusable components scorable jointly against data: dynamical-system and PDE term discovery, physics libraries, formula-like heuristics.
Reasons for skepticism, stated plainly. The strong claims live in one benchmark family (LLM-SRBench), which is componentized-friendly by construction; the paper's own program-domain result shows retained-state recombination can pay in harder nonlinear spaces, so "evolution never compounds" is not established for domains with reliable per-step credit or genuinely editable structures — think AlphaEvolve-scale program search, where the paper's audit doesn't reach. Symbolic accuracy is reported only as a caveated diagnostic (a sparse fitted readout is nearly guaranteed to be judged symbolically inequivalent), so the win is a numeric/OOD win, not a symbolic-SOTA claim. The generic-primitives-only baseline being so strong means part of the gap to prior work reflects prior methods under-using classical sparse regression, not LLM magic. And the DeepSeek headline is a single seed. On the other side of the ledger: pre-registered decision branches, frozen blind splits, matched budgets, leakage audits, and paired cell-level statistics are unusually disciplined for this literature, and it's a single-author paper that repeatedly reports numbers unflattering to its own method.
Where to spend your time: Section 3 (the audit) and Section 6 (the set-level identifiability analysis, especially Figure 5's single-problem dissection) carry the intellectual content; Section 8 is worth reading for how carefully the author scopes the negative claim. The method in Section 4 you could re-implement from this post.