Evidence Integration in Large Language Models

TL;DR: When you hand an LLM an answer — from a tool, a retrieved document, another agent, or a user — it doesn't weigh that evidence by how trustworthy the source is. It reweights its own pre-existing answer distribution, with a measurable exponent, and adds a bump at the proposed value. This one equation predicts three uncomfortable things that the authors then confirm over ten million trials on twelve models: models preferentially swallow the errors they were already about to make, the same evidence that rescues a weak model poisons a strong one, and — the headline result — models adopt candidates they have just correctly rejected at rates of 93–100%. Verification and control turn out to be almost fully dissociated circuits, right down to a residual-stream decomposition where the verbalizable "workspace" carries the model's stated verdict but none of the causal state that determines the answer.

The problem: evidence arrives after the model has an opinion

Every RAG pipeline, tool-calling agent, and multi-agent system implicitly assumes something like scalar trust: a calculator is reliable, so believe it; a random web page is not, so discount it. The paper's opening example shows why that framing is broken. Ask a model 4\times 28+15 and give it a "calculator report" of 137 (truth: 127). The report preserves the correct last digit, so a cheap check can't reject it — and if the model's own error distribution already favored 137, the external report doesn't correct the model, it ratifies the mistake the model was about to make. The same source, the same error rate, but wildly different danger depending on where the errors land relative to the receiver's internal landscape.

The authors' move is to stop asking "does external context override parametric memory?" (the standard knowledge-conflict question) and instead measure the integration operator itself. Crucially, they measure three things on the same items in separate trials: generation (accuracy with no candidate shown), checking (can the model correctly judge a supplied candidate in isolation?), and use (does the model adopt the candidate when it arrives as evidence?). Keeping these apart is what makes the headline dissociation visible at all.

The law: a prior weight and a tilt

Before evidence, the model has an answer distribution q(y\mid x) over candidates — measured directly as the sequence log-probability of "Answer: v" under the no-evidence prompt, before anything is shown. After a message e proposing candidate v, the final distribution is

p(y \mid x, e) \;\propto\; q(y\mid x)^{\,1+a}\,\exp\phi_e(y),

where 1+a is the prior weight (how much of its own pre-existing margin the model carries through) and \phi_e is an evidence tilt supported on the proposed candidate. A Bayesian receiver has a=0: the candidate's prior support already lives in the prior term, and counting it again is exactly what a\neq 0 means. Scalar source trust falls out as the special case where the source's errors are uniform over alternatives.

The integration law as a distribution shift. The prior enters with exponent 1+a — measured at 0.20–0.65 across nine models, well below the Bayesian value of 1 — so models systematically discount their own margins in favor of the proposed value. But because the exponent is positive, the model's characteristic wrong answer (the "attractor") retains its advantage, making receiver-matched errors the most persuasive ones.

Because the final answer-slot log-probabilities are directly readable, 1+a is estimated as a within-item slope of final margin on prior support — this isn't a loose story-fit, it's a regression coefficient with standard errors below 0.02, linear to within 0.02 per squared nat. Every one of the nine arithmetic models lands between 0.20 and 0.65. And the weight is source-dependent: it drops by 0.11–0.60 when the same value arrives as a tool observation rather than a user sentence — Gemma-4-E4B discards its prior entirely (1+a=0) under a tool message.

Three predictions follow, and all three land:

Own errors are more persuasive. Each model's "attractor" (its modal wrong answer for an item, found by sampling on a held-out calibration fold) is accepted far more than magnitude-matched controls — paired differences of +0.06 to +0.59, with Gemma-4-31B accepting matched controls at 0.064 and its own characteristic error at 0.654, a 10.2× lift. On 77–96% of items, the attractor has more internal support than the truth. This is the correlated-error amplification result: among sources with the same error rate, the one whose mistakes mirror the receiver's is the most damaging, formalized as first-order stochastic dominance pushed through the adoption curve.

Help flips to harm as competence rises. The paper defines a reliability frontier \bar r(c): the minimum source reliability worth consulting at receiver competence c. At low competence the frontier is negative in eight of nine models — a source that is always wrong (with near-miss errors) still helps, because a nearby wrong number scaffolds the computation. At high competence the frontier exceeds 1 in five models — no source reliability compensates. No source-only trust policy can be optimal across that range.

Evidence composes sub-additively. Since the second message acts on the landscape the first already reshaped, agreement between two sources should add less than the sum — by exactly the fraction |a| of the first message's shift. Measured slopes of −0.39 to −0.71 against predictions of −0.47 to −0.91 from independent single-message fits. This is a genuine out-of-sample test of the law's functional form, and it's the kind of thing that separates a fitted regression from a theory.

The dissociation: rejecting and adopting the same answer

Now the part that should reshape how you think about "verifier" architectures. On random 3-SAT — where checking an assignment clause-by-clause is trivially cheap and finding one is hard — the models are given constraint-violating assignments in two ways. Judged in isolation with working allowed, eleven of twelve models correctly reject 90–100% of them. Shown the same assignments as evidence in a use trial, all twelve adopt them at 0.93–1.00. Conditioning on the model's own prior rejection changes nothing:

This is not "the model can't check." It checked, correctly, in a separate trial on the identical candidate. And it's not forced compliance: most formulas have multiple satisfying assignments the model could return instead (Ministral-14B does, 779 times; the 70B Llamas produce exactly zero own-solutions). The paper's framing: checking competence C and the recruitment gain \rho — the causal weight verification evidence gets in the decision — are separate parameters, and \rho is near zero almost everywhere. The pattern replicates on held-out physics, molecular biology, quantum, and genetics tasks (conditional adoption up to 0.994), with one telling exception: where checking is cheapest (reverse complements), rejection finally bites, and the single model that recruits extended deliberation (Gemma-4-31B) sees adoption fall from 0.45 to 0.21 as its working length grows. Recruitment follows verification economics.

Inside the network: the verdict is written where the decision isn't made

The mechanistic sections trace an external candidate through four stages: semantic admission (the same token attributed to an unrelated problem loses almost all its power — acceptance 0.857 → 0.331), promotion (nine times larger than truth suppression; the shown value's logit-lens rank improves ~1000×), transport, and late integration — causal control hands off from the source position to the answer slot at 79% of depth in Llama-3.1-8B and 81% in Qwen3-8B, with fractional depth (not layer index) being the transferable coordinate across seven models.

The dissociation then reappears as geometry. A certificate-validity direction is linearly decodable in both Llama (0.840 held-out accuracy) and Gemma (0.752). Steering along it moves Gemma's decision by 4–7.5× its random-direction control and shifts its generated acceptance from 0.37 to 0.53; in Llama, an eighteen-layer sweep finds nothing anywhere in the stack. Represented ≠ recruited, now as an interventional statement.

Table 6, distilled. The J-lens workspace — the subspace whose contents the model can report — holds the verdict narration but exactly none of the causal state controlling adoption. Its complement carries the full effect, per item. And the post-hoc verdict tracks whatever answer the intervention produced: a mechanistic instance of confabulated reasoning.

The J-lens workspace swap (Section 7.4) is the sharpest single result in the paper. Between a run that adopts a shown value and one that doesn't, swapping only the verbalizable workspace component of the state changes nothing, item for item; swapping only its complement reproduces the full flip, exactly. The separation holds across workspace budgets of 5–200 atoms and against dimension-matched random controls, and replicates in Qwen3-8B. The model's stated judgment of the shown value, elicited before the answer, is identical in every arm — including the arms where all adoption flips — and then changes after the fact to match the causally altered answer.

What changes if this holds, and what to doubt

If the receiver-relative picture is right, several standard practices are quietly miscalibrated. Verifier accuracy, self-critique quality, and elicited knowledge measure C, not \rho — they can be near-perfect while contributing nothing to decisions. Fault-injection evaluations that swap in arbitrary wrong tool outputs miss the dangerous regime entirely, because arbitrary errors are exactly the ones models resist; the killer case is a tool or sibling agent whose errors correlate with the receiver's (shared model family, shared retrieval corpus). Vote-counting aggregation (self-consistency, agent ensembles) overweights agreement, since the law predicts — and the data show — sub-additive, order-dependent, state-dependent composition. And provenance premia (tool channel, "my calculator showed…") are pure functions of writable transcript text, so whoever writes the transcript inherits the authority.

Reasons for caution. The tasks are synthetic with exact oracles, and mostly under terse answer-format prompts that discourage recomputation — the Gemma-4-31B deliberation result shows adoption is far lower when the model actually thinks in the use trial, so the ceiling-level SAT numbers partly reflect an elicitation regime, not an immutable property. The core dissociation is behavior-vs-behavior (separate checking trials), which insulates it from probe artifacts, but the mechanistic claim "decodable yet non-causal" rests on fitted linear directions: a null steering result in Llama could mean the causal coordinate is non-linear or elsewhere, though the stack-wide layer sweep and the exact workspace-swap accounting make that reading harder to sustain. The prior weight a<0 is called irrational relative to the model's own q, but if q is overconfident, some discounting is defensible — the frontier-outside-[0,1] argument certifies a departure without fully apportioning it. Full mechanistic depth covers only two models; and all twelve receivers are instruction-tuned, so these "laws" are plausibly RLHF-shaped policies (the paper itself shows same-scale checkpoints implementing different policies), not architecture-level constants.

If you read one section, make it 6.2–6.3 plus Table 5 for the dissociation in its cleanest form, then 7.4 and Table 6 for the workspace swap. Section 3 rewards a slower read if you want the estimator details — the amount of identification machinery (cross-fitting, matched quadruples, errors-in-variables bounds) behind the headline numbers is unusually serious for this genre.