Models Take Notes at Prefill: KV Cache Can Be Editable and Composable

TL;DR: If you change one field inside a long cached prompt and surgically overwrite just that field's key/value vectors, the model keeps acting on the old value — and this paper shows why, causally: during prefill, the transformer already computes the field-conditioned conclusion and writes it onto downstream delimiter tokens. The field's own KV drives less than 1% of the final decision. Once you see the KV cache as a notebook of memoized conclusions rather than a frozen encoding of tokens, two operations fall out: you can edit the notes (a one-line appended erratum beats recomputing the suffix) and compose them (RoPE-rotate a precompiled skill's cache into any position, matching full recompute at O(L) instead of O(L^2) time-to-first-token). The serving payoff is real: in an online vLLM benchmark, the append-only erratum keeps the prefix cache hot (98.5% vs 1% hit-rate) and cuts p90 TTFT by 53–398×.

The puzzle

Every agent framework pays a recurring tax: long, mostly-static context (system policy, tool specs, retrieved docs) gets re-prefilled whenever anything inside it changes. Prefix caching only helps up to the first changed token — because every token's keys and values were computed while attending to everything before it, one edited field invalidates the entire downstream cache. The standard workaround is to hoist all mutable state to the end of the prompt, which pushes an inference-layer constraint into your application layer and breaks down for nested prompts, multiply-referenced fields, and dynamically assembled contexts.

Here's the tempting shortcut that motivates the paper. Suppose order_status changes from pending to shipped at position 500 of a 1000-token prompt. Tokens 0–499 never saw the field; the paper measures their KV deviation as exactly 0.0 when the field changes. So why not recompute only the field's own KV (two tokens, ~0.2% of the prompt) and reuse everything else stale?

This fails completely. The model's decision reverts to the old value, as if the edit never happened. That failure is the interesting datum: the model apparently isn't reading the field at decision time at all.

The mechanism: models take notes at prefill

The explanation the paper establishes — and this is the conceptual core — is that transformers do not defer inference to decode time. During prefill, mid-layer attention computes the field-conditioned conclusion ("status is shipped, so the action is deny") and writes it into the KV vectors of a handful of downstream aggregator tokens: punctuation, newlines, section breaks — the positions later tokens route attention through. At decode, the decision token doesn't re-derive anything from the field; a small set of late-layer "read heads" attend back to those note positions and copy the stored conclusion into the logits.

The evidence is causal patching, not just probing, and it's unusually thorough for a mechanism claim. Four converging probes on a gated agent decision (cancel-vs-deny depending on order_status), replicated across Qwen3, Llama-3.1, Gemma-2, and Mistral:

Two controls are worth highlighting because they close the obvious loopholes. First, the dissociation test: hold the field value byte-identical and flip a single rule token so the conclusion inverts. Transplanting the downstream notes carries the entire flipped conclusion (recovery 0.998–1.009), while patching the changed rule token itself carries nothing — so the notes encode the conclusion, not a copy of the field. Second, false-note injection: write the opposite conclusion's KV into a handful of note positions in an otherwise-consistent cache, and the model follows the injected note against its own live field (recovery ≈1.0). The notes aren't a summary the model consults; they're the thing that governs the decision.

The appendix pushes this to a component-level circuit — distributed write (mid-layer attention onto multiple aggregators), concentrated read (12 nameable late heads recover 0.78 of the decision; 12 random heads recover ~0), causal scrubbing confirming the note alone governs — replicated across all four families. It also finds the by-now-familiar interpretability lesson at the feature level: an SAE feature decodes the conclusion at AUC 1.0 yet is causally useless alone; the causal content is spread over ~10–30 features along a low-rank direction.

Consequence 1: the cache is editable

If the stale conclusion lives in the notes, fixing a changed field means amending the notes. Three options, with a clean cost/reliability ordering:

Append an erratum (the robust default). One salient line before the decision — [STATE UPDATE] account_role → suspended_user; overrides any earlier value and conclusion — creates a fresh, authoritative note the decision token attends to. It's O(1), append-only (so it composes with production prefix caching), and matches the hoist-to-end oracle without any prompt surgery.

Recompute the affected suffix, in full (reliable, expensive) or only the top-K highest-causal-effect note positions (field+selective@K, cheap but unreliable — the minimal K is wildly model-dependent: K^\star \approx 4 at 8B but >64 at 4B, non-monotonic in scale).

Refresh the field alone (~1% compute) — which works only under chain-of-thought. This is a genuinely interesting finding: with a reasoning chain, the model re-reads the field during decode and the near-free edit recovers the decision perfectly (1.00 on Qwen3-8B); without CoT, the identical edit on the identical model is fully ignored (0.00). The gate is the reasoning mode, not model size. There's a catch in the worked example, though: under CoT the in-place edit produces 926 thinking tokens vs 249 for a clean prefill — the recomputation didn't vanish, it moved from prefill into the chain.

The CacheBlend failure is diagnostic: it selects tokens to recompute by KV deviation, i.e., which keys changed — but the tokens that matter are the ones that memoized the conclusion, whose keys may barely change. The mechanism, not just the recipe, is what the paper adds.

The comparison to weight editing (ROME, LoRA) is a nice framing exercise. Both can flip the target decision, but a weight edit is global: the same model instance can't hold status=shipped for one request and pending for another, so concurrent requests get contaminated (rate 1.0), unrelated decisions drift (0.5), and each edit takes 3–6 s vs 114 ms for the erratum. Weights are for durable global facts; the KV cache is exactly the right substrate for mutable per-request state.

Consequence 2: the cache is composable

The same mechanism predicts position-portability. If a skill's notes are localized and re-derivable from context the decision can still see, you can prefill a skill once, in isolation, then splice its cache into any target context: values are position-free, and since libraries cache post-RoPE keys, you re-rotate the keys from source to target positions. This is CacheSlide-style position-aware reuse — the paper is explicit that the caching machinery is prior work (Prompt Cache, CacheBlend, EPIC, CacheSlide, MPIC) — but the mechanism explains why it works and why the one residual error is a seam: the chunk's first few tokens, which in a real prefill would have attended to the now-missing prefix. Recompute a few boundary tokens and the splice is behaviorally indistinguishable from full recompute — logit cosine 0.90–0.999 across twelve models (0.6B to 70B, FP8, a 30B MoE), 24/24 correct skill-following across 8 domains, function-call accuracy 1.00 on four models measured with actual tool calls.

The keystone experiment ties the two capabilities together: edit a field that lives inside a transplanted skill. Every editing method behaves identically on the composed cache and a fully recomputed one — in-place weak, selective@32 ≈ 0.80, erratum strongest, points on the diagonal. One notebook, two operations. A unified edit+compose agent (compose the policy once, erratum the state changes, prefix-cache the rest) stays decision-identical to recompute-every-turn across thirteen models at up to 14.9× lower cumulative TTFT, and a 28-turn stress test with the gating field toggling every turn shows no compounding drift (logit cosine 0.99+, flat).

Applications and the systems win

The user-memory section (Section 7) is the most complete application study and the one I'd read carefully. Memory placement is a real dilemma the mechanism explains: front-placed memory gets pre-digested into downstream notes (so editing it is expensive), end-placed memory must be re-attended every turn. Their answer — precompile memory as a skill, place it late, re-rotate each turn, edit in place — is faithful up to 70B, and they honestly quantify the cost of forgoing pre-digestion: late placement loses ~0 accuracy at 2k memory tokens but +0.09 at 16k and +0.16 at 32k versus early placement. External validity comes from LoCoMo (real multi-session dialogues, ~19.7k tokens): transplant is statistically equivalent to full recompute in QA accuracy on three Qwen3 models (TOST, margin 0.03) and within 2.7 points on Llama-3.1-8B. One sharp caveat they surface themselves: splitting cross-referential facts across independently-precompiled blocks breaks two-hop chains badly (agreement 0.46 vs 0.76 colocated) — because block B, precompiled in isolation, never attended to its referent A.

The vLLM benchmark is where the numbers get dramatic, and the reason is simple once you see it: the erratum is append-only, so it composes with automatic prefix caching, whereas writing the new value into the prefix changes a cached block's content hash and invalidates everything downstream. On a real online server (V1 engine, continuous batching, Poisson arrivals, ~8k-token policy with one mutable field): 98.5% vs 1.0% APC hit-rate, p90 TTFT of 86 ms–1 s vs 22–55 s (53–398×), throughput advantage growing from 1.6× at 2 req/s to 14.5× at saturation. Multimodal caches work too: spliced image KV (with M-RoPE temporal-axis re-rotation) is near-lossless across four VLMs at 2.4–8.4× faster TTFT.

What to make of it

What's genuinely new. Not the caching machinery — the paper is refreshingly honest that repositioning is CacheSlide, seam-repair is EPIC/CacheBlend/MPIC. The contributions are (i) the causal mechanism explaining why naive edits fail and why boundary recompute is exactly what's needed, (ii) the editing axis, which prior KV-reuse systems lack entirely, (iii) the decision-governance evaluation lens (does the spliced/edited cache still govern the tool decision, not just perplexity), and (iv) the CoT-gating result, which is a clean, surprising behavioral dissociation. The reframing itself — prefill as note-taking, the cache as a programmable memory — is the kind of lens that could reshape how people think about context engineering, and it connects neatly to Anthropic's forward-planning findings (rhyme plans stored on line-break tokens) as the backward-looking counterpart.

What to be skeptical about. This is a single-author paper, largely on curated synthetic gated decisions; the off-template controls (multi-hop, conversational phrasing, τ²-bench retail) help, but "decision-identical" is measured mostly on tasks designed to have a crisp gated decision. The paper's own stress tests reveal the soft edges: discrete decision agreement drops to 0.79–0.99 near action boundaries where sub-percent logit differences flip greedy choices, and CoT chain agreement is only 0.31–0.78 even when the final decision is faithful. The near-verbatim attribute-lookup case bounds the mechanism claim (field-only recovery 0.25–0.63 there — when the answer is partly a copy of the field, the field's KV does matter). field+selective@K is presented as unreliable, correctly. And the whole approach requires a per-token attention KV cache: pure SSMs, RWKV, diffusion LMs, and sequence-compressed caches (DeepSeek-V4-class) are out of scope or open, and hybrid models only get the attention fraction. The erratum also grows the context — fine for occasional field changes, less clear for rapidly-churning state, though the 28-turn toggle test is reassuring.

If it holds up. The near-term win is a serving pattern anyone can adopt today: append-only errata instead of in-prefix mutation keeps prefix caches hot, and precompiled skill/memory libraries become linear-cost cache objects. The longer-term suggestion in the conclusion is the more provocative one: if prefill routinely deposits reusable conclusions into the cache, models could be trained to expose addressable, splice-able notes — turning the KV cache from an append-only log into programmable memory.

Where to spend your time: Section 3 plus Appendix C is the heart — the four causal probes and the dissociation/injection controls are what elevate this from a systems trick to a mechanism claim. Then Figure 2c (the CoT gating) and Section 9 for the serving numbers. Code is released, including the vLLM harness.