Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory

TL;DR: Every retrieval-based agent memory system rests on an unstated assumption: a memory that matters to a query will look like that query. This paper names the case where that assumption fails — a tree-nut allergy stored weeks ago should change the answer to "give me a macaron recipe," but the two texts share nothing a retriever can see — calls it the implicit-association blind spot, and builds a benchmark (InMind) whose paired controls cleanly isolate it. The verdict is stark: six modern memory systems (vector, graph, agentic) recall stored facts on demand at up to 100%, and the backbone LLM answers 84% of the indirect questions when the fact is placed in context, yet the same systems apply the fact end-to-end at most 14.4% of the time. An 8× bigger embedding barely moves it. A dumb 200-line markdown profile kept always-in-context recovers most of the gap (68.8%). The failure isn't storage, knowledge, or representation — it's the query-conditioned retrieval interface itself.

The assumption nobody wrote down

The standard architecture for long-term agent memory is inherited wholesale from RAG. A writer distills what the user says into records; when a new query arrives, embed it, fetch the nearest records, paste them into the prompt. Formally: \hat{\mathcal{M}} = \mathrm{Retrieve}(\mathcal{M}; q, \theta), then a = \mathrm{LLM}(q, \hat{\mathcal{M}}), where \mathcal{M} is the memory store, q the query, and \theta the retriever's parameters (index, scoring rule, traversal policy).

The paper's first contribution is simply to state the hypothesis this design silently bets on: if a memory is necessary to answer a query, it is recoverable by a relevance score computed from the query alone. Call it the retrieval hypothesis. It holds trivially for explicit recall — "what am I allergic to?" is itself a retrieval cue for the allergy record. It fails whenever relevance is created by world knowledge that lives in neither text. "I have a tree-nut allergy" and "any good macaron recipes?" share no token, no topic, and no embedding-space neighborhood. What connects them — macarons are made with almond flour, and almond is a tree nut — is a fact about French patisserie stated in neither turn.

The structural bind is worth savoring: the retriever must commit to a relevance judgment before the LLM — the only component that holds the bridging knowledge — ever sees the candidates. The one system component that could recognize the connection is consulted only after the connection has already been missed.

The failure is also doubly hidden, which is what makes it dangerous. Users calibrate trust by asking the agent what it remembers — precisely the interaction retrieval handles perfectly. An agent that answers "you're allergic to tree nuts" and then hands you an almond-flour recipe has earned trust it does not merit, and no natural user probe would reveal it. Appendix 12 shows a real evaluated system (xMemory) doing exactly this, verbatim.

The benchmark as a differential diagnosis

When a system fails an indirect query, three explanations compete, and they demand entirely different fixes: (a) the fact was never stored, (b) the model lacks the bridging knowledge, (c) the fact was stored but never surfaced. Existing benchmarks conflate them. InMind's design is essentially a controlled experiment that separates all three per task:

• Direct naive query ("what am I allergic to?") — tests storage and survival. If this passes, the fact was written and retained.
• In-context control — the target memory is placed directly in the prompt with the indirect query. If this passes, the model has the bridging knowledge.
• Answer-blind target recall — a judge checks whether the answerer's actual retrieved context contained the fact, without seeing the answer. This pins the failure to retrieval, before generation.
• Indirect application — the end-to-end score: does the answer actually apply the bridge?

Construction matters here, and two choices are unusually careful. First, an adversarial similarity filter: 1,000 LLM-extracted candidates were scored with BM25 and MiniLM, and 700 were discarded because the memory was too lexically or semantically close to its query — an ordinary retrieval cue disqualifies a pair. A held-out check with a different encoder (BGE-small) confirms that in InMind, unlike LoCoMo, LoCoMo-Plus, or LongMemEval, the target memory sits no closer to its query than random distractors. There is genuinely nothing for a similarity retriever to exploit.

Second, source grounding: 113 of the 125 tasks trace to citable public documents — FDA allergen guidance, OSHA noise-exposure rules, USCIS travel restrictions for pending I-485 applicants, CPSC recalls. This addresses a subtle circularity in LLM-authored benchmarks: if a model generates the association at construction time, the same prior will tend to re-derive it at test time, so such benchmarks preferentially sample connections models already find natural. Anchoring bridges to expert documents breaks that loop.

The evaluation protocol injects each fact into the middle of a fixed 47-session conversation trace, so the memory must survive 38 sessions of realistic interference before it's queried.

The result: a seventy-point cliff

The signature the theory predicts is distinctive: in-context high, direct recall high, indirect application collapsed. No competing explanation predicts all three — hard tasks would also defeat the in-context model, forgetting would defeat direct recall, a weak model would fail everywhere. The data delivers exactly this signature.

With the fact in context, GPT-5-mini answers 84.0% of indirect queries (Wilson 95% CI: [76.6, 89.4]). Under retrieval, no configuration exceeds 16.0% — and that best score belongs to naive RAG over raw chunks, not to any of the sophisticated memory systems, which top out at 14.4% (MemoryOS). The answer-blind target recall column is the smoking gun: under indirect queries the decisive fact reaches the model's context in only 0.8–5.6% of MiniLM runs and 2.4–12.0% of emb3-large runs. The fact is stored, the model can use it, and it is simply absent at the only moment it matters.

Two specific results deserve extra attention:

Iteration doesn't help. A-RAG is an agentic searcher — it plans, issues keyword and semantic searches, reads chunks, and loops up to fifteen times. It scores 4.8% and 7.2%, among the lowest. The paper's diagnosis is sharp: every sub-query the agent issues comes from the same blind representation. Even a bridge-aware probe like "what about this user could make lilies dangerous?" retrieves nothing, because no stored fact resembles that question either. The probe that works is "does the user keep a cat?" — phrased in the vocabulary of the stored fact, which requires the searcher to first recall unprompted that lilies are toxic to cats, then enumerate what lilies might endanger, and query each guess. Searching harder cannot substitute for hypothesizing the bridge.

Bigger embeddings don't fix it. Swapping MiniLM (384-dim) for text-embedding-3-large (3,072-dim) raises target recall for every system — strong embeddings have absorbed some world knowledge during training, so "grapefruit" drifts a little closer to the drugs it interacts with. But the gains recover a few points of a seventy-point gap. The mechanism argument is the interesting part: an embedding can only bridge associations that left a distributional trace in training text as co-occurrence, and the bridges here are pharmacological, legal, religious, developmental — mostly never written down as co-occurring phrases. Absorbing world knowledge into a similarity score approximates a world model from the wrong direction; the residual is what InMind measures.

The 200-line markdown file that embarrasses everything

If the query-conditioned interface is the culprit, removing it — while changing nothing else — should recover the gap even for a crude method. So the authors build the dumbest possible always-in-state memory: one markdown file, capped at 200 lines / 25 KB, rewritten by GPT-5-mini after each session, prepended whole to the system prompt. No embeddings, no index, no retrieval. Formally, a = \mathrm{LLM}(q, s_t): a state s_t reaches the model before the query arrives.

68.8% versus at most 16.0% for anything query-conditioned, with direct recall matched at 98.4%. One design change — memory visible before the query, so the LLM discovers the cat–lily bridge itself — outweighs the accumulated apparatus of vector stores, knowledge graphs, and agentic search.

The most instructive contrast is with MemoryOS, which already keeps an always-visible user profile alongside its retrieved store and still scores 14.4%, with target recall of only 7.2% — the decisive fact rarely reached even its persistent slice. So persistent visibility alone isn't the answer either; the real variable is what earns a place in the visible state. MemoryOS promotes facts by heat-thresholded recency and frequency, proxies that can't see the distant bridge that will one day make a fact decision-critical. The paper explicitly proposes no system: the probe is a diagnostic, and its finite context budget obviously doesn't scale. What it isolates is routing — deciding at write time which facts must stay visible, and at read time when a query justifies expensive search over the full record — as the open problem, and InMind as the first benchmark that scores that decision directly.

How much to believe

The core sixty-to-seventy-point effect is robust — it's consistent across six independently built systems, two embedding scales, and three retriever families, and no alternative explanation (difficulty, forgetting, weak model) predicts the full signature. The triage question of whether these are "really retrieval problems rather than reasoning gaps" is answered by the design itself: the in-context control shows the model reasons correctly 84% of the time once the fact is visible, and answer-blind target recall shows the fact almost never arrives.

Still, several caveats, most of which the authors state themselves:

• n = 125. Per-system orderings (e.g., HippoRAG 2 declining with the bigger embedding) sit inside ±4–5 point binomial noise. Only the big effects are trustworthy.
• GPT-5-mini is both answerer and judge. The human audit (Appendix 14.2) found the context-aware application judge only 85% accurate, with all 15 errors being false positives — cases where the model produced a generic allergy caveat from world knowledge alone and the judge credited it as memory use. That means the retrieval systems' already-low application scores are, if anything, inflated. It also explains the odd rows where application exceeds target recall.
• No penalty for over-warning. An agent tuned to spray cautions everywhere would score well; measuring that needs negative controls the benchmark doesn't have yet.
• The probe isn't a controlled ablation. It differs from the retrieval systems in updater and state format, not just interface, so the 68.8% demonstrates sufficiency of pre-query visibility rather than isolating it perfectly.
• Domain skew. Tasks lean heavily toward health and safety, where failure is unambiguous — reasonable for a diagnostic, but the associative structure in etiquette, long-term goals, or institutional policy remains unmeasured.
• Always-in-state has real costs the paper flags: a full profile in every context broadens privacy exposure, and facts crowd each other out as volume grows.

Why this matters

The uncomfortable reading, which the paper states plainly in its conclusion, is that the memory-systems field has spent its effort where the problem already worked. Storage, indexing, graph construction, and iterative search all improve what an agent finds once it knows what to look for — and knowing what to look for is the whole difficulty. Query-conditioned retrieval asks a similarity function for a judgment that requires a world model, then consults the world model afterwards. The practical implication for anyone building agents today is blunt: a diligently maintained profile in the system prompt will beat your vector store on exactly the queries where being wrong is expensive, and your users cannot detect the difference by asking the agent what it remembers.

If you read two things, read Section 5 (the always-in-state probe, the MemoryOS contrast, and the routing framing — the intellectually sharpest part) and Appendix 12 (the verbatim Task-155 transcript, where a system recites the user's tree-nut allergy and then cheerfully walks them through an almond-flour macaron recipe with a mango ganache). The failure mode is abstract until you watch it happen.