In short: The right evidence is usually there. Memory systems fail at weighing and applying it, especially when it has changed. Better search will not fix that. Giving every piece of evidence a standing will.
Key points
- A 2026 benchmark found relevant evidence retrievable about ten times more often than missing; failures came in how it was used.
- On trajectory questions, the best systems answered about half correctly on fresh data and roughly one in five once stores filled.
- Tag each record with status, validity dates, provenance and explicit links to the claims it superseded.
- Answer change questions with an ordered timeline of states, not top-k similarity fragments.
- Production graphs without automated refresh drifted 15–20% from ground truth per quarter in one practitioner benchmark.
Most memory roadmaps assume the hard problem is finding the right fact. So the investment goes into better embeddings, a knowledge graph, re-ranking, a bigger context window. The independent evidence of the last year points somewhere else.
What the benchmarks actually found
A 2026 benchmark probed 835 knowledge points across 13 memory configurations. The relevant evidence was retrievable about ten times more often than it was missing. The failures came after retrieval, in how evidence was used. Its authors' conclusion was blunt: the main bottleneck is evidence use, not retrieval.
The weakest category was trajectory questions: how did this change, and when. The best systems answered about half of those correctly on fresh data, and fell to roughly one in five once their stores had filled up. Performance got worse as memory grew, which is the opposite of what a brain should do.
This is consistent with earlier work. A 2025 long-memory benchmark found long-context models lost 30–60% of their accuracy relative to perfect retrieval, and did worst on temporal reasoning and knowledge updates.
What it looks like in practice
The system retrieves both the March view and the June view of the same account. It blends them into one confident paragraph, or it picks whichever is written more fluently. Nothing in the store says which one superseded which, who asserted each, or whether either was verified. The model is asked to adjudicate with no basis for doing so.
Give evidence a standing
The fix is structural, and it is mostly about metadata the model can reason over.
Every record carries a status: current, superseded, disputed or unverified. It carries validity dates: when the claim became true and when it stopped. It carries provenance: who asserted it, from what source, and whether anyone checked. And where one claim replaced another, there is an explicit link between them.
Retrieval then returns evidence with its standing, and the instructions tell the model how to use standing. Verified beats unverified. Current beats superseded. Primary source beats summary. When two current claims conflict, say so rather than choose.
Answer change questions with a timeline
For trajectory questions, top-k retrieval is the wrong shape. Ten relevant fragments sorted by similarity hide the order of events. Return the ordered sequence of states for the entity in question, with dates and the reason for each change. The model is much better at reading a timeline than reconstructing one.
A note on graphs
Graphs help with some of this and cost more than teams expect. One practitioner benchmark found vector search better for specific lookups and graph retrieval roughly three to four times better for aggregation and cross-document questions. The same work found production graphs without automated refresh drifting 15–20% from ground truth per quarter, and teams spending two to three times more engineering on freshness than on the initial build. Budget for upkeep, or do not build the graph.
In practice
- Add status, validity dates, provenance and supersession links to the records your agents retrieve.
- Write explicit precedence rules into the instructions that consume them.
- Build twenty trajectory questions from your own history ("how did our view of X change in 2026?") and score your system on them.
- If you run a graph, put its refresh cost in the budget before its build cost.
What would prove this wrong. A retrieval-only improvement, with no standing metadata, that closes most of the gap on trajectory and knowledge-update questions in an independent benchmark.
Common questions
Why do AI memory systems give wrong answers when the data is there?
The failure comes after retrieval, in how evidence is used. A 2026 benchmark of 835 knowledge points across 13 memory configurations found the relevant evidence retrievable about ten times more often than it was missing.
How do I stop an AI agent mixing up old and new information?
Give each record a status (current, superseded, disputed or unverified), validity dates, provenance and links to what it replaced, and write precedence rules: verified beats unverified, current beats superseded, primary source beats summary, and conflicting current claims are flagged.
Are knowledge graphs better than vector search for RAG?
One practitioner benchmark found vector search better for specific lookups and graph retrieval roughly three to four times better for aggregation and cross-document questions. But graphs without automated refresh drifted 15–20% per quarter, so budget for upkeep or do not build one.
How should an AI system answer questions about how something changed over time?
Return the ordered sequence of states for the entity, with dates and the reason for each change, rather than top-k fragments sorted by similarity. The model is much better at reading a timeline than reconstructing one.