Everyone is building AI memory systems.
But how do we know which ones actually work?
As AI agents move from one-off interactions toward long-term collaboration, memory is becoming a core capability. Yet evaluating memory systems fairly is surprisingly difficult.
Different systems often use different datasets, answer models, prompts, and evaluation methods. When the final score changes, it can be hard to tell whether the difference comes from the memory system itself or from the evaluation setup.
That's why we built Agent Memory Leaderboard (AML).
Why Do We Need a Memory Benchmark?
Memory is more than storing conversation history.
A useful memory system needs to retrieve relevant information, connect information across time, handle changing states, and provide useful context for an agent's current task.
But there hasn't been a common evaluation environment where different memory approaches can be compared under the same conditions.
AML was created to provide that common ground.
The first edition was jointly initiated by nearly 30 universities and research institutions and covers two evaluation tracks:
- Open-source Methods — Text Memory
- Commercial Products — Text Memory
As of August 12, 2026:
136 teams registered for the competition
67 representative memory frameworks successfully completed the first evaluation
The AML website surpassed 200,000 clicks
The website passed 100,000 clicks within the first 10 days
The first leaderboard results are now live.
Making Memory Systems More Comparable
One of the main challenges we wanted to address was evaluation consistency.
In a typical setup, a memory system may be evaluated together with a particular answer model, prompt, or judge.
That makes direct comparison difficult.
A higher score could come from a better memory system — but it could also come from a stronger downstream model or a different evaluation setup.
AML tries to separate these components.
The core interface for participating memory systems is:
Memory System
Add → Search
The memory system receives long-term history through Add, and returns relevant memories through Search.
Then AML handles:
AML Evaluation Platform
Answer → Eval
Answer generation and evaluation are completed by the benchmark platform under the same evaluation process.
This helps reduce the impact of different answer models, prompts, judges, and scoring conventions.
The goal is simple:
Compare memory systems under the same conditions as much as possible.
Memory Is More Than Retrieval
A memory system shouldn't be judged only by whether it can retrieve something that looks similar. Memory quality is not only about retrieving similar information, but about understanding relevance, context, time, and task requirements.
For text memory, AML evaluates multiple capabilities, including:
- Factual recall
- Relational and multi-hop reasoning
- Temporal and event understanding
- Memory governance
- Personalization and care
- Rules and workflow execution
- Epistemic safety and privacy
This matters because real-world agent memory is not just a search problem.
An agent may need to understand:
- What happened before?
- When did it happen?
- What changed afterward?
- Which previous experience is relevant now?
- Which information should be trusted?
- How should that memory affect the current task?
A useful memory system needs to handle these questions together.
The First Results
The first AML evaluation has now been completed.
67 representative memory frameworks successfully completed the evaluation across two tracks covering text memory for both open-source methods and commercial products.
The complete rankings, scores, and system versions are available on the leaderboard.
Leaderboard:
[https://agentmemoryleaderboard.ai/leaderboard/industry/textual ]
What Comes Next?
The first leaderboard is not the finish line.
We plan to keep AML running as a long-term evaluation and public leaderboard for agent memory systems.
Going forward, we will publish deeper technical analyses of the first-round results, including:
- Technical architectures that performed well
- Interesting evaluation patterns
- Performance across individual memory capabilities
- Differences between text and code memory
- What current results tell us about the evolution of agent memory
We also want the benchmark itself to evolve.
If you are working on AI agents, memory systems, evaluation, or related research, we would love to hear what you think a useful memory benchmark should measure next.
Four Evaluation Tracks
To better reflect different types of memory systems, AML organizes evaluation into two tracks:
Open-source Methods
- Text Memory
Commercial Products
- Text Memory
Each track evaluates systems under the same benchmark framework.
Explore AML
Leaderboard:
[https://agentmemoryleaderboard.ai/leaderboard/industry/textual]
GitHub:
[https://github.com/AML-memory/agent-memory-leaderboard]
Hugging Face Space:
[https://huggingface.co/agent-memory-leaderboard]
X:
[https://x.com/AgentMemoryL/status/2087544165433590240]
Contact:
contactus@agentmemoryleaderboard.ai
The first results are out.
Now the real work begins:
Top comments (17)
Does the Add and Search interface assume memory means retrieval? A system that consolidates at write time into a running profile, or folds memory into the model itself, has no clean Search call to expose. I get why fixing the interface makes scoring fair, I'm just wondering whether it also narrows what counts as a memory system.
I was wondering the same thing! I emailed the organizers because I originally hoped to enter my latent-memory system. I hope AML supports this kind of memory soon—I’d be excited to join a latent-memory track in the next round.
Great question — this is a very crucial observation.
The Add/Search interface defines the evaluation boundary of AML’s current Text Memory Track; it is not our full definition of what constitutes a memory system.
Systems that perform consolidation at write time are still fully compatible: the Add operation can be used to update a long-term profile, state graph, or structured summary, and the Search operation simply returns the portion of state relevant to the current query. Internally, it does not have to rely on vector retrieval — it may not even involve a traditional “search” process at all.
However, if memory resides entirely within model weights or hidden states, and cannot expose relevant memory independently before the final answer is generated, forcing it into a Search wrapper would indeed risk conflating memory capability with the answer model’s own ability.
So you are exactly right: this interface improves attribution clarity, but it does trade off some architectural coverage. More precisely, this track benchmarks systems that can provide task-relevant memory evidence through a standard interface — not every possible memory paradigm.
Approaches like in-model memory consolidation and continual learning will be covered in a dedicated track in the future. Stay tuned！
I think benchmarking agent memory gets especially interesting once you separate “can it retrieve the memory?” from “should the agent still trust that memory?”
A system can have excellent retrieval accuracy while still surfacing an outdated architectural decision or a constraint that no longer applies.
I’d be curious whether your benchmark includes temporal drift — not just whether the right memory is retrieved, but whether the system can recognize when an old memory has become invalid.
Interesting timing for this benchmark.
As AI agents move from short conversations to long-term workflows, memory is becoming more than simple retrieval.
The next generation of memory systems will likely need better memory formation, updating, forgetting, and reasoning capabilities — not just storing more context.
Well said.
Long-term memory is not just about retrieving past information. A capable memory system needs to understand what should be remembered, updated, or discarded over time.
This is also why we believe standardized evaluation is important — the community needs better ways to measure these emerging capabilities.
I'd actually be curious whether the multi-hop score and the temporal score cluster the same systems at the top. In a project we did last year, a system that handled renamed companies decently fell apart on chains involving former subsidiaries - the retrieval was technically correct for the parent but wrong for the chain the agent was following. It didn't show up until we dug through the failures case by case. Don't know if your test set covers that specific flavor.
Excellent example — this perfectly illustrates that solid performance on multi-hop tasks and temporal tasks separately does not equal genuine temporal multi-hop reasoning capability.
Our current benchmark suite covers multi-hop, temporal reasoning and knowledge-update questions, but we do not claim these category labels alone fully capture the specific combined scenario you described.
A more meaningful analysis would be to verify whether top-performing systems cluster consistently across both multi-hop and temporal scores, and to further examine the joint failure modes of the two task types — rather than only reporting separate average scores. We are already working on this analysis!
Have you encountered other interesting combined failure patterns in your work? We’d love to discuss more about how to design better test cases for these edge scenarios.
Holding the answer model, prompt and judge fixed while varying only the memory system is the correct design, and it is the thing most leaderboards get wrong, so the interface is worth reading even for people who never enter. The question I would ask of the first results is what the spacing between ranks means. With 67 systems scored against one evaluation set, a lot of adjacent ranks are going to sit inside each other's intervals, and a table of point scores gets read as a total order that the data will not support. A per-system interval, or even just the evaluation-set size plus a bootstrap over it, would let readers see where the genuine tiers are instead of inferring 67 of them. I would also expect judge disagreement to be highest on the multi-hop and temporal categories specifically, so a per-category agreement number would carry more information than the headline.
Great point, we fully agree!
For task types like ScriptMem, we run evaluation via direct selection matching, so the determinism and reliability of the process are well guaranteed.
For benchmarks such as LoCoMo and BEAM, we do use an LLM Judger to compare responses against reference answers or rubrics. Our judger has been manually calibrated — see details in the LoCoMo Refined repo — and all rubrics are strictly aligned with official specifications, as documented in the BEAM repo. All of the above have been open-sourced in our official repository: GitHub - AML-memory/agent-memory-leaderboard.
Fair evaluation is both essential and challenging. We are working hard to address these concerns and always welcome feedback from the community.
We’d love to hear your further thoughts — do you have specific suggestions on how we can better measure and report per-category judge agreement?
The biggest challenge in AI memory today is not building more memory systems, but knowing how to evaluate them fairly.
Different datasets, models, and judging pipelines make comparisons difficult. A standardized evaluation framework like AML could become an important step toward making memory systems more measurable and reproducible.
Exactly. One of the main motivations behind AML was addressing this evaluation gap.
Memory systems should be evaluated independently from the underlying agent framework, with the same data, answer model, and judging pipeline.
We hope AML can help the community build a more transparent and reproducible standard for AI memory evaluation.
What stood out to me was how AML separates adding and searching memories from the rest of the evaluation🤯. It makes the role of the memory layer much clearer and feels like a fairer way to compare different systems than treating the whole agent as a black box. It’s also easier to see whether the gains actually come from the memory system itself.
Thanks for sharing this perspective!
One of the key motivations behind AML was exactly this: making the memory layer more measurable instead of evaluating the entire agent as a black box.
We hope this can help the community better understand what makes a memory system effective. Appreciate your feedback!
yeah! nice job
Standardizing everything except the memory system is exactly right, otherwise you are measuring the harness, not the memory. The dimension I'd want isolated next is temporal: how the system handles facts that change over time versus facts that just accumulate, since those fail very differently. How are you scoring a retrieval that is stale but was correct when it was written?
This is one of the most critical distinctions in temporal memory evaluation: a piece of memory being "stale" does not mean it is incorrect for every query.
If the query asks about the current state but the system only returns an outdated state superseded by new facts, we classify it as stale retrieval. If the query refers to a specific point in the past, that same old fact may instead be the correct piece of evidence. Even if a system returns both old and new states with clear timestamps or validity windows, that is not necessarily wrong — it may be correctly preserving the full evolution trajectory of states.
This is truly an excellent suggestion. Our organizing committee will discuss whether we can address this through more fine-grained category design. Feel free to share any further ideas you may have!
We’d also love to hear your take on how to best structure scoring for dynamic state-update scenarios — what do you think would be the most fair and informative approach?