Everyone is building AI memory systems.
But how do we know which ones actually work?
As AI agents move from one-off interactions toward long-term...
For further actions, you may consider blocking this person and/or reporting abuse
Does the Add and Search interface assume memory means retrieval? A system that consolidates at write time into a running profile, or folds memory into the model itself, has no clean Search call to expose. I get why fixing the interface makes scoring fair, I'm just wondering whether it also narrows what counts as a memory system.
I was wondering the same thing! I emailed the organizers because I originally hoped to enter my latent-memory system. I hope AML supports this kind of memory soon—I’d be excited to join a latent-memory track in the next round.
Great question — this is a very crucial observation.
The Add/Search interface defines the evaluation boundary of AML’s current Text Memory Track; it is not our full definition of what constitutes a memory system.
Systems that perform consolidation at write time are still fully compatible: the Add operation can be used to update a long-term profile, state graph, or structured summary, and the Search operation simply returns the portion of state relevant to the current query. Internally, it does not have to rely on vector retrieval — it may not even involve a traditional “search” process at all.
However, if memory resides entirely within model weights or hidden states, and cannot expose relevant memory independently before the final answer is generated, forcing it into a Search wrapper would indeed risk conflating memory capability with the answer model’s own ability.
So you are exactly right: this interface improves attribution clarity, but it does trade off some architectural coverage. More precisely, this track benchmarks systems that can provide task-relevant memory evidence through a standard interface — not every possible memory paradigm.
Approaches like in-model memory consolidation and continual learning will be covered in a dedicated track in the future. Stay tuned！
I think benchmarking agent memory gets especially interesting once you separate “can it retrieve the memory?” from “should the agent still trust that memory?”
A system can have excellent retrieval accuracy while still surfacing an outdated architectural decision or a constraint that no longer applies.
I’d be curious whether your benchmark includes temporal drift — not just whether the right memory is retrieved, but whether the system can recognize when an old memory has become invalid.
Interesting timing for this benchmark.
As AI agents move from short conversations to long-term workflows, memory is becoming more than simple retrieval.
The next generation of memory systems will likely need better memory formation, updating, forgetting, and reasoning capabilities — not just storing more context.
Well said.
Long-term memory is not just about retrieving past information. A capable memory system needs to understand what should be remembered, updated, or discarded over time.
This is also why we believe standardized evaluation is important — the community needs better ways to measure these emerging capabilities.
I'd actually be curious whether the multi-hop score and the temporal score cluster the same systems at the top. In a project we did last year, a system that handled renamed companies decently fell apart on chains involving former subsidiaries - the retrieval was technically correct for the parent but wrong for the chain the agent was following. It didn't show up until we dug through the failures case by case. Don't know if your test set covers that specific flavor.
Excellent example — this perfectly illustrates that solid performance on multi-hop tasks and temporal tasks separately does not equal genuine temporal multi-hop reasoning capability.
Our current benchmark suite covers multi-hop, temporal reasoning and knowledge-update questions, but we do not claim these category labels alone fully capture the specific combined scenario you described.
A more meaningful analysis would be to verify whether top-performing systems cluster consistently across both multi-hop and temporal scores, and to further examine the joint failure modes of the two task types — rather than only reporting separate average scores. We are already working on this analysis!
Have you encountered other interesting combined failure patterns in your work? We’d love to discuss more about how to design better test cases for these edge scenarios.
Holding the answer model, prompt and judge fixed while varying only the memory system is the correct design, and it is the thing most leaderboards get wrong, so the interface is worth reading even for people who never enter. The question I would ask of the first results is what the spacing between ranks means. With 67 systems scored against one evaluation set, a lot of adjacent ranks are going to sit inside each other's intervals, and a table of point scores gets read as a total order that the data will not support. A per-system interval, or even just the evaluation-set size plus a bootstrap over it, would let readers see where the genuine tiers are instead of inferring 67 of them. I would also expect judge disagreement to be highest on the multi-hop and temporal categories specifically, so a per-category agreement number would carry more information than the headline.
Great point, we fully agree!
For task types like ScriptMem, we run evaluation via direct selection matching, so the determinism and reliability of the process are well guaranteed.
For benchmarks such as LoCoMo and BEAM, we do use an LLM Judger to compare responses against reference answers or rubrics. Our judger has been manually calibrated — see details in the LoCoMo Refined repo — and all rubrics are strictly aligned with official specifications, as documented in the BEAM repo. All of the above have been open-sourced in our official repository: GitHub - AML-memory/agent-memory-leaderboard.
Fair evaluation is both essential and challenging. We are working hard to address these concerns and always welcome feedback from the community.
We’d love to hear your further thoughts — do you have specific suggestions on how we can better measure and report per-category judge agreement?
The biggest challenge in AI memory today is not building more memory systems, but knowing how to evaluate them fairly.
Different datasets, models, and judging pipelines make comparisons difficult. A standardized evaluation framework like AML could become an important step toward making memory systems more measurable and reproducible.
Exactly. One of the main motivations behind AML was addressing this evaluation gap.
Memory systems should be evaluated independently from the underlying agent framework, with the same data, answer model, and judging pipeline.
We hope AML can help the community build a more transparent and reproducible standard for AI memory evaluation.
What stood out to me was how AML separates adding and searching memories from the rest of the evaluation🤯. It makes the role of the memory layer much clearer and feels like a fairer way to compare different systems than treating the whole agent as a black box. It’s also easier to see whether the gains actually come from the memory system itself.
Thanks for sharing this perspective!
One of the key motivations behind AML was exactly this: making the memory layer more measurable instead of evaluating the entire agent as a black box.
We hope this can help the community better understand what makes a memory system effective. Appreciate your feedback!
yeah! nice job
Standardizing everything except the memory system is exactly right, otherwise you are measuring the harness, not the memory. The dimension I'd want isolated next is temporal: how the system handles facts that change over time versus facts that just accumulate, since those fail very differently. How are you scoring a retrieval that is stale but was correct when it was written?
This is one of the most critical distinctions in temporal memory evaluation: a piece of memory being "stale" does not mean it is incorrect for every query.
If the query asks about the current state but the system only returns an outdated state superseded by new facts, we classify it as stale retrieval. If the query refers to a specific point in the past, that same old fact may instead be the correct piece of evidence. Even if a system returns both old and new states with clear timestamps or validity windows, that is not necessarily wrong — it may be correctly preserving the full evolution trajectory of states.
This is truly an excellent suggestion. Our organizing committee will discuss whether we can address this through more fine-grained category design. Feel free to share any further ideas you may have!
We’d also love to hear your take on how to best structure scoring for dynamic state-update scenarios — what do you think would be the most fair and informative approach?