AI Memory Benchmarks: The Complete Guide (2026)

If you're building a complex agentic system, you may need an AI agent memory layer to keep information available across sessions and tasks. But the next question is how to choose the right memory architecture — and how to tell whether it actually works.

This guide is a practical map of the benchmarks used to evaluate AI memory today: what each one tests, how it's constructed, and where to find the paper and data. We also discuss the gaps in existing benchmarks and how to evaluate memory for your own system.

TL;DR

• A memory benchmark is a standardized test that lets different memory systems be compared on the same task and data.

• Benchmarks fall into four groups. Conversational benchmarks (LongMemEval, LoCoMo, BEAM) test recall across sessions. Multi-hop QA benchmarks (HotPotQA, MuSiQue) test connecting facts across sources. Long-context benchmarks (NIAH, RULER) test attention over one fixed input. Agentic benchmarks (MemoryArena, STATE-Bench) test whether memory improves actions.

• Benchmark size matters. If the full history fits comfortably inside the model's context window, a strong baseline is to skip the memory layer and pass everything directly to the model. Larger benchmarks such as BEAM are more useful for testing memory itself because they force the system to store and retrieve information outside the prompt.

• Every benchmark has blind spots and limitations. They tend to evaluate the final LLM answer based on the memory, not the structure behind it. They also saturate as models improve, and scores can vary on re-run due to implementation details.

• Benchmarks are a directional signal, not a verdict. The evaluation that matters most is downstream: whether memory improves your agent on your tasks.

• Cognee currently holds state-of-the-art results on BEAM, the most demanding conversational benchmark, with conversations up to 10M tokens.

What is an AI benchmark

AI memory benchmarks are one type of AI benchmark, so let's recap what those are. AI benchmarks are standardized tests used to assess LLM performance across different tasks. Typically, they check whether a model can produce a known correct response to a given input — answer a question, complete a piece of code, solve a math problem, correctly refuse an unsafe request, and so on. Because the expected outputs are known in advance, results can be scored automatically and compared across models.

As a general principle, each benchmark consists of three parts:

• A dataset — the collection of inputs the system is tested on.

• A task — what the system is asked to do with each input, which could range from simply answering the question, to executing a specific agentic task, like filling a tax form.

• A metric and evaluation protocol — how responses are scored. Depending on the task this could be an exact match, a deterministic software test, or an LLM judge that compares the answer against reference or assesses a specific property like safety.

There are lots of benchmarks that test various LLM capabilities. For example, MMLU tests general knowledge across 57 subjects, GSM8K tests grade-school math, SWE-bench tests whether a model can resolve real GitHub issues. There are also benchmarks for instruction following, tool use, safety, and multilingual ability. The point is all the same: provide a standardized way to compare models fairly on a particular task. That is why new model releases often include a table of benchmark scores.

But an LLM benchmark tests only the LLM itself, and the model is just one part of the system you build around it. Imagine building an AI agent — for example, a coding agent, or a customer support assistant — that has to operate over a long horizon. Such a system has plenty of moving parts beyond the LLM, and one of them is memory. So there are now external memory benchmarks that help assess different memory architectures as a standalone component, much like LLM benchmarks like MMLU or SWE-bench let you compare LLMs.

What is an AI memory benchmark

But first, what is an AI agent memory?

An agent needs to work with far more information than fits in its context — company data, context from past sessions, tool outputs, documents, decisions already made. This data has to be stored somewhere, but more importantly it has to be organized so the right piece can be found and retrieved when the agent needs it later, and kept current as facts change. That is what an AI agent memory is for: a component that persists, structures and maintains accumulated information. Memory is distinct from the context window, which empties when the session ends, and from RAG, which reads from an existing corpus but doesn't write back. (We covered these distinctions in the guide to AI agent memory.)

Memory can be implemented in different ways — using vector stores, memory files, knowledge graphs, or hybrids of these. Each comes with its own trade-offs in retrieval quality, scale, and cost. A memory benchmark gives you a common test for comparing them under the same conditions. Let's now look at how this usually works.

Typically, a memory benchmark evaluates the write-and-retrieve loop. It provides a setup where information appears in one session or document, and a question about it arrives later — in another session, after many distracting turns, or after the original fact has been updated. Answering correctly requires the system to store the information and later retrieve the right version. (Notice how this differs from a retrieval or QA benchmark, where the corpus is given and fixed — here the system builds its own store first, and the outcome of that write step is part of what is being tested.) The benchmark then scores how often the final answers are correct.

This makes memory benchmarks mechanically different from standard LLM benchmarks. A standard LLM benchmark is a loop over rows or tasks: for each entry in the dataset, call the model once, compare the response to the reference, and add up the score. The system under test is usually just the model API — with prompt and anything extra in the harness supplied by the benchmark. Each test is stateless: nothing persists between calls.

A memory benchmark has a different structure, because of what it has to test. The harness cannot simply call the model — the whole point is to test whether the system retained information provided earlier. So a memory system has to exist and be running before the evaluation starts: for example, a service with persistent storage, such as a vector or graph database. Evaluation then happens in two phases: first the harness fills the memory, then it asks questions and scores the answers.

Phase 1 is ingestion. The benchmark harness feeds a history of conversations, or other source data, into the memory system piece by piece (for example, sending multiple chat sessions one after another). The memory system has to do the first part of its job — organize the incoming data in whatever way its architecture requires. That could be entity and relation extraction for a graph, chunking and embedding for a vector store, and so on. To make this possible, benchmarks typically provide an adapter interface — essentially add and search functions — so different memory systems can be plugged in. This can already make evaluation expensive: ingestion alone may require many LLM calls before a single question is asked.

Phase 2 is question answering. The benchmark provides queries with known answers, designed to test whether the stored memory can support the right response. Each query is then sent to the memory system (which already has all the history ingested), and the memory system retrieves the data that it considers relevant. A reader LLM then composes an answer from that retrieved context, and the benchmark scores the final response. Because answers are often free-form, this is commonly done with an LLM judge rather than exact matching.

Compared with a standard LLM benchmark, that's a lot of moving pieces. There is also some non-determinism left in the setup: both the reader LLM and the judge LLM can affect the result, and even different implementations of the "same" benchmark may use different models.

Also notice that the memory "itself" is not inspected by the benchmark. After ingestion, the system may hold extracted facts, nodes, chunks, or other internal structures. But the benchmark does not look inside that store — what was actually saved or how the memory is represented internally. Instead, it evaluates what the memory can retrieve, usually indirectly through the reader's final answer. Some benchmarks also log the retrieved data and let you score it against annotated evidence, but the main reported metric is almost always the final answer accuracy.

What we've described is more or less the standard architecture of a conversational AI memory benchmark. Individual benchmarks vary in the details, which we'll cover throughout the guide. But there are also several other benchmark families that are also useful for evaluating memory systems, or closely related capabilities. We can group the whole landscape into four categories:

• Conversational memory benchmarks test recall across sessions: a fact mentioned in one session must be recalled, connected, or updated later. These are directly focused on memory, and we cover them in the most depth — LongMemEval, LoCoMo, BEAM.

• Multi-hop QA benchmarks test whether a system can connect facts scattered across multiple sources. They weren't designed specifically for memory, but they measure a linking ability that memory systems often rely on.

• Long-context benchmarks test attention over a single long input. Here, there is no read or write step at all — we talk about in-context attention. But they are useful to explain some of the limitations memory layers are meant to address.

• Agentic and stateful benchmarks test whether memory improves an agent's actions that require keeping long-term state. This is the newest group.

All four appear in published memory evaluations, and are useful to understand. The table below summarizes them.

We'll go through each of them in depth, starting with the conversational group. But before that, there is one more dimension worth pointing out: benchmark dataset size.

AI memory benchmark sizes

Benchmarks vary in how much data the memory system has to work with. The ones we'll cover span roughly three orders of magnitude — from a few thousand tokens in older datasets to around 10 million in the largest recent ones.

Meanwhile, 128k is now a fairly common LLM context size, with several frontier models supporting around 1M tokens (including GPT-5.5, DeepSeek V4, Claude Opus 4.8 and others). That matters for memory evaluation: if the entire benchmark history fits inside the model's context window, an obvious baseline is to put all of it directly into the prompt and use no memory layer at all. At that point, you may be testing the model's long-context abilities more than the memory system itself.

So the more meaningful memory benchmarks are the ones where the history is large enough that the system actually has to store information outside the prompt and retrieve only what it needs. That makes BEAM, with histories up to 10M tokens, one of the most interesting benchmarks in this group.

Let's now look at all of those benchmarks in detail. The table below shows all the benchmarks we'll discuss in the guide:

LoCoMo

Paper (ACL 2024): Evaluating Very Long-Term Conversational Memory of LLM Agents | GitHub: snap-research/locomo | Project page: snap-research.github.io/locomo

LoCoMo (Long Conversational Memory) is one of the most widely reported memory benchmarks, though as we'll see, its scale is too small to really stress a memory system today. It simulates two people talking to each other over many sessions. The conversations were generated by LLM agents with assigned personas and timelines, then verified and edited by human annotators. Each speaker has a consistent life story, and events from earlier sessions come up again later — which makes the dataset useful for testing long-term memory rather than just comprehension.

The main evaluation is question-answering over 1,986 annotated QA pairs, divided into five categories:

• Single-hop questions — the answer appears in one session.

• Multi-hop questions — the answer requires combining information across sessions.

• Temporal questions — the answer depends on knowing when something happened ("when did Melanie sign up for a pottery class?")

• Open-domain questions — the answer requires the conversation plus common sense or world knowledge.

• Adversarial questions — the conversation does not contain enough information to answer, so the model should abstain. (These are often excluded from the testing).

The benchmark also defines another task useful for testing memory — event summarization. Given the conversation, it asks to reconstruct what happened in a speaker's life and when. This tests a different ability from QA: instead of retrieving one fact, the system has to assemble a coherent account from information scattered across many sessions.

LoCoMo dataset

In the paper, the dataset contains 50 conversations averaging around 300 turns and 9k tokens each, spread across up to 35 sessions. The public release keeps only the 10 longest, highest-quality conversations, which average around 600 turns and 26k tokens.

The data lives in the GitHub repo as a JSON file (locomo10.json). Each sample is one conversation and contains:

• the sessions themselves, with per-session timestamps and turn-level dialog,

• session summaries,

• annotated event summaries per speaker,

• the QA pairs, each with a category label and, in most cases, the IDs of the dialog turns that contain the evidence (these can help inspect retrieval separately: did the memory system retrieve the turns that actually contain the answer?)

Notice that while the public test set includes 1,986 QA pairs, 446 of those are adversarial questions. Many evaluations exclude them, which is why you may see LoCoMo results reported on only the remaining 1,540 questions.

The data needed to evaluate event summarization is also released, but the repository still marks the official evaluation code as "coming soon." So if you want to test that ability, you need to implement the scoring yourself.

LoCoMo summary

LoCoMo is useful for testing retrieval and reasoning across conversations. Earlier dialogue datasets rarely went beyond a handful of sessions, while LoCoMo extended that to month-long histories with recurring people and events. But its main limitation today is also scale. At around 26k tokens, a full LoCoMo conversation fits comfortably inside modern LLM context windows. So in practice, it is now better viewed as a test of long conversational reasoning than of large-scale memory systems.

LongMemEval

Paper (ICLR 2025): LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory | GitHub: xiaowu0162/LongMemEval | Project page: xiaowu0162.github.io/long-mem-eval | Dataset (HF): longmemeval-cleaned

LongMemEval tests whether a chat assistant remembers its user across sessions. The accompanying paper reports an interesting finding: long-context LLMs lose roughly 30-60% of their performance when they have to find information in a 115k-token interaction history rather than read only the relevant evidence sessions. This is one of the frequently cited arguments for using a dedicated memory layer.

The benchmark itself simulates a user talking to an assistant over a long period. Histories are built from user-assistant chat sessions, and the questions are the kind a user might ask to check whether the assistant remembers them, such as "what did I tell you?" or "what changed since?" The benchmark contains 500 questions embedded in these histories, each targeting one of five abilities:

• Information extraction — recall a specific fact stated in an earlier session.

• Multi-session reasoning — combine information across sessions to answer.

• Temporal reasoning — reason about when things were said or happened.

• Knowledge updates — the user's situation changed between sessions, and the answer must reflect the latest version, not the first one stored.

• Abstention — the question has no answer in the history, the correct behavior is to say so instead of confabulating one.

In practice, these five abilities are represented through seven question types. Single-session recall is split by source — facts stated by the user versus by the assistant — and there is also a preference type, where the response must stay consistent with what the user has said they like. This division became a useful structure for conversational memory evaluation, and many later benchmarks often reuse the same abilities or extend them with additional categories.

LongMemEval dataset

The benchmark is released in two sizes:

• LongMemEval-S — histories of roughly 115k tokens across around 50 sessions. This is the most commonly reported variant.

• LongMemEval-M — around 500 sessions and roughly 1.5M tokens.

The LongMemEval repository contains three JSON files (longmemeval_s, longmemeval_m, and longmemeval_oracle). The oracle-retrieval variant contains only the evidence sessions, and is useful for measuring how much of a system's error comes from retrieval versus reading.

Each test question comes with the full timestamped session history, the haystack session IDs, and human-annotated answer locations. This means retrieval can also be scored directly — the repository reports metrics such as Recall@k and NDCG@k — alongside the end-to-end answer quality. The evaluation script uses a prompt-engineered GPT-4o judge, which the authors report as having over 97% agreement with human experts.

In September 2025 the authors released a cleaned version of the dataset, fixing annotation issues in the original. Scores on the original and cleaned versions are therefore not strictly comparable, although published evaluations do not always make clear which version they use.

LongMemEval summary

Compared with LoCoMo, which focuses mostly on retrieval and reasoning, LongMemEval covers more memory-specific behaviors. It also tests whether the system keeps track of the current version of a fact through knowledge updates, and whether it abstains when the history contains no answer. That matters because a system can do well on straightforward recall while still failing when facts become stale or the answer is simply not there.

Its main limitation is again scale. The commonly used S variant fits within the context window of many modern models, although the release also includes the compilation pipeline, so longer histories can be generated from the same components.

BEAM

Paper (ICLR 2026): Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs | GitHub: mohammadtavakoli78/BEAM | Data (HF): BEAM, BEAM-10M | Project page: beam-light

BEAM simulates long-running conversations between a user and an AI assistant across 19 domains — from personal life and career planning to coding, health, and finance.

One construction detail makes it different from earlier benchmarks. Those often built long histories by stitching together separate sessions on largely unrelated topics. That can inadvertently make the retrieval easier: if sessions are independent, the system only needs to identify the one or few that match the question. BEAM instead builds long, narratively coherent conversations with a single user. The same people, topics, facts, and commitments come up repeatedly, and later information may contradict earlier statements. This creates harder distractors, because the system may need to choose between several mentions of the same entity or topic.

The evaluation includes 2,000 human-validated probing questions testing 10 abilities — the broadest set among the conversational benchmarks we cover. Three are introduced by BEAM:

• Contradiction resolution — the conversation contains conflicting statements, and the system should recognize the conflict and ask for clarification rather than choose sides.

• Event ordering — the answer must reconstruct the sequence in which things happened.

• Instruction following — instructions given early ("always format code snippets with syntax highlighting") must still be followed many turns later.

The other seven are familiar from the benchmarks above: information extraction, multi-hop reasoning, knowledge updates, temporal reasoning, abstention, preference following, and summarization.

The BEAM experiments also show what happens as conversation length grows. BEAM scores answers against rubric "nuggets" — the atomic facts a correct answer must contain — with each scored 0, 0.5, or 1 and averaged. A score of 0.25 therefore means the answer captured about a quarter of the required facts (not that a quarter of questions were answered correctly).

Long-context models score around 0.24-0.28 on the shortest tier, dropping to 0.10-0.13 at 10M tokens. Importantly, performance starts declining before the history exceeds the context window, with scores already falling at 1M tokens. Adding the structured memory system proposed in the paper improves results by 3.5-12.7% over the strongest baseline, depending on the model and conversation length, and by more than 100% for some models at 10M tokens, where the full history no longer fits in context.

BEAM dataset

The BEAM dataset contains 100 conversations across four length tiers: 20 at 128k tokens, 35 at 500k, 35 at 1M, and 10 at 10M. Each conversation has 20 test questions — two for each ability — for a total of 2,000 questions.

In the GitHub repository, each conversation is released as a folder containing the full dialogue, the generation plan, the topic specification, and the probing questions. The questions also carry more structure than in earlier benchmarks: each includes an ideal answer, a difficulty label, the source turn IDs it depends on, and a per-question rubric used by the judge.

BEAM summary

BEAM is currently one of the most complete conversational memory benchmarks. It covers a broad range of memory abilities, uses long narratively coherent conversations, and scales up to 10M tokens — beyond current model context windows. That makes the largest tiers particularly useful for testing actual external memory rather than only long-context reasoning.

We evaluated cognee on BEAM and published the full methodology and SOTA results in a separate deep-dive, with aggregate results on our research and evaluation page.

MemoryAgentBench

Paper (ICLR 2026): Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions | GitHub: HUST-AI-HYZ/MemoryAgentBench | Dataset (HF): ai-hyz/MemoryAgentBench

MemoryAgentBench takes a different angle from the benchmarks above. Instead of building one new conversational dataset, it defines four competencies a memory agent should have and constructs a test for each from existing corpora — including books, QA haystacks, labeled examples, and fact lists. Most of this source data is not conversational, but the benchmark turns it into a memory-like stream: everything is split into chunks — typically 4,096 tokens — that are fed in sequentially, so the system has to ingest and organize as it goes. Questions come only after the whole stream has been ingested, which puts it in the same two-phase setup we described.

The four competencies are:

• Accurate retrieval — find the right information in the accumulated history, including multi-hop cases.

• Test-time learning — infer a pattern from examples seen in the stream and apply it to new inputs, for example by classifying an event based on examples stored in memory.

• Long-range understanding — answer questions that require a global view of the entire stream, such as summarizing a full book or identifying the culprit in a detective novel.

• Selective forgetting — update or replace previously stored information when new evidence contradicts it. In the code, this is implemented as conflict resolution: facts are updated over time, and answers should use the latest version.

The dataset contains 2,071 questions over inputs ranging from 103k to 1.44M tokens. The data comes partly reformulated from existing benchmarks (RULER, InfBench, HELMET, LongMemEval) and partly from two newly built datasets: EventQA and FactConsolidation.

MemoryAgentBench is useful because it turns a diverse set of long-context tasks into an incremental memory setting (by feeding the source data in chunk by chunk instead of passing it all at once), which lets you test more than just retrieval. At the same time, much of the benchmark repurposes existing synthetic long-context datasets rather than introducing new interaction data.

PersonaMem

Paper (COLM 2025): Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale | Paper 2: PersonaMem-v2 | GitHub: bowen-upenn/PersonaMem, PersonaMem-v2 | Datasets (HF): PersonaMem, PersonaMem-v2

PersonaMem shifts the evaluation from "what did the user say?" to "what is this user like now?" Each benchmark sample represents a user with relatively stable attributes, such as demographics, and dynamic ones, such as preferences that change over time. The persona interacts with a chatbot across many sessions on topics like food and movie recommendations, travel planning, therapy, and legal or medical consultations. The benchmark then tests whether the system keeps track of how that user changes over time. There are two versions of it.

PersonaMem v1 contains 20 users with more than 180 simulated user-LLM interaction histories across 15 personalization tasks. It tests whether the model tracks the user's current state — preferences that shifted or facts that changed — rather than relying on what was true earlier. The release comes in three context sizes: 32k, 128k, and 1M tokens.

The main evaluation is multiple choice: given the conversation history and a new user request, the model chooses among four responses, with distractors based on outdated or irrelevant information. This makes the scoring deterministic rather than dependent on an LLM judge. Each question is also annotated with the distance to the most recent mention of the relevant preference, so accuracy can be measured against how far back that information appears.

PersonaMem v2 scales this up to 1,000 personas and more than 300 scenarios, with over 20,000 preferences. More importantly, it shifts toward implicit personalization — information the user reveals indirectly rather than explicitly stating. So there may be no single sentence to retrieve — instead, the system has to infer a preference from patterns across multiple interactions and keep that profile up to date. Frontier models reported in the paper score only 37-48% on this implicit-personalization evaluation.

The conversations are fully synthetic. The authors start from PersonaHub personas and use LLMs to generate profiles, timelines, conversations, and questions. They also filter out questions that can be answered without the conversation history, with v2 using a more extensive filtering pipeline.

PersonaMem adds something the earlier benchmarks mostly miss. A system can remember individual facts correctly and still fail to personalize. For example, a user may repeatedly ask for shorter, less formal writing without ever explicitly saying "this is my preferred style." Later, the system should be able to infer that preference and apply it to a new writing task. V2 is designed to test exactly this kind of implicit personalization.

DMR (Deep Memory Retrieval)

Paper: MemGPT: Towards LLMs as Operating Systems | The MSC dataset: Beyond Goldfish Memory: Long-Term Open-Domain Conversation | Data: ParlAI

We include DMR mostly for historical reasons: it was one of the first widely reported agent-memory evaluations, and its scores still show up in older comparison tables.

DMR is not a standalone dataset. It was introduced in the MemGPT paper in 2023 and built on top of the existing Multi-Session Chat (MSC) dataset, released by Facebook AI in 2021 to study long-term open-domain conversation. MSC contains human-human conversations spanning five sessions. DMR uses those histories and asks questions in a later session that require recalling details from the five earlier ones. But this MSC history is on the order of 60 messages — a few thousand tokens. That was meaningful when context windows were 4k-8k. But today the entire history fits into any model's context window. So DMR is now largely saturated as a memory benchmark: current systems have outgrown the difficulty level it was designed around.

PerLTQA

Paper: PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answering | GitHub: Elvin-Yiming-Du/PerLTQA

PerLTQA is worth mentioning because it makes one specific distinction between memory types explicit in the evaluation. Borrowing from cognitive science, it separates semantic memory — relatively stable knowledge about a person, such as their profile, relationships, and background — from episodic memory — events and experiences, such as what happened, when, and with whom. The dataset is built around this split. It contains 8,593 questions over 30 synthetic characters, each defined by a profile, a social network, and a history of events and dialogues. The paper then evaluates a three-stage framework, scoring each stage separately:

• Memory classification — given a question, identify which kind of memory it requires.

• Memory retrieval — fetch the relevant memory entries.

• Memory synthesis — combine those memories into a correct answer.

That makes PerLTQA interesting because it looks at the memory pipeline in more detail than most benchmarks. Instead of scoring only the final answer, it helps show which part of the process failed — whether the system chose the wrong memory type, retrieved the wrong information, or failed to combine it correctly.

Multi-hop QA benchmarks

This is a different group of benchmarks focused on one ability: answering questions that require connecting facts scattered across multiple sources. For example, the system may need to find one fact, use it to locate another, and then combine the two.

Strictly speaking, these are not memory benchmarks. There is no write step: the corpus is fixed, the questions come with it, and nothing accumulates across sessions. But they are often used as memory proxies because they test the ability to connect related facts that live far apart. We already saw a smaller version of this in LoCoMo, where a multi-hop question may require combining information from several sessions. Some benchmarks in this group push the same idea much further, asking the system to follow chains across millions of documents.

This is also a natural fit for graph-based memory, which is why systems with a knowledge-graph layer often report results on these benchmarks. A knowledge graph stores facts as entities connected by explicit relationships, so a multi-hop question can be answered by following a chain through those relationships. For example, cognee published a paper on optimizing knowledge-graph memory against these benchmarks, along with a deeper look at the results.

HotPotQA

Paper (EMNLP 2018): HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering | GitHub: hotpotqa/hotpot | Site and leaderboard: hotpotqa.github.io

HotPotQA is probably the default benchmark in this group. It contains 113k question-answer pairs over Wikipedia, with each question requiring reasoning across multiple articles. A typical question is "In which city was the band that recorded album X formed?" Answering it requires one article to identify the band and another to find where it was formed.

The benchmark comes in two main settings. In the distractor setting, the system gets 10 paragraphs, 8 of which are decoys, and has to identify the relevant evidence. In fullwiki, it has to retrieve from the full Wikipedia corpus, roughly 5 million articles. The latter is much more relevant if you want to stress the retrieval or memory layer rather than just test reasoning over a small supplied context. Another useful feature is that each question comes with sentence-level supporting facts, so you can evaluate not only whether the final answer is correct, but also whether the system found the right evidence.

The main caveat is age. HotPotQA is based on older Wikipedia data, and Wikipedia-style QA is heavily represented in LLM training data. So strong performance can partly reflect prior exposure to the material rather than retrieval ability alone.

MuSiQue

Paper (TACL 2022): MuSiQue: Multihop Questions via Single-hop Question Composition | GitHub: stonybrooknlp/musique

MuSiQue was created to address a weakness in earlier multi-hop benchmarks such as HotPotQA: many questions that were meant to require several reasoning steps could still be answered from a single clue. To make that harder, MuSiQue constructs 2-4 hop questions from verified single-hop questions, where the answer to one step is needed to reach the next. It also includes unanswerable contrast questions, which help catch systems that guess rather than actually follow the chain.

MuSiQue-Ans contains about 25,000 answerable 2-4-hop questions. MuSiQue-Full adds an unanswerable counterpart for each one, bringing the total to roughly 50,000. But the amount of text the model sees per question is still small compared with large-scale retrieval benchmarks: each question comes with 20 paragraphs containing the supporting evidence and distractors. So MuSiQue is mainly a test of whether the model can follow the required reasoning chain, rather than whether it can find evidence in a very large corpus.

2WikiMultiHopQA

Paper (COLING 2020): Constructing a Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps | GitHub: Alab-NII/2wikimultihop

2WikiMultiHopQA combines Wikipedia text with structured data from Wikidata. The questions are generated from Wikidata structures and predefined reasoning patterns, so they are more templated than HotPotQA questions, but also easier to evaluate step by step. Each question includes a gold reasoning path — a sequence of entities and relations that leads to the answer. That structure also makes the benchmark a natural fit for graph-based memory. A graph system can traverse a similar entity-relation chain, so you can inspect not only whether it reached the correct answer, but also whether it followed the expected reasoning path.

The dataset contains about 193,000 multi-hop questions. In terms of retrieval scale, though, 2WikiMultiHopQA is still small: each question comes with 10 paragraphs containing the supporting evidence and distractors. So the standard setup mainly tests whether the system can follow the right reasoning path within a supplied context, not retrieve evidence from all of Wikipedia.

Long-context benchmarks

This group of benchmarks tests what an LLM can do with information already available in a single long context: retrieve it, connect it, and reason over it. Again, these are not strict memory benchmarks — there is no write step and nothing accumulates over time. But they are still useful in a discussion of memory. In a sense, they test the alternative to a memory layer: how far can you get by simply keeping everything in context? You can think of this as a form of short-term, in-session memory.

The reason this matters is that model performance can still degrade as the input grows, even when the full input technically fits inside the context window — an effect often called context rot. In one set of experiments, focused prompts of around 300 tokens outperformed full 113k-token prompts across the models tested, even on tasks the models could otherwise solve perfectly. So a context window tells you how much text a model can accept, but not how reliably it can use all of it. That is one of the arguments for adding a memory layer even when the full history technically fits in context.

The benchmarks in this group fall into two broad types:

• needle-in-a-haystack tests, which hide a known target in filler and check whether the model can find it,

• task suites, which use more realistic long-input tasks such as QA, summarization, and code, and evaluate performance on those tasks.

Needle in a Haystack (NIAH)

GitHub: gkamradt/LLMTest_NeedleInAHaystack

Needle in a Haystack is the foundational long-context retrieval test. It started as a GitHub project in 2023 rather than a formal benchmark paper, but quickly became a common way to test whether a model could actually use its advertised context window.

Here is how it works: you take a long piece of mostly unrelated text (the haystack), insert one known fact somewhere inside it (the needle), and then ask the model to retrieve that fact. You repeat the test at different context lengths and place the needle at different positions in the input. Plotting those two dimensions gives the familiar heatmap of where retrieval succeeds or fails. The original NIAH setup commonly uses an LLM judge to compare the model's answer with the known needle and assign a score.

But this simple setup has obvious limitations: it only tests whether the model can retrieve one distinctive fact from a large amount of irrelevant text. It does not require combining evidence, tracking changing state, or doing much reasoning once the needle is found. Later benchmarks therefore extend the same idea with multiple needles, aggregation, and multi-hop tracing.

RULER

Paper: RULER: What's the Real Context Size of Your Long-Context Models? (COLM 2024) | GitHub: NVIDIA/RULER

RULER takes the basic NIAH idea and makes it harder. Instead of asking the model to retrieve one obvious needle, it includes 13 tasks across four categories: retrieval with multiple needles and values, multi-hop tracing through variable tracking, aggregation such as frequent-word extraction, and distractor-heavy question answering. Since most of its tasks have fixed answers, the outputs can be checked directly against the gold answer with deterministic scoring.

The main idea is to measure a model's effective context length — how much of its advertised context window it can actually use while maintaining good performance. It provides flexible length configurations, commonly benchmarked up to 32K, 64K, and 128K+ tokens. In the original paper, nearly all tested models scored close to perfectly on basic NIAH, but their RULER performance dropped as context length increased. Although all tested models advertised context windows of at least 32K tokens, only about half maintained satisfactory performance at 32K. Almost all fell below that threshold before reaching their claimed maximum context length.

So RULER gives a more useful answer than "can the model find one fact somewhere in a long prompt?" It tests whether retrieval and simple reasoning still hold up as both the context and the task become harder. One downside is that the data is still synthetic — RULER generates controlled needles and distractor text rather than using naturally occurring long documents or conversations.

BABILong

Paper (NeurIPS 2024): BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack | GitHub: booydar/babilong

BABILong extends the basic haystack setup by requiring the model to find several related facts and reason over them, rather than retrieve a single needle. It builds on bAbI, an older set of synthetic reasoning tasks covering abilities such as counting, deduction, entity tracking, and multi-step fact chaining. BABILong then scatters those relevant facts through much larger amounts of text.

One interesting difference from RULER is the filler itself: BABILong embeds the bAbI facts into book text from PG-19, rather than using entirely synthetic background text. The released benchmark includes context lengths up to 10 million tokens.

That makes it closer to the retrieval problem a memory system may face: find a small set of related facts spread across a very large input, then combine them correctly. But it is still a static long-context benchmark — there is no evolving interaction or write-and-update loop — so it tests long-range retrieval and reasoning rather than memory in the stricter sense.

More haystack variants

There are also other benchmarks that extend the basic haystack setup to test different long-context failure modes:

• NoLiMa removes the easy lexical overlap between the question and the needle, so the model has to find the relevant information through meaning rather than matching words.

• AbsenceBench tests the opposite problem: can the model tell when the requested information is not present at all? This is similar to the memory abstention task, but within a single long context.

• Michelangelo goes beyond retrieving individual facts and asks the model to recover an underlying structure from information spread across a long context.

• Graphwalks tests whether a model can follow multi-hop paths through a graph described in the context.

• OpenAI MRCR tests retrieval when the context contains many similar or near-duplicate pieces of information, making it harder to identify the right one.

Long-context suites

Beyond needle tests, there are broader long-context suites built around more realistic tasks — QA, summarization, code, aggregation, and reasoning over long documents. Unlike pure needle tests, where the goal is mostly "find this fact somewhere in a huge context," these benchmarks ask the model to actually work with that context: answer questions, summarize documents, reason across passages, work with code, or aggregate information. The difference is mainly what kind of long-context ability they stress and at what scale.

• LongBench (and its v2 version, GitHub: THUDM/LongBench) is the broadest general-purpose suite here. V1 is bilingual (English and Chinese) and combines 21 datasets across six categories, including single- and multi-document QA, summarization, few-shot learning, synthetic retrieval, and code. V2 shifts toward harder reasoning tasks and much longer inputs — 503 questions with contexts ranging from 8k to 2M words, including long dialogue histories, code repositories, and structured data.

• InfiniteBench (GitHub: OpenBMB/InfiniteBench) focuses mainly on scale. It was designed for 100k+ contexts and includes 12 tasks covering retrieval, QA, math, and code, with average inputs above 100k tokens.

• L-Eval focuses more on evaluation quality and standardization. It includes 20 subtasks, 508 long documents, and more than 2,000 human-labeled query-response pairs, with inputs ranging from roughly 3k to 200k tokens.

For memory systems, these suites are mostly useful as a reference point. They show how far you can get by keeping everything in the context window and relying on the model to work over it directly. That helps separate problems that genuinely benefit from a memory layer from those that long context can already handle.

Agentic and stateful benchmarks

The last group tests memory inside a working agent. Instead of asking retrieval-style questions, as conversational benchmarks do, these benchmarks measure whether memory helps the agent complete tasks that depend on it.

The agent operates inside a test environment and takes actions — searching, using tools, making bookings, updating records, and so on. Later steps depend on information learned earlier, so the agent has to retain the right state and use it when deciding what to do next. Memory is therefore one component of the overall agent architecture, while the benchmark measures whether the full system completes the task successfully.

This answers a different question from conversational recall benchmarks. A memory system may retrieve the right fact but still fail if the agent does not use it correctly when choosing an action. The trade-off is attribution: if the task fails, you know the overall system failed, but not necessarily whether the cause was memory, reasoning, tool use, or something else.

These environments are also much more expensive to build than QA datasets, so current benchmarks cover relatively few domains. This is still a newer direction in memory evaluation, with two recent benchmarks worth looking at.

MemoryArena

Paper: MemoryArena (ICML 2026) | Github: ZexueHe/MemoryArena | Project: memoryarena.github.io

MemoryArena evaluates memory inside what the authors call a Memory-Agent-Environment loop: the agent takes an action, receives feedback from the environment, stores useful information, and later retrieves it to decide what to do next.

For example, in a travel task, the agent may learn one person's preferences while planning one part of the trip and need to apply them several subtasks later. In progressive search, an earlier search result may become input to a later search. So retrieval alone is not enough — the retrieved information has to influence the next action correctly.

The benchmark contains 766 tasks across four settings: bundled web shopping, group travel planning, progressive web search, and formal reasoning. A task averages 57 agent actions and produces more than 40k tokens of interaction history.

The agent is paired with a persistent memory component, and MemoryArena supports several approaches, including keeping the full history in context, RAG, and external memory systems. During a task, actions and observations are added to memory, then retrieved again when later decisions depend on them. The benchmark measures whether the agent completes the sequence of interdependent subtasks successfully.

This creates a useful contrast with conversational benchmarks. The paper finds that systems performing well on benchmarks such as LoCoMo can still have low task-completion rates on MemoryArena. It also compares different ways of carrying information across the task. External memory does not consistently beat a plain long-context baseline — when the full interaction history is still manageable, keeping the transcript in context can be competitive or better.

But external memory becomes more useful in settings such as progressive web search and formal reasoning, where histories grow longer or the agent needs to preserve intermediate results across multiple steps. The paper also shows the runtime trade-off: external-memory agents have the highest end-to-end latency, RAG systems generally sit in the middle, and long-context agents are fastest.

STATE-Bench

Blog: Introducing STATE-Bench | GitHub: microsoft/STATE-Bench

Microsoft's STATE-Bench takes a similar idea into enterprise workflows. It contains 450 tasks across travel, customer support, and shopping. Each task gives the agent a sandbox environment, domain-specific tools, and a simulated user. The agent has to gather information, follow the relevant procedure, use tools, and update the underlying system correctly.

For memory evaluation, STATE-Bench provides an Agent Learning Track. It gives the system trajectories from earlier tasks, lets it extract and store reusable information from them, and exposes that information to the agent through a retrieval interface. The benchmark then tests whether those learned memories improve performance on separate held-out tasks.

Success is measured mainly from the task outcome. Deterministic assertions check the resulting environment state — for example, whether the correct booking, refund, or account update happened. An LLM judge is used separately for conversational requirements and UX quality. The benchmark also measures reliability by running each task five times. It reports both average task completion and pass^5 — the percentage of tasks that succeed on all five runs. In Microsoft's initial GPT-5.1 baseline without memory, fewer than half of the tasks were completed reliably overall, while travel reached only about 30% pass^5.

Future directions

A number of recent benchmarks also push AI memory evaluation beyond conversational recall.

Mem2ActBench continues the agentic direction. Instead of asking the agent to recall a stored preference or task state, it asks whether the agent can use that information in a tool call — for example, selecting the right tool and filling its parameters from information established earlier. It contains 400 tool-use tasks generated from 2,029 longer interaction sessions.

AgentLongBench focuses on long-running agent interactions. It generates trajectories through repeated agent-environment interaction and then asks questions about what happened across those trajectories. Its evaluation spans contexts from 32K to 4M tokens.

Evo-Memory asks whether memory can improve as the agent handles more tasks. Tasks arrive as a stream, and the system can retrieve previous experience, incorporate new information, and update its memory after each interaction. This shifts the focus from storing facts toward learning from accumulated experience.

MemTrack focuses on tracking changing state across realistic work environments. Its timelines combine events from tools such as Slack, Linear, and Git, including conflicting and cross-referencing information. The challenge is to work out what is currently true after updates have been spread across several systems.

LongMemEval-V2 moves LongMemEval from conversational history to agent experience. It contains 451 manually curated questions over web-agent trajectories, testing static state recall, dynamic state tracking, workflow knowledge, environment gotchas, and premise awareness. Histories can include up to 500 trajectories and 115M tokens. Importantly, it is still a question-answering benchmark: the memory system reads past trajectories and returns compact evidence that is then used to answer the question.

One can see that most of these benchmarks take a different angle from the "classic" memory benchmarks: they focus on agent actions, changing state, long-running interactions, and learning from past experience. These are all promising directions, but there is not yet an established benchmark for each of them.

AI memory benchmark limitations

Benchmarks are useful, but their scores come with important limits. Some are general benchmark problems — it is always hard to reduce a complex system to one standardized test. Others are more specific to memory.

Benchmark scores don't always transfer to real tasks

This is a general benchmark problem: any benchmark fixes a particular task, dataset, and evaluation protocol, so its score only tells you how a system performs under that setup.

As we've seen, many memory benchmarks focus on conversational agents, while real agents may need memory for many other kinds of work. So for your own application, you still need separate validation. MemoryArena also shows this clearly: systems that perform very well on LoCoMo can still have low task-completion rates when memory has to support actions rather than just answers.

Benchmarks have a lifecycle

A benchmark is useful only while it still separates stronger systems from weaker ones. Over time, benchmarks can become saturated: many systems start scoring near the top, so the benchmark no longer distinguishes meaningful differences.

This can happen surprisingly quickly as models, context windows, and baseline methods improve. At the same time, the AI industry has a habit of continuing to report older benchmarks for continuity, even after they become less informative — MMLU is a familiar example in general LLM evaluation. So every benchmark score still needs to be read critically.

We already saw this with DMR. It was meaningful when context windows were only a few thousand tokens, but its entire history now fits easily into modern models. The same pressure applies to other memory benchmarks: LoCoMo's roughly 26k-token conversations are already small by current standards. That is one reason newer benchmarks such as BEAM push to much larger scales.

Benchmarks have quality issues

Several data-quality problems can also make benchmark scores less reliable.

One is contamination. Older public datasets may end up in model training data. HotPotQA is an obvious example: it is based on Wikipedia and has been public for years, so strong performance may partly reflect prior exposure rather than retrieval alone.

Another issue comes from small dataset sizes. Some widely used memory benchmarks are surprisingly small. The public version of LoCoMo, for example, contains only 10 (albeit long) conversations, while LongMemEval has 500 questions. Small datasets can still be useful, but they cover fewer cases and make aggregate scores more sensitive to individual examples.

That leads to a third problem: the quality of the questions themselves. A few ambiguous questions can noticeably move the score on a small benchmark. LongMemEval already went through an official cleaning pass. And, for example, a recent Reddit thread in the AI memory community also scrutinized the quality of examples in popular memory benchmarks.

Finally, synthetic data can introduce artifacts. Many memory benchmarks generate conversations, events, or questions synthetically because it makes it possible to create long, controlled histories at scale. But those examples can still be much cleaner than production behavior. Contradictions may be explicit, preferences may change in neatly defined steps, and relevant facts may be easier to separate from noise than they are in real interactions.

Not all published scores are comparable

Even when two papers or vendors report results on the same benchmark, they may not be running exactly the same evaluation. The score can depend on details such as which subset was used and how the testing was configured. We have already seen several examples of this across implementations: for example, LoCoMo evaluations often exclude adversarial tests. These choices can make nominally identical benchmark results difficult to compare, and vendor-reported results are not always detailed enough to make these differences clear.

There are also built-in sources of variation. Many memory benchmarks score free-form answers with an LLM judge, and different judge models may score the same answer differently. Even repeated runs with the same stochastic judge can vary.

Memory benchmarks have gaps in what they assess

Current benchmarks still capture only part of what matters in a production memory system. Let's discuss some of the gaps that matter.

Internal memory state. Most benchmarks evaluate memory indirectly through retrieval results, final answers, or agent success. That tells you whether memory helped on the tested task, but not much about the quality of the memory itself.

For example, benchmarks usually do not inspect what the system actually stored. Did it lose information that happened not to appear in the test questions? Create duplicate entities? Leave conflicting versions of a fact side by side with no resolution, or accumulate broken links? Two systems could produce the same benchmark answer while maintaining very different internal states. How that state behaves at scale is an important engineering property, but current benchmarks rarely measure it.

Dynamic memory. Most benchmarks also test memory at a fixed point in time: ingest a dataset, run the evaluation, score the result. Production memory keeps changing. New information arrives, old facts are updated, and the system may consolidate, restructure, or learn from previous retrievals. This also creates the risk of catastrophic forgetting — useful older information becoming harder to recover as new information is added. A system may work well immediately after ingestion and behave differently after weeks of updates. Newer work such as Evo-Memory starts exploring this, but there is not yet an established benchmark for it.

Shared memory. Most benchmarks assume one user and one memory store. But real systems may share memory across users or across multiple agents, which raises additional questions: what should be shared, what should remain isolated, and how should conflicting updates from different sources be handled? Current benchmarks barely cover this setting.

Cost and latency. Most benchmarks focus much more on quality than operational cost. But memory sits directly in the execution path: writing, indexing, retrieving, reranking, and sometimes restructuring all take time and compute. That matters more as the store grows, and especially for on-device systems with tighter compute and memory budgets. Some evaluations report cost or latency as secondary metrics, but they are not a central part of most memory benchmarks.

Taken together, these limitations do not make memory benchmarks useless. They make them comparative tools rather than final verdicts. A benchmark can tell you how systems behave under a controlled setup, but it cannot tell you by itself whether a memory system will work for your workload, at your scale, latency budget, and failure tolerance.

How to evaluate AI memory for your use case?

Public benchmarks help compare memory approaches, understand what different memory systems are good at, and identify common failure modes. Different benchmarks tell you different things, so when you're comparing reported results, keep the following in mind:

• Conversational benchmarks are a relevant reference if your agent needs to work with cross-session history. Scale is the most important parameter here: for smaller histories, LongMemEval and LoCoMo numbers could still apply, but both fit inside current context windows. So when run on modern LLMs, they may effectively test the model's ability rather than the memory architecture. If your histories are already large or you expect them to grow beyond 1M tokens, it makes sense to weight results on the benchmarks with large-scale tiers most heavily — like BEAM at 1M and 10M, which start to stress-test memory rather than long-context reasoning.

• Agentic benchmarks are an important reference if you're building action-taking agents. Systems that score well on conversational recall can still have low task-completion rates, which shows that retrieving the right fact and using it correctly are different things. The caveat is that each benchmark tests a particular set of environments — web shopping, travel booking, enterprise support — which likely won't match yours exactly. So they're a good illustration of the problem rather than a measure of your system, though their approach to task design is worth borrowing when you build your own eval.

• Multi-hop QA scores tell you whether a system can link facts that live far apart — relevant if your data is large and interconnected rather than a linear history. Graph-based memory systems tend to report results here, since following entity relationships is what they're good at. The limitation is that these benchmarks have no write step: the corpus is fixed, so a good score says nothing about how well a memory system builds and maintains its store over time.

• Long-context benchmarks tell you what an LLM can do without a memory layer at all. If you are building an AI system where the task-specific inputs are small enough and you don't need a persistent layer, the LLM's performance on these benchmarks can tell you whether the simplest option is good enough. They also show where that stops working, which is often below the advertised context window.

All of these benchmarks provide valuable information about the state of the art, and analysing their results helps narrow down which memory systems are worth trying.

But to see how well a specific memory implementation works in your own system, you need to evaluate it on your own tasks. So the approach is essentially to create your own benchmark for your AI agent. Ultimately you should evaluate the outcome the memory is supposed to improve: task completion, support resolution, code correctness, or whatever matters for your application. Building an application-specific eval usually means a few parts:

• A task set from your workload. Use production interactions where possible, or realistic simulations based on them. Include a slice that specifically stresses memory — for example, tasks that require recalling an earlier preference, handling an update, or connecting information across sessions. You can borrow some of the approaches we've just seen in conversational or agentic benchmarks, but design around your own workflows and data.

• Reliable success criteria. Use deterministic checks where you can: the booking was created, the correct item was selected, the generated code passed its tests. For less structured outputs, use a calibrated LLM judge. The goal is to measure memory through the outcome it is meant to improve.

• Controlled variants. Run the same tasks with memory on and off while keeping the rest of the system fixed. Where useful, add a full-context baseline and alternative memory implementations.

You can run offline checks using such a pre-built test suite, but then also continue to monitor the performance on actual production tasks. DoorDash provides a useful production example. For Ask DoorDash, the team compared sessions backed by its consumer memory system against a baseline with no computed memory over a seven-day production window. For its grocery agent, memory-backed sessions had roughly 24% higher relative checkout conversion, 17% larger baskets, and 7% fewer conversational turns. Its restaurant assistant saw roughly 15% higher relative conversion on open-ended queries, while LLM-judge evaluations found memory-backed sessions about 33% less likely to misunderstand user intent. Notice that these are not abstract memory scores — DoorDash measures whether adding memory improves the actual product outcomes.

Cognee benchmarks

Cognee is an open-source AI memory system built for long-lived agent memory. Our goal is to support complex agentic systems, including setups where multiple agents share context and continue working with it over time. That puts particular focus on some of the gaps discussed above: dynamic memory, shared memory, and memory maintenance and self-improvement as it grows. For public benchmarking, we currently focus on BEAM. Among the conversational benchmarks in this guide, it is the closest match to the scale we care about: coherent histories extending up to 10 million tokens, where keeping the entire history in context is no longer a practical baseline. Our published evaluation reports a state-of-the-art result of 0.79 at BEAM 100K and 0.67 at BEAM 10M, compared with previously reported results of 0.73 and 0.64 respectively. The evaluation uses cognee's existing open-source components rather than a separate benchmark-specific memory architecture.

Accuracy is only one part of evaluating a production memory system, so we also look at the cost and latency of building and querying memory. That matters particularly for our Rust implementation, which runs the memory pipeline locally and is designed for resource-constrained and on-device deployments.

Try cognee

Cognee is open source — you can run it locally or get started with cognee Cloud for a managed setup.