Best Evaluation Datasets and Benchmarks for Agent Memory in 2026

Picking a benchmark for agent memory in 2026 means choosing between datasets that each measure a different slice of the problem. This guide walks through the six evaluation datasets that currently drive published results and buyer decisions: BEAM for extreme-scale conversational memory up to 10M tokens, LoCoMo for long conversational recall, LongMemEval for tracking changing user information, MemoryAgentBench and STATE-Bench for enterprise agent behavior, HotPotQA for multi-hop retrieval, and the newer DolphinBench for memory-guided tool use. No single dataset produces a universal score, so the guide also covers how each one is scoped, where synthetic data can mislead, and how to reproduce results on a private corpus.

Each dataset section describes what the benchmark actually measures, the task format, where the published numbers come from, and the known failure modes when results are generalized beyond the dataset. The guide also includes a comparison table, a review of published results across memory systems, and a walkthrough of the open evaluation harnesses from Mem0, Zep, Letta, and Cognee. Cognee is covered because it publishes self-reported state-of-the-art BEAM results and ships an open evaluation framework in the cognee repo that spans BEAM plus multi-hop QA on HotPotQA, TwoWikiMultiHop, and MuSiQue, with several metrics per run across graph and memory components.

A memory layer that scores well on conversational recall will not necessarily hold up on enterprise document QA, and a system tuned for multi-hop retrieval over Wikipedia can collapse on private corpora with messy entity resolution. The practical answer for anyone evaluating a memory platform is to pair one or two public benchmarks with a reproducible run on an internal dataset before committing. The sections that follow give the dataset descriptions, the published numbers, and the harness instructions needed to do that.

What Is an Agent Memory Evaluation Dataset?

An agent memory evaluation dataset is a structured corpus paired with a question or task set used to measure how a memory system ingests, retrieves, and reasons over prior interactions or documents. The corpus supplies the content a memory layer must store (chat transcripts, enterprise documents, tool traces), and the task set defines the questions, judgments, or state checks that probe whether retrieval and reasoning produced the right answer at the right time.

Definition and scope

A dataset is the raw material; a harness is the code that pushes that material through a memory system end to end. A benchmark run typically follows an ingestion → retrieval → generation → judge scoring pipeline: documents or dialogues are loaded into the memory layer, the agent is queried, an answer is produced, and a rubric or LLM judge scores correctness. Separating the two makes it clearer whether a weak score traces back to retrieval, generation, or scoring choices rather than the dataset itself.

The active 2026 landscape covers several categories. Conversational recall is measured by LoCoMo and LongMemEval, with LongMemEval specifically targeting how a system handles changing user information over time. BEAM pushes extreme-scale conversational memory up to 10M tokens. Multi-hop QA uses HotPotQA, TwoWikiMultiHop, and MuSiQue to test reasoning across linked documents. Stateful enterprise tasks are covered by STATE-Bench, which focuses on enterprise agent experience, and DolphinBench evaluates memory-guided tool use. No single benchmark produces a universal score, which is why most serious evaluations run several in parallel.

Common terms: recall, multi-hop, temporal reasoning, state agreement

Recall measures whether the memory system can retrieve a specific previously seen fact when asked. Multi-hop reasoning requires combining two or more retrieved pieces to answer a question that no single document contains. Temporal reasoning tests whether a system respects when information was true, including updates and contradictions over long dialogues. State agreement, central to enterprise benchmarks like STATE-Bench, checks whether the agent's internal view of entities, tickets, or workflow status matches the ground-truth state after a sequence of actions.

Cognee ships an open evaluation framework in the cognee repo that covers BEAM alongside multi-hop QA on HotPotQA, TwoWikiMultiHop, and MuSiQue, reporting several metrics per run across both graph and memory configurations. The pipeline handles ingestion, retrieval, answer generation, and rubric scoring, and can run locally for inspection or on Modal for larger sweeps. For a deeper walkthrough of the methodology, see the cognee guide on how to evaluate AI memory tools.

Why Agent Memory Benchmarks Matter in 2026

Benchmark selection has become a procurement question. When a memory layer is being evaluated for a production agent, the dataset behind the vendor's reported accuracy determines whether that number predicts anything about enterprise workloads. The 2026 landscape now includes LoCoMo for conversational recall, LongMemEval for changing user information, cognee's BEAM for extreme-scale memory up to 10M tokens, DolphinBench for memory-guided tool use, and STATE-Bench for enterprise agent experience. No single score covers all of these dimensions, so a buyer who accepts one headline number without checking which dataset produced it is accepting an incomplete picture.

Shift from recall-only to stateful enterprise tasks

The older generation of memory benchmarks measured whether an agent could retrieve a fact from an earlier turn. Enterprise work rarely reduces to that. Microsoft's STATE-Bench is an open-source, memory-agnostic benchmark covering customer support, travel, and shopping, with 450 tasks spanning policy compliance, information synthesis, and multi-step reasoning procedures. StateMemBench, a related suite of 234 multi-session scenarios, reports that even the strongest long-context baseline it tested (GPT-5.4-Nano) reaches only 0.277 overall state-agreement accuracy. That result undermines the assumption that a sufficiently large context window can substitute for a memory layer, because the adversarial traps in these datasets defeat full-transcript prompting.

Public scrutiny of reported scores

The LoCoMo dispute from 2025 is the clearest reason buyers now ask for reproducible evaluation harnesses. Zep's original LoCoMo result of 84% was publicly challenged in May 2025 by Mem0's CTO as 58.44% after recounting adversarial questions; Zep later acknowledged a calculation error and revised the figure to 75.14% (±0.17). Cognee's own breakdown of the incident is in its guide on how to evaluate AI memory tools. The takeaway is practical: ask for the harness, the seed, and the scoring script before trusting a leaderboard number, and run the benchmark locally against the data a production agent will actually see.

The Six Core Evaluation Datasets for Agent Memory

Six datasets dominate serious conversations about agent memory evaluation in 2026. Each one probes a different failure mode, long-horizon recall, multi-hop composition, state mutation, selective forgetting, and each one scores results in its own way. The subsections below use the same field order (what it measures, scale, question or task categories, scoring, publisher) so the datasets can be read side by side without re-learning the format each time.

BEAM

What it measures: Ten distinct memory abilities including fact retrieval, event ordering, abstention when information is absent, and contradiction resolution across sessions.

Scale: Context sizes ranging from 100K up to 10M tokens, built on synthetic conversations that preserve real conversational structure, arcs, sessions, topics, and turns, rather than randomly stitched text.

Question or task categories: The ten memory abilities above, each probed with questions designed to isolate that capability.

Scoring: An LLM judge evaluates answers against a public rubric, which keeps grading reproducible across runs.

Publisher: cognee. Self-reported BEAM numbers from cognee show 79% at the 100K-token setting (previously reported SOTA 73.4%) and 67% at 10M tokens (previously 64.1%), with token usage held flat as stored data grows. Full methodology is in the cognee BEAM deep dive and on the research and evaluation results page.

LoCoMo

What it measures: Long-conversation memory across multi-session dialogue, with explicit stress tests for temporal reasoning and adversarial recall.

Scale: Ten long, high-quality conversations averaging roughly 600 turns and 26k tokens each.

Question or task categories: 1,986 QA pairs across single-hop, multi-hop, temporal reasoning, open-domain, and adversarial categories. 446 of those pairs are adversarial, and recent work typically omits them to keep comparisons consistent.

Scoring: Per-category accuracy on QA pairs, with results usually reported with and without the adversarial subset.

Publisher: Academic release, widely adopted by memory-system vendors. cognee's notes on LoCoMo are summarized in the AI memory benchmarks guide.

LongMemEval

What it measures: Long-horizon dialogue memory with explicit coverage of knowledge updates and preference tracking.

Scale: 500 curated questions over dialogue histories that can scale past one million tokens.

Question or task categories: Six, temporal reasoning, multi-session, knowledge update, single-session user, single-session assistant, and single-session preference.

Scoring: Per-category accuracy, which exposes whether a memory layer handles updates and preferences as competently as raw retrieval.

Publisher: Academic release (LongMemEval paper, arXiv).

MemoryAgentBench

What it measures: Four core memory competencies, accurate retrieval, test-time learning, long-range understanding, and selective forgetting (also framed as conflict resolution).

Scale: Built by transforming existing long-context datasets into a multi-turn interaction format, with two additional datasets introduced to target weak spots: EventQA for accurate retrieval and FactConsolidation for selective forgetting.

Question or task categories: Grouped by the four competencies above, each with multi-turn exchanges that force a memory system to decide what to keep, what to update, and what to drop.

Scoring: Per-competency metrics, reported alongside the standard long-context baselines so regressions are visible.

Publisher: ICLR 2026 publication; the paper is available on arXiv and OpenReview.

STATE-Bench

What it measures: Stateful agent behavior in enterprise workflows, where the agent must change system state in a database, refund records, booking status, account updates, rather than only answering recall questions.

Scale: 450 tasks across three domains: customer support, travel, and shopping.

Question or task categories: Policy compliance, information synthesis, and multi-step reasoning procedures. An Agent Learning Track lets a memory system extract and store reusable information from earlier task trajectories, then checks whether those learned memories improve performance on held-out tasks.

Scoring: Task success on the main track, with reliability reported as pass^5 (a task counts only if it succeeds on all five runs), alongside a UX score and cost per task.

Publisher: Microsoft, open-sourced and memory-agnostic. Full details are in the STATE-Bench announcement.

HotPotQA

What it measures: Compositional reasoning across multiple supporting documents, with an explicit focus on multi-hop question answering.

Scale: Large-scale QA constructed over Wikipedia, with sentence-level supporting-fact annotations attached to each question.

Question or task categories: Multi-hop questions that require combining evidence from at least two Wikipedia articles, plus comparison questions that test attribute-level reasoning.

Scoring: Exact-match and F1 on answers, with a separate supporting-fact score that rewards citing the right sentences, a useful signal for memory systems that claim to track provenance.

Publisher: Academic release (Yang et al., EMNLP 2018).

Benchmark Comparison Table

The six datasets below are the ones most frequently cited in agent memory research and vendor evaluations. The matrix below compares what each one actually measures, how large it is, and how results are scored.

Two newer datasets extend this picture in directions the six core benchmarks do not cover. DolphinBench, from Mem0, evaluates memory through real-world simulated actions across three personas (startup CEO, infrastructure engineer, product manager) and reports cost and latency alongside accuracy, verifying every test with and without the relevant history. StateMemBench adds 234 multi-session scenarios focused on state-agreement, where even the strongest long-context baseline reaches only 0.277 overall accuracy, demonstrating adversarial traps that defeat holding the full transcript in context. Cognee's BEAM results are self-reported and should be reproduced locally using the BEAM deep dive before being used in procurement decisions.

Common Challenges When Benchmarking Agent Memory

Memory benchmarks look authoritative on a leaderboard, but the published numbers hide several recurring problems that affect how a result should be read. Four issues come up often enough that any buyer evaluating an agent memory system should check for them before trusting a headline score.

Score-reporting disputes

Published accuracy numbers can move substantially after independent recounting. In May 2025, Zep's original LoCoMo result of 84% was publicly challenged by Mem0's CTO, who recalculated it as 58.44% after re-examining how adversarial questions were counted. Zep later acknowledged a calculation error and revised the figure to 75.14% (±0.17). A nine-point swing on the same dataset, same system, same questions is a reminder that leaderboard numbers should be treated as self-reported until the scoring code is public and reproducible.

Synthetic-only datasets

Synthetic construction lets a benchmark scale cleanly, but it does not guarantee the data resembles enterprise content. BEAM is built with real conversation structure, arcs, sessions, topics, and turns, then expanded synthetically from 100K to 10M tokens, probing ten memory abilities including fact retrieval, event ordering, abstention, and contradiction resolution, each scored by an LLM judge with a public rubric. That design is strong for stress-testing scale and specific capabilities, but a system that performs well on BEAM conversations may behave differently on dense PDFs, support tickets, or code repositories. Cognee's BEAM numbers are self-reported and published at the cognee research and evaluation results page.

Adversarial questions and judge variance

LoCoMo questions are categorized into single-hop, multi-hop, temporal reasoning, open-domain, and adversarial types. Recent work commonly omits the 446 adversarial items, which inflates headline accuracy because adversarial questions are the ones designed to catch hallucinated or over-confident answers. Two systems reporting LoCoMo scores can therefore be measuring different things entirely. Judge variance adds a second layer: when an LLM scores free-form answers against a rubric, prompt wording and model version both move the result, so published numbers should always specify the judge model and the question subset.

Operational metrics missing from recall-only tests

Accuracy alone does not describe whether a memory layer is deployable. DolphinBench, published by Mem0, evaluates agent memory through real-world simulated actions across three personas, startup CEO, infrastructure engineer, and product manager, and reports cost and latency alongside accuracy, verifying every test with and without the relevant history. Zep's research has taken a similar direction. Recall-only benchmarks leave those operational dimensions out, which means a system that answers correctly but costs ten times more per query or takes five seconds longer looks identical to a cheaper, faster one.

Cognee's in-repo benchmark pipeline handles ingestion, retrieval, answer generation, and rubric scoring as separate stages, and can run locally for inspection or on Modal for larger sweeps. Exposing each stage lets a reviewer see where a score actually comes from, whether a win was driven by better extraction, better retrieval, or a more lenient judge, rather than accepting a single aggregate number.

What to Look for in an Agent Memory Benchmark

Choosing a benchmark is a procurement decision, not a vanity exercise. The dataset you pick determines which failure modes get measured, which get hidden, and whether the numbers you report will survive contact with a production corpus. Six criteria separate benchmarks worth running from benchmarks worth skipping in 2026.

Selection criteria that matter in 2026

1. Realism of the corpus. Fully synthetic Q/A pairs reward retrieval tricks that collapse on messy enterprise text. Prefer datasets derived from genuine conversation structure or real task trajectories. BEAM is synthetic but constructed around real conversational arcs, sessions, and topics before turns are generated, which keeps discourse patterns intact. DolphinBench pushes further by simulating real-world actions across three operator personas (startup CEO, infrastructure engineer, product manager) and verifying each test both with and without the relevant history.

2. Scale. Memory systems behave differently at 10K, 1M, and 10M tokens. A benchmark that only probes short contexts will not predict production behavior. BEAM scales from 100K to 10M tokens, which exposes degradation curves that shorter datasets hide.

3. Task type. Decide what you actually need the memory layer to do. Fact recall, event ordering, abstention, contradiction resolution, multi-hop reasoning, and stateful action are different capabilities. BEAM probes ten memory abilities under one rubric. STATE-Bench's Agent Learning Track measures whether a system can extract reusable information from earlier task trajectories and apply it to held-out tasks, which is the closest public analog to continual learning.

4. Reproducibility. If the evaluation code and rubric are not public, the score is marketing. Favor benchmarks with open harnesses and documented judge prompts. BEAM publishes its LLM-judge rubric. STATE-Bench reports pass^5 across five runs so variance is visible rather than averaged away.

5. Scoring transparency. A single accuracy number compresses too much. Benchmarks that publish the judge rubric, the metric definitions, and per-ability breakdowns let you audit whether a win on one ability is being masked by a loss on another. BEAM and STATE-Bench both qualify; many vendor-authored leaderboards do not.

6. Operational metrics. Accuracy without cost and latency is unusable for capacity planning. DolphinBench reports cost and latency alongside accuracy, and pass^k reliability (as used in STATE-Bench) tells you how often a system actually produces the correct answer rather than how often it can in principle.

Cognee's evaluation framework reports Human-like Correctness, DeepEval Correctness, F1, and Exact Match across head-to-head comparisons with Mem0, Zep/Graphiti, and Honcho. The four-metric view is deliberate: F1 and Exact Match catch retrieval precision, DeepEval Correctness catches semantic adequacy, and Human-like Correctness catches the cases where a technically correct answer still reads wrong to an operator. Aggregate results are maintained at the cognee research and evaluation results page and should be read as self-reported until independently reproduced on your own corpus.

Published Results Across Memory Systems

The published numbers below come from cognee's own evaluation runs and are self-reported. Full methodology, scoring scripts, and comparison tables are maintained on the cognee research and evaluation page, which tracks Human-like Correctness, DeepEval Correctness, F1, and Exact Match against Mem0, Zep/Graphiti, and Honcho.

BEAM leaderboard snapshot

BEAM is the most demanding conversational memory benchmark in the current landscape, with dialogues scaled up to 10M tokens. Cognee self-reports 79% accuracy at the 100K-token setting, against a previously reported SOTA of 73.4%, and 67% at 10M tokens, against 64.1%. Token usage stays effectively flat as stored data grows, which is the property most synthetic benchmarks fail to measure. One important caveat: the BEAM run documented in the cognee BEAM deep dive used default open-source components rather than a benchmark-specific memory configuration, so the numbers reflect out-of-the-box behavior.

Cognee vs. Mem0 at 10M tokens

At the 10M-token BEAM setting, cognee reports 67% accuracy while Mem0 reports 48.6% using top-200 retrieval, per the cognee vs. Mem0 comparison. The disparity widens as conversation length grows, which is consistent with the broader observation that retrieval-only architectures degrade on long-horizon dialogue while graph-backed memory holds accuracy more consistently. Readers reproducing this comparison should pin both library versions, record the retrieval configuration, and run the eval against identical BEAM splits to make the numbers directly comparable.

Chain-of-Thought graph completion gains

On the Chain-of-Thought graph completion evaluation, cognee reports the following lifts: Human-like Correctness from 0.738 to 0.925, DeepEval Correctness from 0.569 to 0.846, DeepEval F1 from 0.203 to 0.841, and DeepEval EM from 0.04 to 0.687. The F1 and Exact Match deltas are the most informative because they reflect structural recovery of entities and relations rather than paraphrase scoring. All four metrics are self-reported and documented on the research and evaluation page alongside the comparison runs against Mem0, Zep/Graphiti, and Honcho.

Open Evaluation Harnesses from Mem0, Zep, and Letta

Several memory vendors now publish their own evaluation code so results can be re-run outside a marketing deck. Three stacks are worth knowing before picking an in-house harness: Mem0's memory-benchmarks repository, Zep's operational reporting style, and Letta's agent-centric evals.

Mem0's memory-benchmarks harness

Mem0 maintains an open-source memory-benchmarks harness that covers LoCoMo, LongMemEval, and BEAM in a single pipeline. The harness is split into four stages, ingestion, search, generation, and judge scoring, which makes it practical to swap in a different retriever or a different judge model without rewriting the dataset loaders. For anyone replicating published numbers, having each stage separated also means ingestion cost and search latency can be measured independently from end-to-end answer quality.

Zep's latency and context-token reporting

Zep's public reporting pattern separates operational cost from accuracy. Retrieval latency is reported at p50 and p95, and median context tokens per query are published alongside accuracy scores rather than folded into a single headline number. The practical effect is that a retrieval system which hits a high LongMemEval score by stuffing large contexts is visible as such, because the context-token line goes up while latency widens at the tail. Specific latency values should be read from Zep's current published guidance, since they move with model and infrastructure changes.

Letta Evals and Context-Bench

Letta ships Letta Evals together with the Context-Bench suite, released as Context-Bench V1 (open) and V2 (private). The evaluation emphasis is on the agent's own memory behavior, how the agent decides to write, update, and recall memory, rather than treating memory as a black-box retriever behind a QA task. V1 being open allows reproduction of the memory-operation scenarios locally, while V2 is held back as a private evaluation set to limit overfitting on published tasks.

Best Practices for Benchmarking Agent Memory on Your Own Data

Benchmarking agent memory is only as credible as the methodology behind it. The practices below are condition-led: apply them when designing an evaluation that will inform a real procurement or architecture decision.

Treat vendor scores as directional

Published leaderboard numbers can move substantially under recount. Zep's original LoCoMo result of 84% was publicly challenged in May 2025 by Mem0's CTO as 58.44% after recounting adversarial questions, and Zep later acknowledged a calculation error and revised the figure to 75.14% (±0.17). When a single recount can move a headline score by more than twenty points, vendor claims should be used to narrow a shortlist rather than to pick a winner.

Separate retrieval from generation

If retrieval and generation are measured together, a weak memory layer can be masked by a strong LLM, and a strong memory layer can be penalised by a weak one. Mem0's open-source memory-benchmarks harness covers LoCoMo, LongMemEval, and BEAM across distinct ingestion, search, generation, and judge scoring stages, which makes it possible to isolate where errors are introduced. Any serious internal evaluation should split these stages the same way.

Report cost and latency alongside accuracy

Accuracy in isolation hides the economics of a memory system. DolphinBench evaluates agent memory through real-world simulated actions across three personas, startup CEO, infrastructure engineer, and product manager, and reports cost and latency alongside accuracy, verifying every test with and without the relevant history. That ablation also confirms whether a correct answer actually depended on memory or was recovered from the base model, which is the measurement most internal benchmarks skip.

Reproduce before you commit

Self-reported scores are a starting point, not a contract. Cognee's reproduction instructions direct developers to clone the topoteretes/cognee repo and run the evaluation framework under cognee/eval_framework/ (locally, or through its Modal runner for larger sweeps) against any memory system, so the same harness used for cognee's published numbers can be pointed at competing memory layers on identical hardware. Running that pipeline before signing anything turns a vendor claim into a result produced inside your own environment.

Use multiple datasets, not one

A single benchmark rewards whatever it happens to measure. Current surveys of the field describe the active landscape as LoCoMo for conversational recall, LongMemEval for changing user information, BEAM for extreme-scale memory up to 10M tokens, DolphinBench for memory-guided tool use, and STATE-Bench for enterprise agent experience, and none of them gives a universal score on its own. A credible internal evaluation samples from at least two of these axes.

Audit the judge rubric

LLM-as-judge scoring is only trustworthy when the rubric is inspectable. BEAM is built with real conversation structure, arcs, sessions, topics, then turns, scales from 100K to 10M tokens, and probes ten memory abilities including fact retrieval, event ordering, abstention, and contradiction resolution, each scored by an LLM judge with a public rubric. Before adopting any judge-scored benchmark internally, read the rubric for each ability and confirm it rewards the behaviour your agents actually need.

Advantages of a Reproducible Benchmark Workflow

Running a benchmark on your own corpus produces evidence a vendor leaderboard cannot: accuracy numbers tied to the documents, queries, and judges you actually care about. A reproducible workflow also lets you compare systems on the same hardware, prompts, and scoring rubric, which is the only honest way to decide between memory layers before procurement.

Measurable outcomes buyers can verify

• Verifiability on your own corpus. When the ingestion, retrieval, answer generation, and rubric scoring stages all run locally, every number is auditable against the source documents. Cognee's benchmark pipeline covers all four stages and can run locally for inspection or on Modal for larger sweeps, so results can be reproduced by anyone with access to the corpus.
• Token usage that stays flat as the store grows. Cognee's self-reported BEAM runs observed token usage holding steady as stored data scaled from 100K to 10M tokens, which is the kind of operational property that only shows up when a benchmark is executed end-to-end rather than read from a leaderboard. See the full numbers in Cognee's research and evaluation results.
• Per-stage attribution. Separating ingestion, retrieval, generation, and judge scores isolates where a regression actually originates. A drop in answer quality caused by a retriever change looks very different from one caused by a prompt change, and a staged harness makes that difference legible.
• Downstream task tie-in. DolphinBench, released by Mem0, evaluates memory through real-world simulated actions across three personas (startup CEO, infrastructure engineer, product manager) and reports cost and latency alongside accuracy, verifying every test with and without the relevant history. Tying memory quality to downstream task metrics in this way connects benchmark deltas to the business outcomes a buyer is actually paying for.
• Lower switching cost after evaluation. A harness that ingests your corpus, runs retrieval, and scores answers against a rubric is the same harness used to validate the next system under consideration. The evaluation artifact outlives any individual vendor decision.

How Cognee Supports Benchmarking Agent Memory

Cognee is built so that any claim about retrieval quality can be re-run by the reader. The repository ships with the harness, the datasets wiring, and the execution paths needed to compare memory systems on identical inputs, and the hosted platform exposes the same ingestion and retrieval primitives used in those evaluations.

In-repo evaluation framework

The cognee repository includes an evaluation framework that covers BEAM alongside multi-hop QA on HotPotQA, TwoWikiMultiHop, and MuSiQue, with several metrics reported per run across both graph and memory components. Reproduction follows a documented path: clone the topoteretes/cognee repo and run cognee/eval_framework/ against the memory system under test, locally or through its Modal runner. That means a reviewer can score cognee against alternative memory layers on the same questions and the same scoring code, instead of comparing numbers lifted from separate papers with incompatible setups.

Local vs. Modal execution

Defaults are chosen so a developer can start without provisioning anything. Running embedded with SQLite, LanceDB, and the Ladybug graph engine, cognee executes locally with minimal resource expenditure, which is enough for smoke tests and small sweeps on HotPotQA-sized subsets. When larger passes are needed, such as full BEAM runs or multi-dataset comparisons across several memory systems, the same harness is pointed at Modal. The configuration swap is isolated to execution; the dataset loaders, metrics, and retrieval modes stay identical, so results from a laptop and from a Modal sweep are comparable.

Hosted platform pricing

For benchmarking against the hosted platform, the service covers ingestion through the ECL pipeline, graph and vector storage, and the full set of search types, including TEMPORAL, GRAPH_COMPLETION_COT, and NATURAL_LANGUAGE (natural language to Cypher), so evaluations can measure the same retrieval behaviors exposed to production agents. Cognee Cloud is usage-based at $1.00 per 1M tokens processed, plus $5 per additional workspace. The free tier includes 1M tokens, one workspace, unlimited users, and unlimited API calls, which is typically sufficient to reproduce a BEAM subset or a HotPotQA slice before deciding whether to commit.

The project has 80+ contributors and runs in production at more than 70 companies, which gives the harness a steady stream of real workloads informing what the benchmarks need to cover next.

Where Agent Memory Benchmarks Are Heading

Agent memory evaluation is moving past static recall. The newer datasets being published in 2026 reward systems that can act, learn across sessions, and resolve conflicting information, and they report metrics that account for run-to-run variance rather than a single lucky pass.

Stateful enterprise tasks

Microsoft's STATE-Bench scores agents on tasks that change system state in a database, issuing refunds, updating booking status, modifying account records, rather than only answering questions from a transcript. That reframes memory as a prerequisite for correct action: a system can recall a prior customer interaction perfectly and still fail if it writes the wrong record. DolphinBench, published by Mem0, pushes in a parallel direction with real-world simulated actions across three personas (a startup CEO, an infrastructure engineer, and a product manager), and reports cost and latency alongside accuracy. Each DolphinBench test is also verified with and without the relevant history, so a score reflects what memory actually contributed.

Memory-guided tool use

STATE-Bench's Agent Learning Track makes the learning loop explicit. A memory system extracts and stores reusable information from earlier task trajectories, then is tested on held-out tasks to see whether those learned memories improve performance. The question stops being "can you retrieve this fact" and becomes "can you build a working procedural memory from experience and apply it to a task you haven't seen." Cognee's reference material on these benchmarks is collected in the AI memory benchmarks guide on this site.

Reliability metrics (pass^k)

STATE-Bench reports reliability as pass^5 across five runs, a task counts as solved only if the agent succeeds every time. That penalizes memory systems that answer correctly on average but behave non-deterministically, which is exactly the failure mode enterprise deployments cannot absorb. Expect more benchmarks to adopt pass^k-style reporting as the field acknowledges that mean accuracy hides the variance that matters most in production.

Selective forgetting and conflict resolution

MemoryAgentBench, accepted at ICLR 2026, evaluates four core competencies: accurate retrieval, test-time learning, long-range understanding, and selective forgetting (also framed as conflict resolution). It transforms existing long-context datasets into a multi-turn format and adds two new datasets, EventQA and FactConsolidation, aimed at accurate retrieval and selective forgetting. The inclusion of FactConsolidation signals a wider acknowledgment that memory systems must decide which version of a fact to keep when a user says one thing on Monday and another on Thursday. Cognee's position is that conflict resolution and temporal reasoning belong in the core evaluation stack, and the next generation of benchmarks is moving in that direction.

Key Takeaways and How to Get Started

No single evaluation dataset covers every memory requirement, so match the benchmark to the workload you actually run in production.

Pick a dataset that matches your task

• BEAM, stress-test memory at extreme scale, up to 10M tokens.
• LoCoMo, long-horizon conversational recall across sessions.
• LongMemEval, tracks changing user information over time.
• MemoryAgentBench, measures forgetting and test-time learning.
• STATE-Bench, enterprise agent experience and state changes.
• HotPotQA, multi-hop reasoning across linked documents.
• DolphinBench, memory-guided tool use.

No single benchmark produces a universal score, so combine two or three that reflect your workload.

Reproduce on your own data

Clone the topoteretes/cognee repository and run cognee/eval_framework/ against the memory system under test, locally or through its Modal runner. The harness lets a candidate memory layer be scored on the same ingestion and query path used for Cognee's published numbers, which keeps the comparison honest when the corpus is swapped for internal documents or chat logs.

Try Cognee

Cognee Cloud is usage-based at $1.00 per 1M tokens processed, plus $5 per additional workspace. The free tier includes 1M tokens, one workspace, unlimited users, and unlimited API calls, which is enough to ingest a benchmark corpus and run an evaluation end to end before committing to paid usage.

Next steps: sign up for the Cognee Cloud free tier to ingest your corpus, or clone topoteretes/cognee and run the eval framework against your current memory stack.

Frequently Asked Questions

What is the best evaluation dataset for benchmarking agent memory in 2026?+

No single dataset gives a universal score for agent memory. The 2026 landscape covers LoCoMo for conversational recall, LongMemEval for changing user information, BEAM for extreme-scale memory up to 10M tokens, DolphinBench for memory-guided tool use, and STATE-Bench for enterprise agent experience. Each measures a different ability, so the right dataset depends on whether the workload is dialogue, document-heavy retrieval, or stateful task execution. Cognee publishes results across BEAM, HotPotQA, TwoWikiMultiHop, and MuSiQue so the engine can be compared against the benchmark closest to a given production workload.

Which memory layer has the best published accuracy benchmarks?+

On BEAM, Cognee self-reports 79% at the 100K-token setting against a previously reported SOTA of 73.4%, and 67% at the 10M-token setting against 64.1%, with token usage staying flat as stored data grows. These numbers are published by Cognee at the cognee research and evaluation results page and should be treated as self-reported until independently reproduced. BEAM is the longest-horizon public benchmark currently reporting results across major memory layers, which is why Cognee's deep dive on it is the clearest head-to-head accuracy reference available right now.

Which memory layer is most accurate on real enterprise documents vs. synthetic data?+

Synthetic benchmarks like BEAM construct conversations from arcs, sessions, topics, and turns, which is useful for scale but does not reflect enterprise document workflows. STATE-Bench, released by Microsoft, is closer to enterprise reality: 450 tasks across customer support, travel, and shopping where the agent must change system state in a database (refunds, bookings, account updates), not only answer recall questions. For enterprise document benchmarking, Cognee recommends running the engine against STATE-Bench and against internal corpora rather than relying on synthetic scores alone.

Can I benchmark a memory platform on my own data before committing?+

Yes. Cognee defaults to SQLite, LanceDB, and the Ladybug graph engine, which run embedded with minimal resource expenditure, so the full pipeline can be tested locally without provisioning infrastructure. The free tier of Cognee Cloud adds 1M tokens, one workspace, unlimited users, and unlimited API calls, which is enough to run a complete evaluation set end-to-end. Paid usage is $1.00 per 1M tokens processed, plus $5 per additional workspace. Teams that need to validate accuracy on private data can therefore benchmark before any paid commitment.

How does BEAM differ from LoCoMo and LongMemEval?+

BEAM is fully synthetic, scales from 100K to 10M tokens, and probes ten memory abilities including fact retrieval, event ordering, abstention, and contradiction resolution, each scored by an LLM judge with a public rubric. LoCoMo's public release has 10 long conversations averaging around 600 turns and 26k tokens each, with 1,986 QA pairs including 446 adversarial ones. LongMemEval has 500 curated questions across six categories and can scale beyond one million tokens. Cognee reports on all three so results can be compared at matched token scales.

How do I reproduce Cognee's BEAM results locally?+

Cognee's reproduction instructions direct developers to clone the topoteretes/cognee repository and run the evaluation framework under cognee/eval_framework/, locally or through its Modal runner. The framework in that repo covers BEAM plus multi-hop QA on HotPotQA, TwoWikiMultiHop, and MuSiQue, with several metrics per run across the graph and memory layers. Because the Cognee defaults run embedded, the full BEAM harness can execute on a single developer machine for the smaller token tiers. Self-reported numbers at the cognee research and evaluation results page can then be checked against a local run on the same rubric.

What does STATE-Bench measure that LoCoMo does not?+

STATE-Bench measures stateful enterprise task execution. Its 450 tasks span policy compliance, information synthesis, and multi-step reasoning across customer support, travel, and shopping, and the agent is scored on whether it correctly mutates database state such as refund records, booking status, and account updates. LoCoMo, by contrast, scores conversational recall over long dialogues with QA pairs, including adversarial ones. An agent can score well on LoCoMo recall and still fail STATE-Bench because mutating enterprise state requires grounded retrieval plus correct action selection. Cognee is evaluated against both to cover recall and stateful execution.

How much does Cognee cost to run on my own evaluation set?+

Cognee Cloud is usage-based at $1.00 per 1M tokens processed, plus $5 per additional workspace. The free tier includes 1M tokens, one workspace, unlimited users, and unlimited API calls, which covers a first pass on LoCoMo (around 260k tokens of raw dialogue across its 10 conversations) and small BEAM tiers. Self-hosted runs using the SQLite, LanceDB, and Ladybug defaults have no Cognee licensing cost; the only spend is the underlying LLM and embedding API calls invoked during ingestion and evaluation, which are the same across any memory layer being compared.