How to Benchmark AI Agent Memory on Your Own Data: Datasets and Platforms (2026)

Cognee is our first choice for developer-controlled memory evaluation, with an Apache-2.0 core and a documented local trial path. This guide covers public memory datasets, held-out evaluation on your documents, and platform trial options, using a shared protocol for answer accuracy, retrieval precision, latency, and cost per query. The recommendation concerns evaluation suitability; the most accurate platform for your workload must be determined from your own held-out results.

What Does an AI Agent Memory Benchmark Measure?

An AI agent memory benchmark tests whether an agent can retain, retrieve, and apply information across a defined workload. The workload matters: recalling a fact from a conversation, learning from incrementally supplied text, and reusing experience while taking actions test different capabilities.

Conversational workloads such as LoCoMo and LongMemEval test recall and reasoning across sessions. Incremental-text workloads such as MemoryAgentBench test how memory handles information arriving over successive interactions, including learning and selective forgetting. Interactive-action workloads such as MemoryArena test whether earlier experience helps an agent complete later tasks with environment feedback. Choose a benchmark that matches how your agent receives and uses information; conversational recall scores are not direct evidence of enterprise-document accuracy.

Public Datasets for AI Agent Memory Evaluation

Start with LongMemEval when updates, dates, and abstention matter; use LoCoMo for conversational recall. Add incremental or interactive workloads when your application requires them rather than treating these datasets as interchangeable.

MemBench is a research benchmark; MemoryBench is evaluation software. Supermemory’s MIT-licensed MemoryBench harness supports custom benchmarks and side-by-side provider comparisons. Keep that distinction explicit in evaluation reports.

For context on reported platform results, see our review of published AI memory benchmarks. Use public results to select tests worth reproducing on your domain data, not as a substitute for that evaluation.

2026 Update: LongMemEval-V2

Treat LongMemEval-V2 as a separate benchmark. It contains 451 manually curated questions over multimodal web-agent trajectories, including web and enterprise domains, with haystacks reaching 115 million tokens. It evaluates answer accuracy and query latency. Its question set, source material, and scale differ from original LongMemEval, so report V2 results separately rather than combining them with, or presenting them as directly comparable to, original LongMemEval scores.

Why Benchmark AI Agent Memory on Your Own Data in 2026?

Public benchmark results help select tests; they do not establish the most accurate memory platform for a company’s PDFs, specifications, policies, or internal knowledge. Conversational, incremental-text, and interactive-action benchmarks exercise different workloads. MemoryArena’s authors report that agents with near-saturated performance on conversational benchmarks such as LoCoMo can perform poorly in their interactive agentic setting. That disparity supports workload-specific testing without establishing an enterprise-platform ranking.

When deployment depends on company documents, accuracy needs to be measured against held-out questions with known answers from that corpus. The evidence we reviewed does not establish a controlled, independently replicated winner across Cognee, Mem0, Zep, Letta, and Supermemory on the same enterprise-document corpus. Claims of superior enterprise accuracy therefore remain hypotheses to test on your own data; the most accurate layer for that workload must be determined by held-out results.

Common Challenges in Benchmarking AI Agent Memory

A cross-platform score is useful only when you know what each system retrieved, which evidence supports the answer, and which settings shaped the result. We recommend treating evidence mapping, update handling, comparison budgets, and deployment identity as explicit evaluation controls, not assuming that a shared question set makes every run comparable.

Key Problems Encountered in Memory Evaluation

Problem 1, Incompatible retrieval units. A retrieved chunk, graph fact, and summary are not interchangeable scoring units. Before computing retrieval precision or recall, map each returned item back to common source-document IDs and evidence spans. Define the relevant evidence for each question against those sources, rather than counting every platform-specific object as equivalent. Report answer accuracy separately from evidence retrieval so a plausible answer cannot conceal irrelevant or missing context.

Problem 2, Updates and missing answers. Test questions about superseded information, answers valid at a specific date, and questions with no supported answer as separate categories. For each case, record the required facts, evidence spans, as_of_date, and should_abstain expectation. Keep reference answers outside the ingested corpus. This lets you distinguish failures to use updated evidence from failures to respect a historical date or abstain when evidence is absent.

Problem 3, Unequal budgets. Mem0’s evaluation repository warns that embeddings, extraction models, judge models, and retrieval depth affect scores. Pin the corpus, questions, answering model, judge, prompts, retrieval depth, and context-token budget wherever configurable. Record settings you cannot control, including proprietary extraction behavior, rather than claiming identical internals. Include a simple vector-retrieval baseline and, where feasible, a full-context control to make the comparison more informative.

Problem 4, Harness and deployment differences. Audit what the evaluation harness actually returns. A public MemoryBench issue reported 30 results for Supermemory versus 10 for other providers. Verify the pinned commit and log actual returned counts and context tokens; the report does not establish that the issue persists in your release or invalidate every current run. Label managed-service and open-source results separately, too. Mem0’s repository states that published platform scores include proprietary optimizations absent from its open-source SDK, so self-hosted results should not be assumed to match them.

Use these controls together: common evidence mappings, category-specific tests, pinned configurations, and explicit deployment labels. Cognee’s documented evaluation components, including benchmark adapters, corpus building, answer generation, analysis, and exact-match, F1, context-coverage, and rubric metrics, support assembling a controlled trial. Their availability does not establish that any platform has passed these checks or automatically resolve comparison differences.

What to Look for in a Memory Platform You Can Benchmark Before Committing

We recommend Cognee as the first platform to trial: its Apache-2.0 public core, Python installation, local extraction option, and documented evaluation framework provide a developer-controlled starting point. This is an evaluation-suitability recommendation, not a claim that Cognee is the most accurate platform on every enterprise corpus. Select the production memory layer using held-out results from your own workload.

Must-Have Features for Own-Data Evaluation

Before comparing accuracy, check that each candidate supports a trial you can reproduce:

• Accessible trial route: identify the exact hosted service, local package, or container deployment you will test, including database and model prerequisites.
• Representative ingestion: verify that you can load your documents and preserve the identifiers, timestamps, and metadata needed by your questions.
• Inspectable evidence: require enough retrieval output or execution traces to distinguish missing evidence from incorrect answer generation.
• Configurable scoring and budgets: control the answerer, evaluator, scoring rules, and query budgets; record latency and cost alongside accuracy.
• Reproducible versions and deployment labels: pin packages and configuration, and label managed and self-hosted runs separately.

The table below compares documented trial capabilities. Row order puts our recommended starting point first; it is not a measured second-through-fifth ranking of setup speed or accuracy.

Cognee is our editorial starting point because it combines a local own-text route with documented evaluation components. Mem0 and Letta also have substantive evaluation tooling; Supermemory's MemoryBench is a separate harness, not the MemBench research dataset. MemoryBench documents providers for Supermemory, Mem0, and Zep; Cognee and Letta need additional integration, so do not plan around a zero-code five-platform run. The reviewed sources do not establish a controlled, independently replicated accuracy winner across all five on one enterprise-document corpus. Your held-out results must settle that question.

How to Build a Held-Out AI Agent Memory Benchmark from Your Documents

We recommend the following implementation workflow for comparing memory layers on your own corpus. It is an editorial evaluation design, not a published benchmark specification or evidence that any platform has passed these tests. Keep the source corpus, questions, and grading rules consistent across platforms; document any platform-specific configuration.

1. Sample a representative corpus.

   Action: Select documents across the formats and departments your agent will encounter. Include tables, obsolete policies, hard negatives that resemble relevant evidence, and documents whose answers require combining multiple sources. Avoid selecting only clean documents with easy lookup questions.

   Required artifact: A corpus manifest describing the selected documents, their formats, departments, and intended test coverage.

   Test purpose: Expose failures that a narrow conversational or single-format dataset would miss. Choose coverage based on your workload rather than treating a particular sample size as universally sufficient.

2. Preserve source metadata.

   Action: Retain document IDs, versions, timestamps, and access labels throughout preparation and ingestion. Keep obsolete and current versions distinguishable rather than flattening them into interchangeable text.

   Required artifact: A metadata record for every source document, linked by a stable document ID.

   Test purpose: Make evidence traceable and support tests of policy updates, time-sensitive answers, and access isolation.

3. Split development and held-out tests before tuning.

   Action: Designate development and held-out test partitions before adjusting ingestion, retrieval settings, prompts, or grading rules. Use the development partition for iteration and reserve the held-out partition for comparison after configuration is fixed.

   Required artifact: A versioned split manifest identifying which cases belong to development and which remain held out.

   Test purpose: Reduce the risk of mistaking tuning to known questions for generalization to unseen questions. Held-out questions can still query ingested source documents; their answers must not become ingestion material.

Create Ground-Truth Questions and Evidence Labels

4. Author questions with explicit answer requirements.

   Action: Write each question alongside a reference answer, required facts, and supporting document IDs and evidence spans. Store this answer key outside the ingested corpus. Label single-document lookup, multi-hop synthesis, numeric constraints, updates, unanswerable questions, and access-isolation cases separately.

   Required artifact: A versioned test-case file using a schema such as:

   question_id: string
   question: string
   reference_answer: string
   required_facts: string[]
   evidence_document_ids: string[]
   evidence_spans: string[]
   question_type: string
   as_of_date: string | null
   tenant_id: string | null
   should_abstain: boolean
   

   Test purpose: Distinguish a plausible answer from one that contains the required facts and has supporting evidence. The date, tenant, and abstention labels let you evaluate whether an answer is appropriate for the requested context, not merely present somewhere in the corpus.

Choose Evaluation Tooling and Platform Adapters

Use our guide to evaluating AI memory tools to identify scoring, provenance, and reproducibility components. Then verify how each tool accepts your documents and test cases; an existing benchmark runner does not automatically provide a custom-document adapter.

5. Configure ingestion and the appropriate adapter.

   Action: Map the corpus and test-case schema into each platform's ingestion and evaluation interfaces. Record versions, settings, and any adapter code needed to run the same questions.

   Start with Cognee's evaluation components: its documented framework includes benchmark adapters, corpus building, answer generation, analysis, DeepEval and direct-LLM evaluation paths, and exact-match, F1, context-coverage, and rubric metrics. Verify the available interface rather than assuming a proposed one-command runner or LongMemEval adapter has shipped.

   Letta Evals accepts user-authored JSONL datasets and YAML suites, with configurable graders, repeated runs, cached regrading, and CI execution. Its current quickstart targets Letta Code. Pin the package version and target configuration because its interfaces have changed, including removal of the older letta-agent target.

   MemoryBench requires custom benchmark implementation: loading, questions, sessions, ground truth, question types, and registration through its Benchmark interface. Cognee and Letta are not in its documented default provider list, so do not budget for a zero-code five-platform comparison.

   Mem0 also has evaluation tooling: its memory-benchmarks repository supports Cloud and OSS backends, published-dataset runners, configurable answerer/judge settings, and a results UI. Check the additional adapter work needed for your own corpus.

   Required artifact: A version-pinned configuration and adapter bundle for each platform.

   Test purpose: Make integration differences explicit rather than letting incompatible inputs or undocumented settings distort the comparison.

6. Run repeated trials and archive the evidence.

   Action: Run the fixed held-out set repeatedly. Use deterministic checks for IDs, dates, numbers, and enumerated facts; apply a reference-grounded rubric to synthesis answers. Calibrate model graders against expert-reviewed examples rather than relying on a single grading signal.

   Required artifact: An evaluation archive containing grader definitions, configurations, traces, final outcomes, category-level failures, and latency, token-use, cost, and error measurements for each trial.

   Test purpose: Separate repeatable behavior from run-to-run variation and retain enough evidence to investigate failures. Review results by question category as well as in aggregate so that strong lookup performance does not conceal update, abstention, or access-isolation failures.

Best Practices for Measuring Memory Accuracy, Retrieval, Latency, and Cost

Use separate measurements for answer quality, evidence retrieval, and operational performance. Combine code-based checks, model grading, and expert review rather than treating one score as sufficient. The protocol below is our recommended evaluation design, not a report of vendor-tested results or evidence that any platform has passed it.

Score Answer Accuracy and Evidence Retrieval Separately

Practice 1, Grade against known answers. Use deterministic checks for IDs, dates, numbers, and enumerated facts. For synthesis questions, write a reference-grounded rubric specifying which claims the answer must contain. Calibrate model graders against expert-reviewed examples before applying them across the test set. Run repeated trials, report variation across runs, and break out failures by question category rather than publishing only an aggregate accuracy score.

Practice 2, Define the retrieval unit and metric. For a simple, unranked retrieval measure, use:

precision@k = relevant retrieved items within cutoff k / total retrieved items within cutoff k

Name what counts as an item: a chunk, source passage, or evidence ID. Record the actual number returned when a system supplies fewer than k items. Also name the scoring implementation: Ragas’s ranked context-precision metric is not mathematically identical to this simple ratio; its ID-based context precision uses retrieved and reference context IDs.

For cross-platform scoring, map chunks, graph facts, and summaries back to common source-evidence IDs before comparing precision and recall. Define recall against the reference evidence required for each question. Keep answer accuracy independent: a plausible answer must not conceal irrelevant retrieval or missing supporting evidence.

Control the Comparison and Report Operational Costs

Practice 3, Freeze the comparison settings. Hold the corpus, questions, answering model, judge, prompts, and context-token budget fixed wherever configurable. Record embedding and extraction settings, retrieval depth, actual returned counts, and any proprietary extraction differences. These controls make the experiment interpretable without implying that platforms have identical internals. Mem0’s evaluation repository specifically warns that embeddings, extraction models, judge models, and retrieval depth affect scores.

Practice 4, Include baseline controls and explicit configurations. Run a simple vector-retrieval baseline through the same answering and grading harness. Where feasible, add a full-context control that gives the answerer the source material directly. Treat these as comparison conditions, not substitutes for the platform trials.

For Mem0, test vector-only and graph-enabled configurations explicitly. Its documented graph behavior adds related entities in a relations array alongside vector-search results; it does not automatically reorder the vector hits. Record which returned material the harness passes to the answerer, including whether it includes relations, and enforce the same context-token budget.

Practice 5, Measure latency under stated conditions. Report retrieval-only and end-to-end p50/p95 latency separately, with the timing boundaries documented. State concurrency and cold- versus warm-cache conditions, and report failures alongside latency rather than silently excluding them. Measure ingestion time and time until the data becomes searchable separately from query latency. These are measurements to collect on your workload, not published platform performance claims.

Practice 6, Account for the complete query path. Calculate cost per query from search fees, query-time embeddings, reranking, answer generation, and tool calls wherever those components apply. Report ingestion and storage costs separately, along with evaluation-judge costs, so the cost of running the benchmark is distinguishable from the cost of serving queries. Keep the same accounting boundaries across every platform.

What an Own-Data Memory Benchmark Lets You Decide

Use held-out results to decide which platform and configuration deserve further evaluation, not to declare a universal winner. Category-level failures matter: a system that handles synthesis adequately may still miss exact dates, identifiers, or numbers. Examine those failures alongside repeated-trial results before deciding whether its behavior fits your workload.

Keep answer accuracy separate from evidence retrieval. When comparing chunks, graph facts, and summaries, interpret retrieval scores against common source evidence. A plausible answer with missing support is a different failure from retrieving the right evidence and producing the wrong answer; those results point to different configuration questions.

Latency and cost provide separate acceptance criteria. Retrieval-only and end-to-end p50/p95 results help you assess responsiveness under the tested concurrency and cache conditions. Query costs inform the operating budget, while ingestion, storage, and evaluation-judge costs remain separate considerations. Do not collapse these dimensions into a composite score that hides an unacceptable trade-off.

Treat self-hosted and managed results as specific to the configuration tested. A result from a local trial does not establish how a differently configured production deployment will behave. Model choices, answer-generation settings, and operating conditions must remain attached to the findings you use to make a decision.

We recommend Cognee first for a developer-controlled evaluation. Its Apache-2.0 core supports a Python-based local trial, including local extraction through cognee[gliner] and building memory from your own text without a cloud LLM key. Its documented evaluation framework includes corpus building, answer generation, benchmark adapters, metrics, and analysis components that support collecting evidence. The keyless path does not establish production answer quality; production answer generation needs separate configuration and measurement.

How Cognee Supports a Developer-Controlled Memory Trial

Cognee is our first recommendation for an own-data trial because of its local setup path and documented evaluation support, not because published scores establish it as the most accurate option for enterprise documents.

The Apache-2.0-licensed public core supports Python installation, with cognee[gliner] providing a local extraction option for building memory from your own text without a cloud LLM key. Start with the documented add/cognify ingestion flow, then test graph, vector, and summary retrieval against your held-out questions. Keep extraction and answer generation separate: a keyless local extraction demo does not establish production answer quality. Configure and measure the answer-generation model independently.

The documented evaluation framework includes benchmark adapters, DeepEval and direct-LLM evaluation paths, and exact-match, F1, context-coverage, and rubric metrics. It also covers corpus building, answer generation, and analysis. A one-command runner and a LongMemEval adapter are tracked as roadmap work; check the repository for current implementation status before building your trial around them.

The BEAM report provides runnable evaluation configurations and archived artifacts, but its results have a narrow scope. The primary 100K result was 0.79 over 20 questions from one held-out conversation, averaged across four rounds. The 0.67 result at 10M was exploratory because routing was selected on the scored questions. Neither result establishes enterprise-document superiority or an independently replicated accuracy winner across platforms. Use the artifacts as evaluation starting points, then let your held-out results determine accuracy on your workload.

For a hosted trial, Cognee pricing states: $1.00 per 1M tokens processed, plus $5 per additional workspace. The free plan lists one workspace and 1M included tokens, with no card required. Those are platform processing charges, not a complete benchmark budget: external inference and evaluation expenses remain separate and must be included in your measured trial cost.

Key Takeaways and How to Start Your Memory Benchmark

Choose a public benchmark that matches your workload to inform evaluation design, then prepare held-out questions with known answers from your internal documents. Public scores alone do not establish enterprise-document accuracy. Compare configurations using the same corpus, questions, answering model, judge, prompts, and context budget wherever configurable; document differences you cannot control. Keep answer accuracy, evidence retrieval, latency, and full cost as separate decision inputs rather than hiding trade-offs in one score.

Start with Cognee’s documented local trial, or review its free plan, then apply the same evaluation design to alternatives relevant to your workload. Cognee is our first recommendation to trial, not a proven universal accuracy winner. Configure and measure production answer generation separately from a keyless local demo, and commit only when your held-out results meet your accuracy, latency, and cost requirements.

Related guides

• Best Evaluation Datasets and Benchmarks for Agent Memory
• AI Memory Benchmarks: The Complete Guide (2026)
• How to Evaluate AI Memory in 2026: 5 Tools Compared
• cognee on BEAM: SOTA Results Without a Benchmark-Specific Memory System
• AI Agent Memory Benchmarks: Cognee vs. Mem0, Graphiti, and LightRAG

FAQs About Benchmarking AI Agent Memory on Your Own Data

What is an AI agent memory benchmark?+

An AI agent memory benchmark tests how a system uses stored information to answer questions or support agent behavior. Keep three components separate: the dataset supplies sessions, questions, and expected answers; the evaluation harness runs ingestion, search, answering, and scoring; the memory platform is the system under test. MemoryBench, for example, is an evaluation harness, not the MemBench research dataset. Its checkpointed stages and configurable judges support provider comparisons, but the resulting scores describe performance on the selected workload, not universal memory quality.

Why should I benchmark memory on my own documents?+

Public conversational or interactive benchmark results do not directly establish accuracy on your company's PDFs, specifications, policies, or internal knowledge. Conversational recall, incremental text processing, and interactive actions test different workloads, and none alone establishes enterprise-document superiority. An own-data benchmark tests whether a platform can answer questions grounded in the material your application actually needs. Treat published scores as evidence for selecting candidates, then use held-out questions with known answers to test the deployment decision. This keeps a strong public benchmark result from becoming an unsupported assumption about your documents.

Which memory platform should I benchmark before committing?+

We recommend Cognee first for a developer-controlled evaluation, not as a proven accuracy winner. Its Apache-2.0 core, documented Python installation, and local extraction option through cognee[gliner] provide a path to building memory from your own text without a cloud LLM key. Configure and measure production answer generation separately; a keyless local extraction demo does not establish end-to-end answer quality. Mem0, Zep/Graphiti, Letta, and Supermemory remain workload-dependent comparisons. Mem0 also provides evaluation tooling with Cloud and OSS backends, configurable answerer and judge settings, and a results interface.

Can Cognee, Mem0, Zep, Letta, and Supermemory all be tested locally?+

Local testing is available across these options, but Zep requires a distinction. Cognee documents local extraction, and Mem0 provides an OSS setup with Docker Compose and Qdrant. Graphiti is the self-hosted open-source route associated with Zep; it is not Zep Cloud, and deprecated Zep Community Edition should not be used. Letta supports local and self-hosted configurations, while Supermemory currently documents a local server and Docker deployment. Check the release and feature coverage you test: local availability does not establish parity with managed services, and Graphiti results are not Zep Cloud results.

Can MemoryBench compare all five platforms on my custom dataset without adapter work?+

No. MemoryBench supports custom benchmarks, but adding yours requires implementing and registering its Benchmark interface, including data loading, questions, sessions, ground truth, and question types. Its documented default provider list includes Supermemory, Mem0, and Zep; Cognee and Letta require additional provider integration. The harness offers checkpointed ingestion, search, answering, and evaluation, plus configurable judges and structured reports, but those features do not eliminate adapter work. Plan the dataset implementation and missing provider integrations before assuming that one command will produce a comparable five-platform evaluation on your documents.

Which memory layer is most accurate on enterprise documents?+

The reviewed evidence does not establish a controlled, independently replicated accuracy winner across Cognee, Mem0, Zep, Letta, and Supermemory on the same enterprise-document corpus. Cognee is our first recommendation for evaluation suitability, not a demonstrated universal accuracy leader. Determine the most accurate option for your workload from held-out results on your documents. Keep the comparison's scope explicit: a Graphiti run is not a Zep Cloud result, and Letta's full agent behavior should be evaluated separately from isolated retrieval. Public benchmark scores cannot substitute for that evidence.