How to Get Ridiculously Good LoCoMo Scores

There are several ways to get an eye-catching memory benchmark score. You could let a powerful model reread the entire conversation before answering. Or make the evaluation more forgiving: skip difficult questions, loosen the judge's criteria, or give the answer model information it shouldn't have. Or… you could actually improve the memory system.

All three can push the score higher, but only the last path is a genuine technical achievement. Passing off the first two as memory gains is basically gaming the benchmark.

LoCoMo tests question answering over long, multi-session conversations, and it’s one of the most-reported AI memory benchmarks. Its scores depend on the memory pipeline, the answer model, the evidence available at question time, and the grading rules — a leaderboard rarely shows all of that in one column.

In this post, we’ll walk through the shortcuts, the legitimate alternatives to a dedicated memory layer, and the engineering changes that can genuinely improve retrieval.

Every example here links to a public dataset file, prompt, evaluator or paper, so you can check the details yourself. None of them shows that anyone leaked answers or intentionally rigged their LoCoMo scores.

1. Four ways an evaluation can flatter a system

Want a higher score without improving the memory? Here’s how easily it can be done.

1.1. Let the answer key slip into ingestion

If the answer key gets into memory during ingestion, even weak retrieval can look highly accurate.

Each sample in LoCoMo’s public data file holds a conversation alongside a qa set of questions, reference answers, category labels, and the dialogue turns that support each answer. The same sample carries an annotated event_summary, which is the ground truth for a separate summarization task, plus observation and session_summary fields that the LoCoMo authors generated with GPT-3.5 and used as retrieval databases for their own RAG baselines.

An ingestion script that loads the whole sample indexes the answer key, and it also gives the memory system notes it didn’t write.

A clean and honest evaluation keeps ingestion limited to conversation data and leaves reference answers, evidence labels, and the pre-generated summaries out of searchable memory. Questions should arrive only as test queries, after ingestion is complete. It’s also important to record which fields were ingested because a high score alone can’t show whether that separation held.

1.2. Give the judge a generous rubric

An answer can be mostly correct, narrowly incomplete, or completely wrong. An LLM judge has to decide which of those outcomes deserves credit, and its instructions can change the result substantially.

If, for example, a question asks for four projects mentioned in a conversation, a strict all-items rule would reject an answer naming only three, while partial credit would recognize that the system recovered most of the information. Either policy can make sense, depending on what the test is meant to reward, but they produce different accuracy numbers.

The April 2026 revision of Mem0’s LoCoMo judge prompt marks an answer correct if it contains at least one correct item from a reference list (“Getting 1 out of 2, 2 out of 4, etc. is always acceptable”), so naming one of the four projects passes. It also accepts dates within 14 days of the reference, durations within 50%, and paraphrases. When evidence is supplied, an answer the evidence supports passes even if it differs from the gold answer.

The judge prompt in the appendix of the 2025 Mem0 paper is generous in a different way — it tells the judge that “as long as it touches on the same topic as the gold answer, it should be counted as CORRECT.” In an open issue on Mem0’s benchmark repository, a developer re-graded Mem0’s published platform answers with GPT-4o mini as the judge. They reported 94.32% under the current prompt and 88.46% under the earlier one Zep and Mem0 had published, a six-point difference on identical answers.

In order to reproduce a published number, you need to record the judge model, prompt revision, sampling settings, and scoring script.

1.3. Change which questions count

LoCoMo’s 1,986 questions ask about individual facts, connections across sessions, temporal events, and outside knowledge. Another 446 are adversarial, testing whether a system recognizes that the conversation doesn’t contain the answer. Evaluation protocols don’t always score all five categories.

The Mem0 paper reports results for four categories (single-hop, multi-hop, temporal, and open-domain) and excludes the adversarial questions, explaining that ground-truth answers weren’t available for them. Mem0’s current harness scores the same four, and so do Letta’s file-based LoCoMo evaluator and Memobase’s published comparison.

How a harness handles those categories can move a headline number a long way. In 2025, Mem0’s CTO pointed out that Zep’s reported 84% counted correct adversarial answers in the numerator while leaving adversarial questions out of the denominator. Zep consequently acknowledged the calculation error and corrected its score to 75.14%.

Even the category names don’t carry over between tables. Mem0’s harness labels LoCoMo’s category 1 (282 questions) as multi-hop and category 4 (841 questions) as single-hop, while Memobase’s README labels the same IDs single-hop and open-domain. The data supports Mem0’s labels: 95% of category 1 questions cite evidence from more than one session, against 2 of the 841 in category 4.

A four-category score says nothing about how well the system abstains when a conversation lacks the requested information, and open-domain performance also reflects the answer model’s general knowledge. So, before comparing two percentages, check that they come from the same question set, category labels, and weighting.

1.4. Write the answer prompt around the test

The answer model’s prompt can also carry knowledge about the dataset. Mem0’s published LoCoMo answer prompt tells the model that “All events occurred in 2022-2024” and to “Never output 2025 or 2026.”

The prompt also includes worked examples that teach the model how to make certain inferences. One says that if someone mentions playing a game exclusive to a particular console, the model can infer that they own that console. The GitHub issue linked above points out that LoCoMo contains a question asking which console a speaker owns, where this exact reasoning would help the model find the answer. There’s a possibility that Mem0 developed these examples after analyzing failures on LoCoMo itself, potentially giving its model benchmark-specific hints. As of this post’s publication, Mem0 hasn’t responded to the issue.

Prompt rules derived from the test questions raise the score on those questions without declaring how the system handles new conversations. Write prompts against development conversations, as described in section 3.3, and publish the answer prompt with the score.

2. High scores without a dedicated memory layer

A model can also answer questions about a conversation without any separate persistent-memory pipeline. That’s a legitimate baseline, as long as the comparison says what the model was allowed to access.

2.1. Read the entire conversation every time

A LoCoMo conversation runs to about 26,000 tokens, well within a modern context window, so you can send the whole transcript to the answer model with each question. A capable long-context model can search that material directly, and extra reasoning passes can additionally improve its answers.

The Mem0 paper includes this full-context baseline, and it scored the highest LLM-judge accuracy of every method tested: about 73%, against 68.44% for Mem0’s best memory configuration. It was also slow, at about 17 seconds of p95 latency, because the model reads the entire conversation for every question.

This measures how accurately a particular model answers with the source history supplied. It doesn’t, however, measure selective memory ingestion, consolidation, or retrieval. It also changes the economics — repeated questions resend the whole conversation, even when prompt caching lowers the effective cost.

Report full-context answer quality together with the input-token budget, caching assumptions, and latency. A compact external AI agent memory layer has to justify its extra ingestion and retrieval work against that baseline on cost and latency as well as accuracy.

2.2. Let an agent search the raw files

Another design gives the agent conversation files and tools to inspect them. It can search for terms, open a relevant session, follow an event across dates, and check its answer against the original dialogue.

Letta’s file-based LoCoMo evaluator writes each conversation session to a file and gives the agent a semantic file-search tool and an answer tool. Letta reported 74.0% for this setup with GPT-4o mini, above the 68.5% Mem0 reported for its best graph variant, although the two numbers come from different harnesses.

Files are persistent external storage, so calling this memory-free would be misleading. What the setup doesn’t necessarily test is whether a dedicated memory layer extracted and kept the important facts ahead of time: it can score well by going back to the original evidence on demand. That capability deserves its own label in a comparison table.

3. Tune the memory pipeline, then test whether it generalizes

We build cognee, so we’ll use it to illustrate two engineering decisions: how much information to prepare at ingestion, and how much work to do when a question arrives. These are experiments you could run yourself.

3.1. Prepare richer memories during ingestion

A basic pipeline might store conversation chunks and embeddings. A more structured one extracts people, events, dates, preferences, and relationships, then connects them in a knowledge graph, so later questions can be answered from a smaller amount of retrieved evidence. cognee’s default pipeline builds that graph and stores dates as timestamp nodes.

You could go further by deriving event durations, keeping successive versions of a preference, or resolving references such as “she” to the right person. Don’t assume those happen on their own: write them into the extraction step, which in cognee means a custom extraction prompt or a custom graph model. Each additional pass also adds ingestion cost and another chance to introduce errors.

To evaluate the change, keep the answer model, judge, question set, and retrieval configuration fixed. Compare how often the correct evidence is recovered, how accurate the answers are, and what the extra ingestion costs. Check the underlying records too, since a confidently extracted but wrong relationship can make later answers worse.

3.2. Or give retrieval more work at question time

You can also ingest the conversation with a simpler pipeline and use more elaborate retrieval for difficult questions. The answer process might decompose a multi-part question, combine vector retrieval with graph relationships, rerank evidence by date, and search again if the first result is incomplete. cognee ships these as separate search types (GRAPH_COMPLETION_DECOMPOSITION, HYBRID_COMPLETION, TEMPORAL, and GRAPH_COMPLETION_COT), so each can be switched on for a test run.

More retrieval work raises the chance of recovering information scattered across several turns, and it adds searches, model calls, and latency. Ten verification steps may fix a handful of answers, but the extra computation has to be justified by the result.

In our BEAM evaluation, we wrote separate prompts for each question type, and routing question types to different retriever parameters pushed the 100K score above 0.8. Multi-turn agentic retrieval raised scores further, but we left those runs out of the reported results because they felt outside the spirit of the benchmark. Those are BEAM findings; on LoCoMo, the same changes would need their own controlled run.

3.3. Compare ingestion and retrieval separately

The simplest experiment changes one part of the pipeline at a time, then tests whether combining the changes produces another improvement.

Keep the question set, answer model, answer prompt, judge, and scoring rules fixed across all four runs. Record ingestion cost separately from per-question cost. An accuracy gain bought with much longer query processing can still be worthwhile, as long as readers can see the expense.

For LoCoMo, split development and evaluation by conversation, so that no conversation contributes questions to both sets. Questions about a single dialogue often share people, events, and dates, and optimizing on some of them can indirectly reveal information needed for the others. Freeze the pipeline before testing the held-out conversations. The public release contains only ten conversations (a subset of the 50 in the original paper), so disclose how many you used for development and, when possible, also test on conversations from outside LoCoMo.

4. What should accompany a LoCoMo score?

A published percentage becomes more informative when the evaluation makes its conditions inspectable:

Where feasible, publish per-category results, answer outputs, and the retrieved evidence. A high aggregate score can hide weak temporal reasoning or poor abstention, and a correct final answer doesn’t prove that the memory system retrieved the evidence responsible for it. Published answers also let others re-grade them, which is how the judge-prompt difference in section 1.2 was measured.

Before You Trust a LoCoMo Score

A full-context model, a file-search agent, and a graph-based memory system can all produce convincing LoCoMo answers, but they use different resources and demonstrate different capabilities. Before comparing scores, check which questions were counted, what information the system could ingest, how the judge and answer model were prompted, and how retrieval was configured.

If you're evaluating your own memory system, develop it on one set of conversations and test it on another. Report answer accuracy alongside supporting evidence, latency, and token costs — that will give you a much clearer picture of whether your changes are worth the additional computation.

FAQ

Answers to the most common questions from this guide.

What is a J score on LoCoMo?+

J is the share of answers an LLM judge labels correct. The original LoCoMo paper scored answers with an F1 partial-match metric, and the Mem0 paper added LLM-as-a-Judge (J) because token-overlap metrics can give a high score to a factually wrong answer that shares most of the reference’s words. Mem0’s harness and many later reports use a binary CORRECT/WRONG judge, so J scores are comparable only when the judge model and prompt match.

How is LoCoMo different from LongMemEval?+

LoCoMo asks questions about ten long conversations between two people. LongMemEval has 500 questions about chat-assistant histories and scores five abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Knowledge updates, where a later message replaces an earlier fact, have no category of their own in LoCoMo.

Is LoCoMo still hard enough to test memory?+

It still tests temporal and multi-session recall, but each conversation is short enough for a current model’s context window, so it doesn’t force a system to retrieve selectively. To test scale, pair it with one of the larger agent memory benchmarks, such as BEAM, which runs conversations from 100,000 to 10 million tokens.

Can I compare my LoCoMo score with a vendor’s published number?+

Only if both were graded the same way. The most reliable comparison runs both systems through one open evaluation harness with the same answer model, judge, and question set. When a vendor publishes per-question answers, as Mem0 does in its benchmark repository, you can also re-grade those answers with your own judge and compare like for like.