cognee on LoCoMo: 91% With Standard Memory Components and a Strict Judge

After our BEAM results, the obvious next question was how cognee does on the benchmark everyone in AI memory actually quotes. LoCoMo is small by today's standards, its reference answers are imperfect, and the scores published on it are graded in ways that are hard to compare. It is also the number people ask us for.

So we ran it, with the same rule as before: standard cognee, no benchmark-specific memory system. This post covers the setup, the result, what we excluded and why, and how our grading compares with the judges behind other published LoCoMo numbers. The full technical report and all artifacts live in the cognee repository.

About LoCoMo

LoCoMo (Long Conversational Memory, ACL 2024) contains 10 conversations between two people, each spanning 19 to 32 dated sessions and 369 to 689 turns, with 1,986 questions about them. The questions come in five categories:

• Single-hop (841): the answer sits in one session.
• Multi-hop (282): the answer needs facts from several sessions.
• Temporal (321): the answer is a date or a span, often relative to when something was said.
• Open-domain (96): the answer combines something a speaker said with general world knowledge.
• Adversarial (446): the conversation does not contain an answer. The correct response is to say so, and the dataset pairs each with a plausible wrong answer.

The original scorer uses token overlap with the reference answer. Almost every published evaluation since has swapped that for an LLM judge, and the judges differ enough that scores from different vendors are not directly comparable. We come back to that below; for where LoCoMo sits next to LongMemEval, BEAM and the rest, see our AI memory benchmarks guide.

What we ran

Ingestion

The source is locomo10.json from the LoCoMo repository. We cut every session into windows of six consecutive turns. Each window starts with a header naming the session, its date and time, and the two speakers, so the date stays next to the events it dates; a shared photo is kept as its caption. A short overview document per conversation lists the speakers and the session timeline. Across the ten conversations that gives 1,048 windows.

Each conversation's documents go through one remember() call into their own memory, with cognee's standard extraction and summary prompts. remember() runs add, cognify (chunking, entity and relationship extraction into the knowledge graph, chunk summaries, embeddings) and improve. The chunk size, 4,096 tokens, is larger than any window, so every window becomes exactly one chunk. Extraction and summaries used gpt-5-mini; embeddings used text-embedding-3-large.

Excluding a question from evaluation removes nothing from ingestion. Every conversation is stored in full.

Retrieval and answering

For each question, cognee's hybrid retriever selects context in one pass from three channels: conversation passages (chunks), entities with their relationships, and further relationships not already listed with an entity. The main configuration pulls 20 passages, 20 entities and 20 facts per question, with up to 10 relationships per entity. gpt-5-mini then answers from that context.

One answer prompt is used for every question, and the question is asked as written. Neither its category label nor its gold answer reaches the answering model. That matters more than it sounds: the label "adversarial" tells a model that the conversation holds no answer, which is exactly the thing the question is supposed to test. Our prompt asks for the shortest complete answer, resolves relative dates against the session date, collects every item when a question asks for several, and replies "Not mentioned in the conversation" when the context does not attribute the fact to the person asked about.

What we scored

We evaluate single-hop, multi-hop, temporal and adversarial questions, and leave out two groups.

Open-domain questions (96) need general knowledge, so they test more than memory. Questions with faulty reference answers (143) can make a faithful answer count as an error. To find those we used two audits: an external audit by Penfield Labs, which flags 99 score-corrupting errors among the non-adversarial questions, and our own, in which gpt-5.1 checked every non-adversarial gold answer against the conversation and its session dates and flagged 93. The two overlap on 49 questions.

A typical case from conversation conv-49: "When did Evan have a drunken night with his friends?" has the reference answer "January 9, 2023". But in a session dated 10 January 2024, Evan says it happened "yesterday". The correct date is 9 January 2024, and a memory system that gets it right is marked wrong by the dataset.

Fifteen flagged questions are also open-domain, so 224 questions are excluded in all and 1,762 of the 1,986 are evaluated. The exclusion list, with each audit's error type and reason, is published alongside the results so you can disagree with any individual call.

Adversarial questions stay in. Recognizing that a conversation holds no answer is part of conversational memory, and several published LoCoMo numbers leave them out entirely.

The judge

An LLM judge, gpt-5.1 at medium reasoning effort, grades each answer CORRECT or WRONG in a separate call, so the answering model never grades its own output. The judge sees the question, its category, the gold answer and the response. It does not see the retrieved context or the conversation.

The rules are stricter than the ones behind most published LoCoMo scores. Paraphrases and equivalent formats are accepted. A date must be at least as precise as the gold answer and consistent with it, and a duration must describe the same span. For a gold answer with several parts, a response is correct when it gives at least half of them, rounding up, and contradicts none; extra details are allowed unless they contradict the gold. A response that hedges between several alternatives where only one matches is wrong, and so is declining to answer when the gold is a concrete fact. For adversarial questions, only declining is correct.

Results

Two published runs of the main configuration answered and judged the questions on the same memories, built by one ingestion per conversation. They scored 1,598 (90.7%) and 1,608 (91.3%) of the 1,762 evaluated questions.

By category, for the run that scored 91.3%:

The spread between categories is narrow: 89.8% on multi-hop to 92.5% on single-hop. Adversarial questions land at 90.6% under the same prompt as the answerable categories, without a special "refuse" mode. The smaller categories have a few hundred questions each, so a point or two of difference between them is within noise.

What the default retrieval depth gets you

The main configuration retrieves deeper than cognee's defaults, so we also ran the defaults as the more economical option: 5 passages, 5 entities and 5 facts per question, with cognee's global context index built after remember() and two ingestion adaptations. Relative time expressions were annotated with absolute dates before ingestion ("next month [= April 2023]"), and the extraction and summary prompts were adjusted to name the person in every fact, date every event, and have the summarizer write a few questions someone might later ask about the chunk. Ingestion never saw the benchmark's questions.

That configuration reached 89.7% (1,580 of 1,761 scored questions) in one run, within two points of the main result. Accuracy near 90% is reachable at the default retrieval depth with existing cognee components; the deeper retrieval buys the last point and a half.

On judging, and why the number is lower than it could be

LoCoMo scores from different vendors are graded differently, which is most of the reason the LoCoMo recount dispute happened. We inspected the current judge implementations behind several published results:

The sources and revisions for each row are in Appendix A.5 of the report. Supermemory changed its LoCoMo judging on 19 October 2026; the row describes the revision after that change.

Under a topical-match judge with a 14-day date tolerance and no adversarial questions, our number would be higher. We did not run that configuration, because it would not tell us much about memory. "Touches on the same topic" is a low bar for a system whose job is to return the right fact, and a date that is two weeks off is wrong for most things people use memory for. The 91% is the number under rules we would be comfortable applying to a production system.

Caveats

The same ten conversations were used to develop the setup and to report the result, so the evaluation is not held out. We did not tune per-question or per-category, and the answer prompt and retrieval settings are the same for all 1,762 questions, but the window format, prompt wording and retrieval depth were chosen while looking at this dataset. How consistently the performance extends to other data is the open question, as it is for every LoCoMo number.

Answering and judging also vary between runs. Our two published runs on identical memories differ by ten questions, and one judge verdict in the default-depth experiment could not be parsed (the judge wrote WRNG) and is left unscored rather than counted either way. Identical scores on a rerun are not guaranteed.

LoCoMo itself is roughly 26k tokens per conversation. It tests whether a system keeps dated facts about two people straight across a few dozen sessions. It does not test memory at the scale BEAM does, and it does not test whether an agent acts correctly on what it remembers. We treat it as one data point alongside BEAM, not as a ranking.

Reproducing it

The evaluation ships in the cognee repository as a benchmark package. From the repository root:

# main configuration
uv run python -m cognee.eval_framework.locomo.run_locomo_eval --conversations 0-9

# default retrieval depth with adapted ingestion
uv run python -m cognee.eval_framework.locomo.run_locomo_eval --conversations 0-9 \
    --resolve-dates \
    --extraction-prompt cognee/eval_framework/locomo/prompts/graph_extraction_conversations.txt \
    --summary-prompt cognee/eval_framework/locomo/prompts/summarize_content_questions.txt \
    --global-context-index \
    --config-json-path cognee/eval_framework/locomo/configs/locomo_hybrid_5_5_5_gci.json

Each command creates a fresh run directory, ingests the conversations, answers and judges the evaluated questions, and writes a summary. The dataset is downloaded when missing; the report lists its SHA-256. The report_artifacts/ folder holds per-question answers and judge verdicts for both configurations, the summary tables, the run configurations with model names and prompt hashes, both audits' exclusions, and the audit prompt. The same framework runs on your own corpus; our guide to benchmarking AI agent memory on your own data walks through that.

The stack was cognee 1.5.4 with Ladybug for the graph, LanceDB for vectors and SQLite for relational storage, the same embedded defaults a fresh pip install cognee gives you.

What to make of this

BEAM and LoCoMo stress different things, and cognee's standard pipeline holds up on both without a benchmark-shaped memory system in between. That is the result we care about. The 91% is a useful number because of what is attached to it: a published exclusion list you can argue with, a judge prompt you can read, and per-question verdicts you can recount.

We will keep running cognee against a broader range of memory tasks and publishing the methods and artifacts each time. If you are evaluating memory systems, read the judge before you read the score, and if a vendor's number does not come with one, ask for it.

— Vasilije Marković, Founder and CEO, cognee

Related guides

• cognee on BEAM: SOTA Results Without a Benchmark-Specific Memory System
• AI Memory Benchmarks: The Complete Guide
• Best Evaluation Datasets and Benchmarks for Agent Memory
• How to Benchmark AI Agent Memory on Your Own Data
• How to Evaluate AI Memory Tools
• Behind the Viral Benchmark Numbers

FAQs about cognee on LoCoMo

What score does cognee get on LoCoMo?+

About 91%: 1,598 (90.7%) and 1,608 (91.3%) of 1,762 evaluated questions correct across two published runs of the same configuration, graded by a gpt-5.1 judge with strict rules on dates, hedging and adversarial questions. A configuration at cognee's default retrieval depth reached 89.7%.

Why does cognee evaluate 1,762 questions instead of all 1,986?+

We exclude the 96 open-domain questions, which need general knowledge rather than memory, and 143 questions whose reference answer an external audit by Penfield Labs or our own gpt-5.1 audit found faulty. Fifteen overlap, so 224 are excluded. The full exclusion list with reasons is published in the cognee repository, and every conversation is still ingested in full.

Are adversarial questions included in cognee's LoCoMo score?+

Yes. All 446 adversarial questions are in the headline number and are answered under the same prompt as every other question, with no category label passed to the model. Only declining to answer counts as correct. Several published LoCoMo scores exclude adversarial questions or report them separately.

Can cognee's LoCoMo score be compared with Mem0's or Zep's?+

Not directly. Those evaluations use judges that accept a response "touching on the same topic" as the reference or allow dates within 14 days, and most exclude adversarial questions. Our judge requires factual agreement and dates at least as precise as the gold, and includes adversarial questions. Under their rules our number would be higher; we did not run that configuration.

How can I rerun the cognee LoCoMo evaluation?+

The benchmark package is in the cognee repository under cognee/eval_framework/locomo/. One uv run command ingests the ten conversations, answers and judges the evaluated questions, and writes a summary. The report lists the models, settings, dataset hash and prompt hashes, and report_artifacts/ holds the per-question records from the published runs.