cognee on LoCoMo: 91% on Long-Term Conversational Memory

After BEAM, we wanted a second, different test of cognee's memory: LoCoMo, a benchmark of very long-term conversational memory. The question was the same as before. How does the existing open-source pipeline do, without a system built for the benchmark?

About LoCoMo

LoCoMo (paper, repository) contains 10 conversations between two people. Each has 19 to 32 dated sessions and 369 to 689 turns, and together they come with 1,986 questions about what was said.

The questions fall into five categories: single-hop, multi-hop, temporal, open-domain and adversarial. Open-domain questions combine what a speaker said with general knowledge. Adversarial questions can't be answered from the conversation at all: the right response is to say so, and the dataset pairs each with a plausible wrong answer.

How we ran it

Every session is cut into windows of six consecutive turns. Each window starts with a header naming the session, its date and time, and the two speakers, so the date stays next to the events it dates. The ten conversations give 1,048 windows, and each conversation goes into its own memory through one remember() call, with cognee's standard extraction and summary prompts.

At question time, cognee's hybrid retriever selects context in one pass from three channels: conversation passages, entities with their relationships, and further facts. gpt-5-mini then answers from that context under a single answer prompt, used for every question. The question is asked as written: neither its category nor its gold answer reaches the answering model, since a label like "adversarial" would tell it something an ordinary question doesn't.

How we graded it

Scope. We evaluate single-hop, multi-hop, temporal and adversarial questions. We leave out open-domain questions, which need general knowledge and so test more than memory. Adversarial questions stay in, because recognizing that a conversation holds no answer is part of conversational memory.

Audits. Some of LoCoMo's reference answers are wrong, which can make a faithful response count as an error. For example, the reference answer to "When did Evan have a drunken night with his friends?" is "January 9, 2023", but in a session dated 10 January 2024 Evan says it happened "yesterday". We combined an external audit by Penfield Labs with our own and excluded the 143 questions either one flagged. With the open-domain questions, that leaves 1,762 of the 1,986 questions. The exclusion list is published with each audit's reason.

Judge. gpt-5.1 grades each answer as correct or wrong in a separate call, so the answering model never grades its own output. It accepts paraphrases and equivalent formats, but a date must be at least as precise as the gold answer and consistent with it. For an adversarial question, only declining counts as correct.

Results

cognee answered about 91% of the 1,762 evaluated questions correctly: 1,598 (90.7%) and 1,608 (91.3%) in the two published runs. By category, for the run that scored 91.3%:

Accuracy ranges from 89.8% on multi-hop to 92.5% on single-hop questions, and adversarial questions reach 90.6% under the same prompt as the answerable ones. Each category has a few hundred questions, so a point or two between them should be read with caution.

The main configuration retrieves more context per question than cognee's default. At the default depth, with adapted ingestion and a global context index, cognee reached 89.7% in one run, within two points of the main result.

How this compares with other LoCoMo numbers

Published LoCoMo evaluations grade differently. Some judges accept a response that touches on the same topic as the reference, or allow dates within two weeks, and several leave adversarial questions out of the headline score. Our judge requires factual agreement and dates at least as precise as the expected answer, and our headline includes adversarial questions. The full report compares the judges behind Mem0, Zep, Hindsight and Supermemory's published numbers, at the revisions we inspected. For what else can move a LoCoMo score, and what a published score should disclose, see How to Get Ridiculously Good LoCoMo Scores.

What to make of this

The results hold up across all four question categories, which supports cognee's ability to handle long conversational histories. The audits help by setting aside faulty reference answers, and the published exclusions let anyone check those decisions.

The evaluation also has a limit worth stating: the ten conversations were used to develop the setup, so it isn't a held-out test. How consistently this carries over to other data and to real use remains to be tested. We'll keep evaluating across a broader range of memory tasks, with the methods and artifacts published for inspection and reuse.