
Which AI Memory Layer Has the Lowest Retrieval Latency at Scale?

AI memory retrieval latency sounds like a clean thing to compare, until the published figures are placed next to each other. Zep, Mem0, Pinecone, Redis, and Cognee all publish latency, but they measure different spans, with different reader models, judges, and retrieval depths behind the number.
This guide compares retrieval latency across the memory layers most often evaluated for agent workloads: Cognee, Mem0, Zep and Graphiti, Pinecone, and Redis. It defines what retrieval latency actually includes, lists what to measure (p50, p95, p99, cold versus warm), explains how graph traversal depth and vector index choice move those numbers, and provides a reproducible benchmarking method so the comparison can be rerun on your own corpus. Published figures are cited from vendor documentation, the Mem0 paper, and the Zep paper wherever they exist.
Independent coverage of agent memory frameworks warns that latency columns across vendors are not directly comparable, because each vendor measures a different span and uses different reader models, judges, and retrieval depths. A third-party head-to-head makes the point concretely: Zep publishes retrieval-only numbers, while Mem0 reports an end-to-end figure that stays at or under 1.1 seconds across every benchmark it ran, including BEAM at 10M tokens. One column is a slice of the pipeline, the other is the whole pipeline. Dropping both into the same table without that context will mislead.
Two architectural decisions shape any memory layer's latency profile and are worth stating up front: how many datastores sit on the retrieval path, and what language the query path runs in. Cognee 1.0, for example, ships graph, vectors, sessions, and metadata on a single self-hosted Postgres instance, removing a network hop between graph and vector queries, and its retrieval engine is written in Rust, which is the lever behind the 0.24s recall figure discussed later in the guide. The sections that follow define retrieval latency precisely, lay out the published numbers that can be cited honestly, and give a benchmarking recipe for the numbers that cannot.
What does AI memory retrieval latency include?
Retrieval latency inside an AI memory layer is the wall-clock time from the moment a query enters the system to the moment assembled context is handed back to the model. The headline number depends entirely on which spans are included, which is why two platforms can both claim fast retrieval while measuring very different things.
Memory layer vs vector database
A bare vector database answers one question: given an embedding, return the nearest neighbors. A memory layer answers a harder one: given a natural-language query and a session, return the context the model should reason over. That means graph edges, session state, and metadata filters all participate in the lookup, not just approximate nearest-neighbor (ANN) search.
The topology of the memory layer decides how many of those lookups cross a network boundary. A stack built from a separate vector store, a graph engine, and a metadata service pays a round trip for each. A single-store design, where graph tables, vector indexes (for example pgvector HNSW indexes), sessions, and metadata share one Postgres instance, turns retrieval into one coordinated query instead of a fan-out.
Spans that get measured: embed, search, graph traversal, rerank, context build
A realistic retrieval path through a memory layer touches at least five spans:
- Embed, convert the query string into a vector.
- ANN search, retrieve candidate nodes from the vector index.
- Graph traversal, expand candidates along typed edges to pull related entities and provenance.
- Rerank, score candidates against the query, often with a cross-encoder or LLM.
- Context build, serialize the final payload the model will consume.
A number that covers only step 2 will look dramatically faster than one that covers all five, even on identical hardware and data. Any honest latency comparison names the spans it includes.
Those spans should be directly observable rather than inferred. OpenTelemetry is the common denominator: a memory layer that records each operation (ingest, graph build, search, delete) as a millisecond histogram, with an attribute naming the search type, lets operators compute p50, p95, and p99 per operation and per query type instead of reading a single aggregate number. The same spans should show up whether retrieval is driven from a notebook, a service, or a shell.
Why does retrieval latency at scale matter?
Retrieval latency has moved out of the infrastructure column and into the product column. When a memory layer sits inside an agent loop or a voice interface, every millisecond spent fetching context is a millisecond the user waits, and the cost compounds across steps, sessions, and corpus growth. Two forces make this acute: agent architectures that call retrieval many times per task, and realtime interfaces where perceived delay breaks the interaction. A third pressure is corpus size, benchmarks now span from 100K tokens to 10M tokens, and retrieval quality must hold across that range without blowing up token budgets.
Agent loops and voice workloads
Multi-step agents amplify retrieval cost by step count. A planner that issues eight retrieval calls at 400ms each spends 3.2 seconds inside memory before the model generates a single user-visible token. Voice and realtime UIs are stricter: conversational turn-taking research puts the comfort threshold for a response under roughly 300ms, which is also where published hybrid retrieval systems have landed. Graphiti, for example, reports P95 retrieval latency of 300ms by combining semantic embeddings, BM25 keyword search, and direct graph traversal while avoiding any LLM calls on the hot path.
Cost of over-retrieval
Latency gets worse when retrieval quality forces larger context windows to compensate. Pulling more candidates to hit an accuracy target raises token spend on every call and slows downstream generation, which then slows the next retrieval in the loop. Cognee's BEAM run argues that this spiral is avoidable: 79% at 100K context and 67% at 10M context, ahead of previous best results of 73.4% and 64.1%, with token usage staying flat as the memory grows. Flat token usage at larger corpus sizes is what keeps per-call latency predictable when the underlying graph scales from gigabytes to terabytes.
What makes retrieval latency hard to measure at scale?
Vendor latency tables look authoritative until the measurement spans are compared side by side. Independent coverage warns that latency columns across memory vendors are not directly comparable because they measure different spans and use different reader models, judges, and retrieval depths. Before trusting a published figure, the span, hardware, dataset, and load profile behind it must be reconstructed, which is only possible when the vendor publishes the harness that produced it.
Problems encountered when comparing memory layers
- Different spans measured. A recent head-to-head notes that Zep publishes retrieval-only numbers while Mem0 reports an end-to-end figure that holds at or under 1.1s across every benchmark including BEAM at 10M tokens. Retrieval-only and end-to-end spans cannot be placed in the same column. A harness that records the full pipeline lets retrieval, graph traversal, and generation be separated or combined on demand.
- Cold versus warm asymmetry. Serverless vector stores pay a cold-start penalty when compute spins up, which is why Pinecone introduced Dedicated Read Nodes with a memory-plus-local-SSD warm path for billion-vector workloads. A single mean hides which path the production traffic actually takes. Benchmark runs should report cold and warm percentiles separately for every search strategy.
- Graph traversal cost. The graph-enhanced Mem0g variant delivers higher accuracy than base Mem0 but raises total latency to 1.091s p50 and 2.590s p95 on LOCOMO, against 0.708s and 1.440s for base Mem0, per the Mem0 paper (arXiv 2504.19413). Any comparison that mixes a graph-augmented recall path with a flat vector path will misrepresent both. Hold traversal depth constant across strategies so the cost of each hop is visible.
- Load degradation. Almost every published figure is a single-turn, single-client measurement. Zep's own account of a scaling incident, where graph search p95 went from 150 ms to 600 ms before vector and BM25 search were moved to a dedicated service, shows how far concurrent traffic can move the tail. Single-turn numbers do not survive concurrent traffic. Load profiles that mirror production session counts should be part of every measurement, with per-span latencies exported to an observability backend for the same scrutiny.
What should a low latency memory layer provide?
When retrieval latency at scale is the primary constraint, vague vendor adjectives like "fast" or "scalable" are not enough. The evaluation should reduce to measurable properties that can be reproduced on your own corpus. Below is a concrete checklist.
Necessary features
-
A flat latency curve across corpus sizes. The useful signal is how p95 behaves as the graph grows by orders of magnitude, not a single number. Zep's platform page, for example, publishes p95 retrieval of 148 ms at 10K, 152 ms at 100K, 156 ms at 1M, 161 ms at 10M, and 168 ms at 100M graph size, a ~20 ms spread across four orders of magnitude. Ask any candidate memory layer for the same curve on the same axes.
-
Published p50, p95, and p99, not averages. Means hide the tail behavior that actually degrades agent UX. A vendor publishing only an "average response time" is a signal to run your own benchmark before trusting the number.
-
No LLM calls on the hot path. Any synchronous model call during retrieval adds hundreds of milliseconds and injects provider variance into p99. Graphiti, for instance, reports P95 retrieval of 300 ms specifically by combining semantic embeddings, BM25 keyword search, and direct graph traversal and avoiding LLM calls during retrieval. The checklist item: confirm that query-time retrieval does not invoke an LLM.
-
Per-operation observability. Aggregate dashboards do not help when p99 regresses on a specific query type. Look for per-operation latency histograms exported through OpenTelemetry, with an attribute that names the search strategy, so a p95 regression can be attributed to graph traversal depth, vector search, or ingestion back-pressure rather than guessed at.
-
A reproducible self-benchmark on your data. Published numbers reflect the vendor's corpus, embedding model, and query mix. Look for an open evaluation harness in the vendor's repository that benchmarks retrieval on standard multi-hop QA datasets and compares search strategies, with no third-party evaluation service required. The checklist item: can the candidate platform be benchmarked end-to-end on your corpus before a procurement decision?
-
Single-store deployment to cut network hops. Each additional datastore in the retrieval path (separate graph DB, separate vector DB, separate metadata store) adds a network round trip to p95 and another failure domain to p99. A memory layer that keeps graph, vectors, sessions, and metadata on one Postgres instance removes those hops; pgvector's HNSW index documents the speed-versus-recall trade-off that then governs the vector span.
If a memory layer cannot answer each of these six questions with a number or a repository path, retrieval latency at scale is being asserted rather than demonstrated.
Published retrieval latency: Cognee, Mem0, Zep, Graphiti, Pinecone, Redis
The table below collects published retrieval latency figures across widely cited AI memory layers and vector stores. Numbers come from vendor pages, docs, and peer-reviewed or independent benchmarks. Each row records the exact span measured so readers can judge whether a figure covers pure vector search, hybrid graph retrieval, or an end-to-end agent memory call.
Comparison table
| Platform | Category | Published p50 | Published p95/p99 | Scale measured | Span measured | Source |
|---|---|---|---|---|---|---|
| Cognee (Rust engine, OpenAI-backed) | Graph + vector memory | 0.23s (small doc set); 0.27s (War and Peace) | — | Single-document to novel-length corpus | End-to-end search call via cognee.rust vs 2.66s-4.03s Python SDK | memory-on-device guide |
| Cognee Enterprise (cognee.rust) | Graph + vector memory | 0.24s recall | — | Enterprise reference workload | Recall via Rust engine vs 1.77s Python SDK; cold start fits inside a Lambda or Workers call | cognee.ai/product/enterprise |
| Cognee 1.4 (LanceDB + Kuzu, top_k=15) | Graph + vector memory | CHUNKS_LEXICAL ~10ms; CYPHER ~1ms; CODING_RULES ~1ms | — | Small knowledge base | Median latency per search type | docs.cognee.ai search types |
| Mem0 (LOCOMO, ECAI 2025) | Agent memory | Search 0.148s; total 0.708s | Search 0.200s p95; total 1.440s p95 | LOCOMO | Search-only and end-to-end agent memory call | arXiv 2504.19413 |
| Mem0g (graph variant, same paper) | Agent memory + graph | Search 0.476s; total 1.091s | Total 2.590s p95 | LOCOMO | Search-only and end-to-end agent memory call | arXiv 2504.19413 |
| Zep | Temporal knowledge graph memory | — | 148ms (10K) → 152ms (100K) → 156ms (1M) → 161ms (10M) → 168ms (100M) p95 | 10K to 100M graph size | Retrieval only; reported under 200ms regardless of graph size | getzep.com |
| Zep (benchmark context) | Temporal knowledge graph memory | — | 155ms at 94.7% on LoCoMo (5,760 tokens); 162ms at 90.2% on LongMemEval (4,408 tokens) | Benchmark datasets | Retrieval latency alongside accuracy | getzep.com/platform/graphiti |
| Graphiti | Temporal knowledge graph | — | 300ms P95 | Hybrid retrieval workload | Semantic embeddings + BM25 + graph traversal; no LLM calls at query time | neo4j.com (Graphiti post) |
| Pinecone Dedicated Read Nodes | Vector DB (ANN) | — | — (qualitative) | Billion-vector scale | Reserved capacity with memory + local SSD warm path | Pinecone docs |
| Redis 8 | Vector DB (ANN) | ~200ms median at 90% precision; ~1.3s median at 95% precision | — | 1B vectors, 50 concurrent queries, top 100 NN | Pure ANN query; ~66,000 insertions/sec sustained | redis.io/blog/searching-1-billion-vectors-with-redis-8 |
How to read the table
Each row measures a different span, so the numbers are not directly comparable. Zep's figures are retrieval-only against its temporal knowledge graph, which is why they stay flat from 10K to 100M nodes. Mem0's headline LOCOMO numbers split search (p50 0.148s, p95 0.200s) from the total agent call (p50 0.708s, p95 1.440s). Mem0g trades latency for accuracy and lands at 2.59s p95 total. Redis 8 reports pure ANN query time with no graph traversal or LLM involvement, at ~200ms median for a billion vectors at 90% precision under 50 concurrent queries.
Cognee's figures split by engine. The cognee.rust engine returns 0.23-0.27s on OpenAI-backed runs and 0.24s on the enterprise reference workload, roughly an order of magnitude faster than the Python SDK on the same inputs. The per-search-type medians on version 1.4 with LanceDB and Kuzu (CHUNKS_LEXICAL ~10ms, CYPHER ~1ms, CODING_RULES ~1ms at top_k=15) describe local operations on a small knowledge base, not full end-to-end agent calls. A third-party head-to-head makes the same caution explicit: Zep publishes retrieval-only numbers, Mem0 reports a figure that holds at or under 1.1s across every benchmark including BEAM at 10M tokens, and vendor latency columns use different reader models, judges, and retrieval depths.
These rows should not be averaged into a single ranking without normalization. For a defensible comparison, pick one span (retrieval-only or end-to-end), one dataset, one hardware profile, and one precision target, then rerun each candidate under those conditions. The benchmarking method later in this guide walks through how to do that on your own corpus.
Which patterns keep retrieval latency low in production?
Low-latency retrieval at scale rarely comes from a single optimization. It comes from stacking patterns across the query path, the storage layout, and the ingestion pipeline. The patterns below are drawn from published engineering writeups and product documentation across Cognee, Graphiti, Zep, and Pinecone, with each pattern tied to the feature or offering that enables it.
Advanced patterns in production
Pattern 1, Rust runtime on the query path. A Rust engine with embedded storage removes network round trips entirely for local workloads and keeps the retrieval path inside a single process. The reference implementation here is a Rust port of a Python memory SDK built for phones, smartwatches, and embedded hardware, using an embedded graph store and an embedded vector store (LanceDB persistent on desktop, pure-Rust brute-force in-memory on Android).
Pattern 2, Single-store memory to remove cross-store hops. Putting the entire memory layer, graph, vectors, sessions, and metadata, on a single self-hosted Postgres instance, with no separate graph database to operate, eliminates the fan-out joins across a vector DB, a graph DB, and a metadata store that often dominate tail latency in multi-service memory stacks. On the vector side, pgvector's HNSW index exposes hnsw.ef_search so recall and latency can be traded explicitly per query.
Pattern 3, Pre-built graphs with no LLM at query time. Graphiti reaches P95 retrieval latency of 300 ms using a hybrid of semantic embeddings, BM25 keyword search, and direct graph traversal, avoiding any LLM calls during retrieval, on graph backends such as Neo4j, FalkorDB, or Neptune. Shifting entity extraction and graph construction to ingestion keeps query-time cost predictable as the graph grows, because retrieval depends on index lookups rather than model inference.
Pattern 4, Dedicated read capacity for billion-vector workloads. Pinecone's Dedicated Read Nodes provide reserved capacity with a warm data path of memory plus local SSD, avoiding cold fetches from object storage and keeping latency low at billion-vector scale. For corpora where serverless cold reads dominate the p99, reserving read capacity is the mechanism that controls tail behavior.
Pattern 5, Per-search-type routing. A memory layer that exposes distinct search types with different median latencies lets the application choose the cheapest path for each query. Cognee's 1.4 docs report, on a small knowledge base with default top_k=15 using LanceDB and Kuzu: CHUNKS_LEXICAL around 10 ms, CYPHER around 1 ms, and CODING_RULES around 1 ms. Routing short structured questions to CYPHER and keyword lookups to CHUNKS_LEXICAL keeps the median down without forcing every query through the heaviest path.
Pattern 6, Ingestion optimization to shrink the write-side tail. Slow ingestion backs up queues and inflates read latency during bursts. Instrumenting the ingest and graph-build operations with the same latency histogram as search makes write-side p95 visible next to read-side p95, so regressions in the ingestion path show up before they surface as stale reads. A tighter ingestion path reduces the window in which readers see partially indexed data and keeps freshness under control as the corpus grows.
Pattern 7, Isolating graph and vector workloads. After a scaling incident, Zep brought graph search back to 150 ms P95 (down from 600 ms) and context retrieval to 200 ms P95 by offloading all vector and BM25 search to a dedicated service and letting the graph database focus on graph operations, reporting 65 ms P75 search latency at massive scale. Separating workloads by engine prevents one query class from starving another when traffic is uneven.
How do you benchmark retrieval latency yourself?
Published numbers rarely survive contact with production traffic. The six practices below are written for anyone preparing a head-to-head evaluation of AI memory layers and want results that will hold up to review from an engineering lead, a procurement team, or a skeptical reader on Hacker News.
Expert tips
1. Report p50, p95, and p99 separately, never just the mean. Averages hide the tail that actually hurts agent UX. In the Mem0 paper, base Mem0's total latency is 0.708s at p50 but 1.440s at p95, roughly double; the p95 is the figure a user feels when an agent stalls. Publish all three percentiles for every system under test, and label the sample size and time window used to compute them.
2. Separate cold and warm paths in the results table. Serverless and autoscaled vector stores behave very differently when a shard is spun down: the first query after idle pays for compute to spin up and for data to be fetched from object storage, which is the gap Pinecone's Dedicated Read Nodes exist to close. Blending the two produces a number that describes neither path. Record cold-start latency as its own column, and note how long a node must be idle before the next query counts as cold.
3. Hold the reader model, judge, and retrieval depth constant across vendors. Independent coverage of agent-memory frameworks has warned that latency columns across vendors are not directly comparable, because they measure different spans and use different reader models, judges, and retrieval depths. Pin the embedding model, the top-k, the reranker (or lack of one), the LLM used for answer synthesis, and the LLM-as-judge configuration. Publish the configuration file alongside the numbers so a third party can rerun it.
4. Instrument per-span latency with OpenTelemetry. End-to-end wall-clock numbers conflate network, embedding, graph traversal, vector search, and answer generation. Emit a millisecond histogram per memory operation (ingest, graph build, search, delete) with an attribute naming the search strategy, then route those spans into any OTel-compatible backend (Jaeger, Tempo, Honeycomb). That makes it possible to attribute a slow p99 to the component that caused it rather than to the system as a whole.
5. Run under realistic concurrency, not single-turn loops. Serial benchmarks flatter every vendor. Redis's billion-vector benchmark is one of the few that states its concurrency (50 concurrent queries), and Zep's scaling-incident writeup shows graph search p95 quadrupling under load before the workloads were separated. Mirror the concurrency profile of the target workload, agent swarms, parallel tool calls, multi-tenant inference, and record how p95 and p99 move as sessions scale.
6. Use a reproducible evaluation harness, not ad-hoc scripts. One-off notebooks drift between runs and across contributors. Use a self-contained evaluation framework that benchmarks retrieval quality on standard multi-hop QA datasets and compares search strategies without depending on a third-party evaluation service. Running the same harness against each candidate memory layer, Cognee, Mem0, Zep, a raw vector store, a hand-rolled graph, produces an apples-to-apples report that can be versioned, reviewed, and rerun whenever a dependency is upgraded.
What do you gain from a low latency memory layer?
When retrieval holds its latency budget under load, several operational benefits follow. The ones below are tied to published measurements; anything outside those numbers is omitted on purpose.
Measurable benefits
- Sub-second recall from a serverless cold start. Cognee's enterprise page reports recall in 0.24s via the Rust engine, compared with 1.77s on the Python SDK, with a cold start that fits inside a Lambda or Cloudflare Workers invocation. That makes the memory layer usable from short-lived function environments without a warm pool.
- Flat latency as the graph grows. Zep's platform page publishes p95 retrieval of 148 ms at 10K, 152 ms at 100K, 156 ms at 1M, 161 ms at 10M, and 168 ms at 100M graph size, a 20 ms spread across four orders of magnitude. A related form of flatness shows up on BEAM, where a graph-backed layer held 79% accuracy at 100K context and 67% at 10M context with token usage staying flat as data grows.
- Ingestion that keeps pace with query traffic. Low query latency only helps if writes do not back up behind it. Instrumenting ingest and graph-build operations with the same histogram as search keeps the ingestion path visible, so it does not quietly become the ceiling on end-to-end freshness.
- On-device feasibility. Running cognee-rs on a Samsung Galaxy S24 Ultra, the full pipeline over Alice in Wonderland completed in 139.12s, and once memory was built, chunk and summary search landed in the low-single-second range (~1s). A memory layer that runs this way on a phone also runs predictably on constrained edge hardware.
- Predictable cost per query. Usage-based pricing tied to tokens processed, rather than per-seat or per-node fees, lets latency budgets and spend budgets be reasoned about with the same unit, because both scale with what is actually ingested and queried.
How Cognee reduces retrieval latency across the stack
Cognee reduces retrieval latency through three concrete mechanisms: a Rust port of the Python SDK on the query path, a single-Postgres deployment that collapses graph and vector storage into one engine, and per-search-type routing with OpenTelemetry instrumentation that lets operators verify numbers against their own data.
Rust engine on the query path
The cognee.rust engine is a Rust port of the Python cognee SDK built for on-device memory on phone, smartwatch, and embedded targets, using an embedded graph store and an embedded vector store (LanceDB persistent on desktop, pure-Rust brute-force in-memory on Android); the Rust SDK for edge guide covers the build targets. On OpenAI-backed runs published by cognee, the Rust engine returned search in 0.23s on a small document set and 0.27s on War and Peace, compared with 2.66s and 4.03s respectively for the Python SDK. The enterprise page reports recall in 0.24s via cognee.rust versus 1.77s on the Python SDK, with cold start that fits inside a Lambda or Workers invocation. The four-verb API, remember, recall, improve, forget, composes the lower-level add → cognify → memify → search pipeline and is reachable from CLI, language bindings, and an HTTP server.
Single-Postgres deployment
Cognee 1.0 ships the entire memory layer, graph, vectors, sessions, metadata, on a single self-hosted Postgres instance, with no separate graph database to operate; pgvector is a supported vector backend alongside LanceDB (the default), and Neo4j, Kuzu, and FalkorDB remain available as graph backends for larger topologies. Collapsing storage into one engine removes the cross-system network hop that otherwise dominates traversal latency when a query joins vector hits against graph neighborhoods. The same deployment model is what the published figures are measured against, so numbers reproduced on local Postgres should track the reference profile rather than a multi-service topology.
Per-search-type routing and observability
Different search types carry different latency profiles, and cognee exposes both the routing and the measurement. The search-type docs report median latency per search type on cognee 1.4 with LanceDB and Kuzu: CHUNKS_LEXICAL around 10 ms, CYPHER around 1 ms, and CODING_RULES around 1 ms, measured on a small knowledge base with default top_k=15. Memory operations are instrumented through OpenTelemetry: memory.operation.duration is recorded as a histogram in milliseconds for add(), cognify(), search(), and forget(), with memory.query.type attributes that let operators break latency down per search type in whatever backend already collects traces. For verification on real corpora, cognee ships a self-contained evaluation framework at cognee/eval_framework/ that benchmarks retrieval quality on standard multi-hop QA datasets and compares search strategies in an interactive HTML dashboard, with no third-party evaluation service required.
Cognee pricing is $1.00 per 1M tokens processed, plus $5 per additional workspace.
Next steps
The honest answer to "which memory layer has the lowest retrieval latency at scale" depends entirely on which span of the pipeline is being measured and under what workload.
Which layer wins depends on the span measured
For pure ANN vector lookup, a dedicated vector database is the floor: Redis 8 reports ~200 ms median for a billion vectors at 90% precision under 50 concurrent queries, and smaller indexes on reserved capacity run far below that. For flat retrieval-only p95 across graph sizes, Zep's platform page reports 148 ms at 10K, 152 ms at 100K, 156 ms at 1M, 161 ms at 10M, and 168 ms at 100M, effectively under 200 ms regardless of graph size. For end-to-end search latency on LOCOMO, Mem0's ECAI 2025 paper records the lowest among tested methods at p50 0.148s and p95 0.200s, with total median latency of 0.708s and p95 of 1.440s. Cognee's position is narrower and more specific: when a graph-structured memory layer with entities, relationships, vectors, sessions, and metadata must run from a single store, the Rust-based engine advertises recall at 0.24s versus 1.77s on the Python SDK, with cold start that fits inside a Lambda or Workers call, and the 1.0 release consolidates graph, vectors, sessions, and metadata onto a single self-hosted Postgres instance.
Try Cognee on your own workload
Published numbers are a starting point. The decision should rest on measurements against representative queries, graph depth, and corpus size. Clone the topoteretes/cognee repository, point the evaluation framework at a dataset that mirrors production retrieval patterns, and record p50, p95, p99, and cold-versus-warm numbers before committing to a memory layer. Cognee pricing is $1.00 per 1M tokens processed, plus $5 per additional workspace, so a full benchmark against production data is inexpensive to run end to end.
FAQ
Which AI memory layer has the lowest retrieval latency at scale?
The answer depends on what is being measured. For pure vector ANN, a dedicated vector database is the floor; Redis 8 publishes ~200 ms median at a billion vectors and 90% precision. For retrieval inside a memory layer, Zep reports p95 between 148 ms at 10K and 168 ms at 100M graph nodes. Mem0 publishes p50 of 0.148s and p95 of 0.200s on LOCOMO search, with total end-to-end median at 0.708s. Cognee's Rust engine returns in 0.23-0.27s on documents up to War and Peace.
How should retrieval latency be measured (p50, p95, p99, cold vs warm)?
Report p50, p95, and p99 separately, and distinguish cold from warm. Serverless vector stores pay for compute spin-up and object-storage fetches on the first query after idle; reserved-capacity tiers hide this, which is why the two should never be blended into one number. Cognee records the same breakdown through OpenTelemetry: the memory.operation.duration histogram captures ms-level timing for add(), cognify(), search(), and forget(), and the memory.query.type attribute lets operators split latency per search strategy rather than reporting a single blended average.
How does Cognee's Rust core affect cold start and on-device retrieval latency?
The Rust engine is what makes Cognee viable inside short-lived serverless runtimes and on consumer hardware. The enterprise page reports recall at 0.24s via cognee.rust versus 1.77s on the Python SDK, with a cold start small enough to fit inside a Lambda or Workers invocation. On a Samsung Galaxy S24 Ultra running Alice in Wonderland fully on-device, the complete cognee-rs pipeline finished in 139.12s, and once memory was built, chunk and summary search stayed in the low-single-second range (around 1s). Rust removes most of the Python-side overhead that dominates cold paths.
Why aren't Mem0, Zep, and Cognee latency numbers directly comparable?
They measure different spans. A third-party head-to-head notes that Zep publishes retrieval-only numbers while Mem0 reports a figure that holds at or under 1.1s across every benchmark, including BEAM at 10M tokens, meaning Mem0's number includes more of the end-to-end path. Independent coverage also warns that latency columns across memory vendors use different reader models, judges, and retrieval depths. Treat the published figures as useful anchors for the vendor's own workload, then re-run the same query mix against each system before drawing conclusions for a specific application.
Does graph traversal always add latency compared to pure vector search?
Not automatically, but it often does when the traversal is paired with LLM calls at query time. The graph-enhanced Mem0g variant improves accuracy over base Mem0 while raising total p95 from 1.440s to 2.590s on LOCOMO, per the Mem0 paper. Graphiti, by contrast, reaches P95 retrieval latency of 300ms by combining semantic embeddings, BM25 keyword search, and direct graph traversal with no LLM calls during retrieval. Cognee follows the same principle: graph-aware search strategies are measured through OpenTelemetry so the cost of each traversal depth is visible per request.
Which memory layer can be benchmarked on your own data before committing?
Cognee ships a self-contained evaluation framework at cognee/eval_framework/ that benchmarks retrieval quality on standard multi-hop QA datasets and compares search strategies through an interactive HTML dashboard, with no third-party evaluation service required. The same framework can be pointed at private corpora, and the OpenTelemetry histograms described above capture p50/p95/p99 for every search() call during the run. The combination means retrieval accuracy and latency can be measured on the exact dataset and query mix a workload will see in production, rather than relying on vendor-chosen benchmarks.
Related reading





