AI Agent Memory Architecture: Components, Diagram and Design Choices
An AI agent memory architecture is the arrangement of stores, pipelines and interfaces that lets an agent remember things across turns, sessions and tasks instead of starting every call from zero. The model stays stateless; the architecture around it decides what gets written, where it lives, how it is retrieved and when it is forgotten. This guide is for engineers who need a reference AI agent memory architecture diagram and an explanation of each box and arrow: memory types, write and read paths, storage trade-offs, multi-agent sharing, and how Mem0, Graphiti, Letta and Cognee map onto the same picture.
What is an AI agent memory architecture?
An AI agent memory architecture is a system design, not a single database. It answers five questions: what the agent keeps in its context window, what it persists outside it, how raw interaction becomes stored memory, how stored memory gets back into the prompt, and how outcomes change what is stored. For the motivation, see AI agent memory and the longer agent memory guide.
Memory architecture vs the context window. The context window is working memory: whatever the model can see on this call, volatile and rebuilt on every request. A memory architecture exists because the window cannot hold a year of conversations and every tool result; it decides which slice of persistent memory gets loaded for a given turn. The LLM memory explainer covers the context-window side.
Memory architecture vs RAG. Retrieval-augmented generation reads from a corpus the agent never writes to. Agent memory is read-write: the agent, or a pipeline acting for it, creates records, updates them when facts change and deletes them. RAG is usually one retriever inside the read path; a memory architecture also needs a write path, conflict handling, temporal validity and forgetting. See GraphRAG vs RAG.
AI agent memory architecture diagram
The agent runtime sits in the center, the three long-term stores along the bottom, the write path on the left, the read path on the right and the feedback loop around the outside.
User / environment (top left). Messages, documents or events arrive. This is the only source of new information.
Agent / LLM runtime (center). The runtime plans, calls tools and decides what to remember. Inside it is working memory, the context window, with four slots: system prompt, recent turns, tool results and injected memories. The fourth slot is what the rest of the architecture feeds, and its size is the budget everything else competes for.
Tools / MCP interface (top right). Memory is exposed as tools: remember, recall, forget. Whether native function calls or a Model Context Protocol server, memory becomes an explicit action rather than hidden middleware. See the MCP memory guide.
Write path (left column). Extract entities, facts and events; dedupe and resolve them against what is stored; store with temporal validity. The note under step three, "conflicts: supersede, keep history", separates a memory store from a cache. One arrow leaves into the episodic log, one into semantic memory.
Long-term memory (bottom band). Episodic memory holds the message log, tool call traces and session summaries; the dashed arrow up to the runtime is the session log written each turn and replayed on resume. Semantic memory holds facts, entities and relationships with validity windows, split between a vector index (embeddings of chunks and entity summaries) and a knowledge graph (typed nodes and edges with valid_from / valid_to); the two-way arrow means each indexes the other. Procedural memory holds instruction files, skills and tools, and retrieval weights learned from feedback.
Read path (right column). Rewrite, embed and classify the query; retrieve from vector, graph and keyword indexes; rank by recency, relevance, validity and weight; trim to budget and inject. The dashed "query" arrow shows retrieval is triggered by the current turn.
Feedback loop (outer edges). Task outcomes flow from the tools interface to the feedback box and back into the stores as weight updates: a memory that led to a correct answer is reinforced, one that led to a failure is down-weighted or corrected. Without this loop the architecture can only accumulate.
The four memory types and where each lives
The four-way split comes from the CoALA paper (Sumers, Yao, Narasimhan and Griffiths, 2023), which proposes working memory plus episodic, semantic and procedural long-term stores as the frame for language agents. MemGPT (Packer et al., 2023) reached a similar layering from the operating-systems side: a bounded main context, recall storage for past messages and archival storage for everything else, with the model paging between them through function calls. The CoALA explainer covers the decision cycle.
| Memory type | What it holds | Lifetime | Typical storage | Written by |
|---|---|---|---|---|
| Working | System prompt, recent turns, tool results, injected memories | One model call | The prompt itself | Harness, every turn |
| Episodic | Message log, tool traces, session summaries | Days to months, then summarized | Relational or document store | Harness, automatically |
| Semantic | Facts, entities, relationships, with validity windows | Until superseded or forgotten | Vector index plus knowledge graph | Write path after extraction |
| Procedural | Instruction files, skills, tools, retrieval weights | Until revised | Repo files, skills registry, policy tables | Humans, plus feedback loop |
Episodic memory is cheap to write and expensive to read, so most architectures summarize episodes into semantic facts. Semantic memory is where the hard engineering is, and it benefits most from a graph: "who owns this service" and "what was true in March" are relationship and time queries, not similarity queries. Procedural memory is the least automated: CLAUDE.md, AGENTS.md and .cursor/rules are written by humans, and the feedback loop is what lets the system write some of it.
The write path: how memories get created
The top row of the lifecycle diagram expands the write path into five stages.
Raw input. Turns, documents and tool output. A 40-turn debugging session may yield two durable facts, so the first decision is the trigger: every turn, end of session, explicit remember calls, or a background job.
Extract. An LLM or smaller extractor pulls entities, facts and events into structured output: subject, predicate, object, source, timestamp. The Generative Agents paper (Park et al., 2023) attached an importance score here so routine observations are stored but rarely retrieved.
Resolve. Extracted entities are deduplicated and linked to known nodes; "Ana", "Ana Petrović" and "the new PM" may be one person. Resolution combines embedding similarity, exact match on identifiers and an LLM tie-breaker.
Reconcile. New facts are compared with existing ones. If the deployment target changed from Heroku to Fly.io, the options are overwrite, append a contradiction, or supersede with time. The diagram chooses the third: the old fact gets a valid_to, the new one a valid_from, and both stay queryable. See temporal knowledge graph.
Persist. Facts go to the graph, their text to the vector index, and the episode stays in the log with a pointer back. Atomicity matters: a node without its embedding is invisible to similarity search, and an embedding without its node cannot be traversed.
Two operations run on the stores rather than inline. Consolidation compresses old episodes into summaries and merges related facts; the A-MEM paper (Xu et al., 2025) describes a Zettelkasten-style variant where each new note updates related notes. Forgetting removes facts for correctness, decay or compliance. Superseded facts are hidden from default retrieval but kept for audit; deleted facts must leave graph, vector index and log together, or an orphaned embedding can resurface deleted content.
The read path: how memories get retrieved
The bottom row of the lifecycle diagram runs on every turn that needs context, which makes it the latency-critical half.
Query. The current task or turn, often with the last few turns attached for coreference.
Rewrite and embed. The query is rewritten, embedded and classified. Classification sets the route: a question about a person goes to graph lookup, "what did we decide last week" to episodic search with a time filter, "how do I deploy" to skills. Sending everything to pure vector search is the most common reason agent memory feels random.
Hybrid retrieve. Vector search finds semantically similar chunks and entity summaries. BM25 finds exact identifiers, error codes and names that embeddings blur. Graph hops expand from matched entities along typed edges, which is how a query about a service finds its owner and last incident without those words in the query. Results are merged, commonly with reciprocal rank fusion.
Rank and budget. Candidates are scored on relevance, recency, validity (is the fact current as of the query time) and weight (feedback-learned importance). Generative Agents introduced the recency-importance-relevance combination; validity is what makes temporal memory work. The list is trimmed to a fixed share of the context window and deduplicated.
Inject into context. Surviving memories enter the prompt with sources, so the model can cite, the user can verify and the feedback loop can attribute an outcome to a specific memory.
An interactive turn usually needs the whole read path inside a few hundred milliseconds, which rules out LLM reranking on every call; a long-running coding agent can afford deeper retrieval. See retrieval latency at scale.
Storage choices: vector store, knowledge graph, relational, or all three
The architecture needs similarity search, relationship traversal, and transactional records with timestamps. The question is how many systems you use to get them.
| Option | Strengths | Weaknesses | Use when |
|---|---|---|---|
| Vector store only | Simplest write path; good semantic recall; mature managed options | No relationships or temporal validity; duplicates accumulate; multi-hop questions fail | Prototypes, single-user assistants |
| Knowledge graph only | Entities, relations, provenance and time are native; explainable | Needs an extraction pipeline; fuzzy queries weak without embeddings | Relationship- and history-heavy domains |
| Relational / document only | Transactions, retention and multi-tenancy are solved | Neither similarity nor traversal is natural | Episodic log, session state, audit trail |
| All three, separate systems | Each store does what it is best at; independent scaling | Three systems to keep consistent; deletes must succeed in all | Large deployments with infrastructure teams |
| All three, one engine | One transaction, one permission model, one backup | Bound by the engine's weakest store; fewer hosting options | Full architecture without three databases |
A practical middle path is Postgres with pgvector for embeddings, a graph layer over relational tables and plain tables for the episodic log: not the fastest at every scale, but one transactional system. See graph on Postgres and vector databases vs graph databases.
Multi-agent and multi-tenant memory
The single-agent diagram extends to many agents and users by adding scope to every record and query.
Scopes. A memory belongs to a user, an agent, a project or an organization, encoded on each record and filtered before ranking: organization memory shared by everyone, project memory shared by the agents on it, agent memory private to one agent's procedures, user memory private to one person. See organization, agent and user memory.
Sharing between agents. Agents share memory through a common store with cross-scope read access, through handoff summaries, or through a shared procedural layer. The first gives the most continuity and the hardest consistency problems: two agents writing conflicting facts about one entity is the reconcile step, now with concurrency.
Isolation between tenants. A filter column is not enough. A nearest-neighbor query without a hard tenant filter can return another tenant's chunk, and a graph hop from a shared entity can cross into a private subgraph. Enforce scope at the storage layer (row-level security, per-tenant indexes or datasets) and make forget tenant-aware. See multi-tenant AI memory.
Permissions. Read, write, delete and share should be separate rights on a scope.
Reference architectures compared
Four open-source systems implement this architecture with different emphases. Descriptions are drawn from their repositories and papers; the alternatives posts (Mem0, Zep, Letta) cover product comparison.
Mem0 (github.com/mem0ai/mem0) centers on the write path. Its paper describes an extraction phase that turns recent turns plus a running summary into candidate memories, and an update phase in which an LLM decides per candidate whether to add, update, delete or ignore. Memory is scoped by user, agent and run; the default store is a vector database, with an optional graph variant.
Zep / Graphiti (github.com/getzep/graphiti) centers on semantic memory as a temporal knowledge graph. Episodes are ingested, entities and relations extracted, and each edge carries bi-temporal fields so a contradicting fact invalidates the old edge rather than deleting it. Retrieval combines semantic, BM25 and graph-distance search. Graphiti is the engine; Zep is the hosted product.
Letta / MemGPT (github.com/letta-ai/letta) centers on working memory. The agent holds editable memory blocks inside the context and uses tools to edit them, search recall memory (past messages) and search archival memory (a vector store). The model, not a pipeline, decides what to write, which makes the write path explicit and auditable. Blocks can be attached to multiple agents.
Cognee (github.com/topoteretes/cognee) centers on the combination: one pipeline that extracts into graph and vector index together, temporal validity, a feedback operation that reweights memories, and the option to run all stores on one Postgres instance. It is described below.
These are not mutually exclusive; a production system might use Letta-style blocks for working memory, a Graphiti-style temporal graph for semantic memory and a Mem0-style extraction policy.
Design checklist for your own agent memory architecture
Working through the diagram top to bottom:
- Working memory budget. How many tokens are reserved for injected memories, and what is evicted first?
- Write trigger. Every turn, end of session, explicit
remembercalls, or a background job? - Extraction schema and entity resolution. Free-form facts, triples, or a typed ontology; what makes two mentions the same entity?
- Conflict policy. Overwrite, append or supersede with validity windows? Only the third answers "what did we believe in March".
- Episodic retention. How long raw logs are kept and when they are summarized.
- Retrieval routing and ranking. Which query types go to vector, lexical, graph or episodic search; which signals rank them; are weights learned from feedback?
- Source attribution. Does every injected memory carry provenance?
- Forgetting. Can one
forgetcall remove a fact from graph, vector index and log together, within tenant scope? - Scopes, permissions and interface. Which scopes exist, are read, write, delete and share separate rights, and is memory exposed as native tools, MCP, or both?
- Evaluation. Task success with and without memory, retrieval precision, p95 latency. See AI memory benchmarks.
How Cognee implements this architecture
Cognee is an open-source memory engine (Apache 2.0, docs.cognee.ai) that implements the diagram as one system.
Write path as an ECL pipeline. Extract ingests raw input (text, files, turns, tool output). Cognify runs the middle of the write path: chunking, entity and relation extraction, resolution against the existing graph, optional ontology alignment. Load persists into a knowledge graph, a vector index and relational metadata in one operation, so a fact is never in one store and missing from another. Temporal mode keeps superseded facts with validity windows.
Graph, vector and relational in one engine. Backends are pluggable: Neo4j, Kuzu or FalkorDB for the graph, LanceDB or pgvector for vectors, Postgres for relational data. The single-Postgres option runs all three on one instance, the "all three, one engine" row above.
Four verbs. The Cognee 1.0 API exposes remember (ingest and cognify), recall (hybrid retrieval with ranking and a token budget), improve (the feedback loop) and forget (remove a fact and its derived artifacts), mapping to the write path, read path, feedback loop and forgetting operation in the diagram.
Tools / MCP interface. The MCP server exposes remember, recall and forget over stdio, Streamable HTTP and SSE, so Claude Code, Cursor, Claude Desktop, Codex, Cline and Continue share one memory. It ships as cognee/cognee-mcp; for Claude Code:
Scopes and integrations. Memory is organized into datasets with read, write, delete and share permissions. Integrations cover the Claude Agent SDK, OpenAI Agents SDK, LangGraph, Google ADK, CrewAI and n8n; a Mem0Source adapter imports existing Mem0 memories. Self-hosting is free; Cognee Cloud has a free tier and a Standard plan priced per million tokens processed, as of October 2026 (see pricing).
Next steps. Starting from a single vector store, the highest-leverage additions in order are temporal validity on facts, an episodic log with summaries, hybrid retrieval with routing, and a forget that reaches every store. If you would rather not assemble those pieces, start with the Cognee repository and the memory agent explainer.
FAQ
What are the components of an AI agent memory architecture?
The core components are the agent runtime with its working memory (the context window), three long-term stores (episodic for logs and sessions, semantic for facts and relationships, procedural for rules and skills), a write path that extracts, resolves and reconciles facts with temporal validity, a read path that retrieves, ranks and injects memories within a token budget, a tools or MCP interface, and a feedback loop that reweights memories from outcomes.
What is the difference between agent memory and RAG?
RAG reads from a corpus the agent does not write to; agent memory is read-write. A memory architecture adds a write path (extraction, entity resolution, conflict handling), temporal validity so superseded facts are kept with validity windows, and a forget operation. In practice RAG is one retriever inside the read path, alongside graph traversal, lexical search and episodic lookup.
Should agent memory use a vector database or a knowledge graph?
Use both if the agent must answer relationship or time questions; use a vector store alone only for prototypes and single-user assistants whose memory is mostly "recall similar past text". Vector search handles fuzzy semantic recall; a knowledge graph handles entities, typed relationships, provenance and validity windows. Most production architectures pair them, as separate systems or in one engine.
How do agents decide what to remember?
Through a write policy that is model-driven (the agent calls a remember tool when it judges a fact durable, as in MemGPT and Letta), pipeline-driven (an extractor runs over turns and an LLM decides whether each candidate is added, updated, deleted or ignored, as in Mem0), or both. Good policies keep facts that outlive the session and are not already stored, and attach an importance score used in ranking.
What is procedural memory in AI agents?
Procedural memory is the store of how the agent does things rather than what it knows: instruction files such as CLAUDE.md or AGENTS.md, skills and tool definitions, learned workflows, and retrieval weights adjusted from feedback. In the CoALA framework it is one of three long-term stores alongside episodic and semantic memory. It is loaded at session start, written mostly by humans, and increasingly updated by feedback loops.
