LLM Memory Explained: Types, Architectures, and How to Add It

Large language models are stateless: every API call starts fresh, with no knowledge of prior conversations unless that information is explicitly stored and re-supplied. LLM memory is the external system that fixes this, giving applications the ability to recall user preferences, learn from feedback, and maintain context across sessions. This guide covers what LLM memory is and why it matters, the main types (short-term, long-term, episodic, semantic, procedural), the core architectures from conversation buffers to vector RAG and knowledge graphs, criteria for evaluating memory systems, a framework comparison, and how to add memory to an application with Cognee, an open-source memory engine referenced throughout.

What Is LLM Memory?

LLM memory refers to the external storage and retrieval systems that let a stateless language model behave as if it remembers across calls and sessions. Every call to an LLM starts fresh with no knowledge of prior conversations unless that information is explicitly stored and re-supplied in subsequent calls. Without this layer, agents cannot learn from feedback, recall user preferences, or maintain context between sessions. The context window is the bounded per-call input the model sees at inference time, while memory is the persistent store that selectively loads the right facts back into that window when needed.

Cognee is an open-source Apache-2.0 Python package (pip install cognee) built as a memory engine for LLM applications. It constructs a knowledge graph plus a vector index from ingested documents, chunks, nodes, and summaries, then serves agent queries through a memory-native API with four operations: remember, recall, forget, and improve. This design treats memory as a first-class system with structured representations layered on top of embeddings, rather than a side effect of stuffing more tokens into the prompt.

Why LLM Memory Matters in 2025

When agents run across sessions, stateless LLM calls become a hard ceiling: every request starts from scratch unless prior turns, preferences, and feedback are explicitly re-supplied, so agents cannot learn, personalize, or hold context over time. As teams move from one-shot chat to long-running assistants and knowledge-heavy workflows-think research pipelines like Bayer processing 10,000 scientific papers into reusable research memory for hypothesis generation-a dedicated memory layer becomes core infrastructure. Cognee fills that role as a graph-plus-vector memory engine with 20 search types, including TEMPORAL and GRAPH_COMPLETION_COT, used in production agent workflows.

Types of LLM Memory

Short-Term (In-Context / Working) Memory

Short-term memory is the active window loaded into the model on each call. It holds the current prompt, recent turns, retrieved snippets, tool outputs, and system instructions, everything the model can attend to while generating the next token. In the CoALA framework (Princeton, 2023), this corresponds to in-context working memory: the scratchpad that disappears when the request ends. It is fast and precise but bounded by the context window, which forces trade-offs between how much history, how much retrieved knowledge, and how much reasoning space fits in a single call.

Long-Term Memory: Episodic, Semantic, Procedural

Long-term memory in cognitive science splits into three types that CoALA carries into language agents: episodic memory stores raw records of past interactions and events, semantic memory stores concepts and factual knowledge, and procedural memory encodes how to perform actions and skills. CoALA further divides procedural memory into two forms specific to language agents: implicit knowledge baked into the LLM's weights during pretraining, and explicit knowledge written directly in the agent's code, prompts, and tool definitions. Together with in-context memory, these four categories cover what a durable LLM memory system needs to represent.

Session Memory vs Permanent Memory

Cognee maps those categories onto a two-layer runtime split. Session memory acts as short-term working memory: for each request it loads the relevant embeddings and graph fragments into runtime context so the model sees only what matters for the current task. Permanent memory holds long-term knowledge artifacts, facts extracted from documents, conversations, and code, that are continuously cross-connected inside the knowledge graph. New sessions pull from permanent memory on demand, and new observations are written back, so episodic, semantic, and procedural knowledge accumulate without bloating any single prompt.

LLM Memory Architectures: Buffers, Vector RAG, Graph, and Hybrid

Conversation Buffers and Summaries

Buffer memory keeps raw dialogue turns inside the prompt, and summary memory compresses older turns into a running recap that is prepended on each call. Both approaches are easy to implement with a few lines of code and work well for short sessions or single-user chat. The ceiling is low, however: raw buffers hit the context window quickly, and summaries lose specific facts, dates, and entity references as they compress. Neither approach supports retrieval across sessions, cross-user knowledge, or structured reasoning over accumulated information from documents and prior interactions.

Vector Stores with RAG

Retrieval-augmented generation embeds text chunks into a vector store and pulls back the top-k nearest neighbors by cosine similarity at query time. It scales to large corpora and handles unstructured documents gracefully, but similarity search has a known blind spot. In one evaluation, contradicting facts scored a mean cosine similarity of 0.812 against an original fact while verified duplicates scored 0.800, meaning no threshold can cleanly separate the two classes. Pure vector RAG therefore tends to surface stale or conflicting passages alongside correct ones, with no built-in way to arbitrate between them.

Knowledge-Graph Memory

Knowledge-graph memory stores typed entities and the relationships between them as nodes and edges, so a retriever can traverse multiple hops to answer questions that depend on connections rather than lexical overlap. Grounding the graph in an ontology validates extracted entities against a formal schema, which keeps the structure consistent as new documents are ingested. This produces explainable retrieval paths, supports multi-hop reasoning across people, projects, and events, and makes it practical to detect when a new fact conflicts with existing nodes rather than silently duplicating them in an embedding space.

Hybrid Graph + Vector Architectures

Hybrid architectures combine graph edges with vector embeddings in a single store, so retrieval can start from semantic similarity and then traverse typed relationships, or vice versa. Cognee is one example: it maintains a vector database for embeddings of chunks, nodes, and summaries, and layers a knowledge graph on top using an Extract, Cognify, Load pipeline grounded in ontologies. The unified store exposes 20 search types, including classic RAG_COMPLETION, GRAPH_COMPLETION for relational lookups, GRAPH_COMPLETION_COT for chain-of-thought multi-hop traversal, TRIPLET_COMPLETION, and a TEMPORAL mode for time-aware queries.

Common Challenges in LLM Memory and How Memory Systems Solve Them

Memory implementations often look convincing in prototypes and then break under production load. Context overflows, embeddings drift toward near-duplicates, retrieval misses the connections that actually matter, and time-sensitive answers quietly go wrong. The result is agents that forget, contradict themselves, or confidently return outdated information. Cognee treats these as first-class concerns rather than retrieval tuning problems, combining a knowledge graph with vector embeddings so that structure, provenance, and time are preserved alongside semantic similarity, and so that known failure modes can be detected and traced at ingestion rather than discovered by users.

Key Problems Encountered

• Statelessness across calls: Every LLM call starts fresh with no knowledge of prior conversations unless information is explicitly stored and re-supplied, which means agents cannot learn from feedback, recall user preferences, or maintain context across sessions.
• Contradictions and stale facts in accumulated memory: As new documents arrive, they often disagree with earlier ones, and cosine similarity cannot cleanly separate contradictions from duplicates, in one evaluation contradictions scored 0.812 mean similarity to the original fact while duplicates scored 0.800, so no threshold reliably distinguishes the two.
• Weak multi-hop recall from pure vector similarity: Vector search returns passages that look alike, but questions that require chaining several facts together, person to project to organization to date, fall outside what nearest-neighbor retrieval can resolve.
• Lack of temporal awareness for time-scoped questions: Overwriting or deduplicating records destroys the history needed to answer questions like "What was John's role in 2025?" and leaves the system unable to distinguish current state from past state.

Memory systems address these failures by giving facts structure, identity, and a timeline instead of leaving them as loose chunks. Cognee handles statelessness by persisting ingested content into a knowledge graph that any session can query. Its opt-in contradiction detection runs during the Cognify stage: when a second ingestion disagrees with an existing fact, Cognee writes a contradicts edge carrying both fact texts, the reason, and a confidence score, so conflicts are visible rather than averaged away. Temporal Mode keeps superseded records in place so time-scoped questions remain answerable, and a unified graph-plus-vector store supports 20 search types, including chain-of-thought graph traversal for multi-hop queries that pure similarity search cannot resolve.

What to Look for in an LLM Memory System

Choosing an LLM memory system comes down to a handful of evaluation criteria that compound over time. Recall quality determines whether the right facts surface for multi-hop questions. Latency and cost shape how often memory can be queried per turn and whether the economics work at scale. Staleness and contradiction handling decide what happens when sources disagree or facts change. Deployment flexibility matters because production systems rarely tolerate a single vendor's stack, and governance, encryption, residency, auditability, is non-negotiable for regulated workloads. A memory layer that scores well on one axis but fails on another will quietly break downstream agents.

Must-Have Capabilities

• Combined graph and vector retrieval so semantic similarity and explicit relationships both inform answers.
• Temporal or versioned memory that preserves historical state and keeps time-scoped questions answerable.
• Contradiction handling that flags conflicting facts rather than silently overwriting them.
• Pluggable storage backends so existing graph and vector infrastructure can be reused.
• Ontology grounding to validate extracted entities against a formal schema and keep the graph consistent.
• Compliant deployment options including self-hosted, air-gapped, and bring-your-own-cloud.

Cognee maps to each of these directly. Its backends are pluggable across Neo4j, Amazon Neptune, PostgreSQL, Kuzu, Redis, LanceDB, and pgvector, so teams reuse the stores they already run. The Extract, Cognify, Load (ECL) pipeline grounds memory in ontologies, validating entities against a formal schema as documents accumulate. Temporal Mode retains superseded records so queries like "What was John's role in 2025?" still resolve, and opt-in contradiction detection during Cognify writes a contradicts edge with both fact texts, a reason, and a confidence score. Cognee is GDPR-compliant, encrypted at rest and in transit, and supports air-gapped and BYOC deployments. On the recall side, Markovic et al., 2025 reported consistent correctness and F1 gains across HotPotQA, TwoWikiMultiHop, and MuSiQue when tuning the interface between the knowledge graph and the LLM.

How Teams Add Memory to LLM Applications with Cognee

Engineering teams and research groups adopt Cognee when a chatbot buffer or plain vector store stops scaling to the shape of their data. A representative case is Bayer, which uses Cognee to process 10,000 scientific papers into research memory that supports hypothesis generation across projects. The pattern is the same for smaller teams: pipe documents, chats, or code into Cognee, let it build a knowledge graph plus vector index, then query that memory from an agent, a backend service, or a coding assistant. The strategies below show which Cognee features map to each step of that workflow.

• Ingest documents and conversations: the cognify six-stage pipeline (classify, permission check, chunk, LLM entity and relation extraction, summarize, embed and commit edges to the graph), which only reprocesses new or updated files on re-runs.
• Answer multi-hop questions: the GRAPH_COMPLETION retrieval mode, one of 20 search types that also includes RAG_COMPLETION, GRAPH_COMPLETION_COT for chain-of-thought traversals, and TRIPLET_COMPLETION.
• Handle time-scoped queries: TEMPORAL mode, which retains superseded records so questions like "What was John's role in 2025?" remain answerable.
• Catch conflicting facts: opt-in contradiction detection during Cognify, which writes a contradicts edge carrying both fact texts, a reason, and a confidence score when ingested documents disagree.
• Give coding assistants persistent memory: the Cognee MCP server, launched locally with one Docker command and one mcp add or JSON config per client for Cursor, Claude Code, Cline, and other MCP clients sharing the same endpoint.
• Reuse existing infrastructure: pluggable backends across Neo4j, Amazon Neptune, PostgreSQL, Kuzu, Redis, LanceDB, and pgvector, so graph and vector storage can run on databases already in production.

What separates Cognee from buffer libraries, pure vector RAG tools, and single-store memory services is the combination in one package: a repeatable ingestion pipeline, a graph-plus-vector index with temporal and contradiction semantics, 20 search types rather than a single similarity call, an MCP server for developer tools, and backend portability. Teams get knowledge-graph memory without binding themselves to a specific database vendor or a bespoke pipeline, and the same memory layer serves agents, applications, and IDE assistants from one source of truth.

Best Practices for Implementing LLM Memory

Strong LLM memory systems come from a few consistent engineering choices rather than a single trick. The practices below reflect Cognee's documented approach to building memory for production LLM applications: separating what the model needs right now from what the system needs to remember forever, grounding extraction in a schema, combining graph and vector retrieval, and treating conflicting or outdated facts as first-class citizens instead of edge cases. Teams that follow these guidelines tend to get more accurate answers, cleaner audit trails, and infrastructure that fits the stack they already run.

• Separate session memory from permanent memory: Keep short-term working context, such as the relevant embeddings and graph fragments loaded for the current turn, distinct from long-term knowledge artifacts that stay continuously cross-connected in the graph. This split makes it easier to control latency, scope, and retention.
• Ground memory in an ontology: Validate extracted entities against a formal schema using an Extract, Cognify, Load (ECL) pipeline so the knowledge graph stays consistent as documents accumulate instead of drifting into inconsistent labels and duplicate nodes.
• Prefer graph plus vector retrieval over pure similarity: Store graph edges for relational traversal alongside vector embeddings for semantic similarity, which supports retrieval modes ranging from classic RAG to chain-of-thought graph traversal for multi-hop questions.
• Turn on contradiction detection during ingestion: Enable the opt-in check at the Cognify stage so that when two documents disagree, the second ingestion records a contradicts edge carrying both fact texts, the reason, and a confidence score for later review.
• Keep superseded records in place with temporal memory: Use a Temporal Mode that retains historical information so time-scoped questions like "What was John's role in 2025?" remain answerable, rather than overwriting values when they change.
• Choose storage backends that match existing infrastructure: Pick from pluggable options such as Neo4j, Amazon Neptune, PostgreSQL, Kuzu, Redis, LanceDB, and pgvector so memory fits into the stack already in production instead of forcing a migration.

Advantages of a Dedicated LLM Memory System

A dedicated memory layer changes what an LLM application can promise. Instead of hoping the context window carries enough signal, teams get measurable gains on multi-hop recall, time-scoped accuracy on questions that depend on when a fact was true, and auditability for every answer. A purpose-built system also reuses existing storage infrastructure rather than forcing a new database, and it fits regulated deployments where data residency and encryption are non-negotiable. The result is fewer hallucinations on complex questions, clearer provenance, and a path to production that security and compliance teams can approve.

• Higher multi-hop correctness: Markovic et al., 2025 reports consistent gains on correctness and F1 across HotPotQA, TwoWikiMultiHop, and MuSiQue when the interface between a knowledge graph and the LLM is tuned, and Cognee's graph-completion retriever placed ahead of Mem0, Graphiti, and LightRAG on a HotPotQA subset.
• Time-scoped answers via Temporal Mode: Historical information is retained so queries like "What was John's role in 2025?" remain answerable, and superseded records stay in place instead of being overwritten.
• Traceable contradictions: When two documents disagree, an opt-in check during the Cognify stage records a contradicts edge carrying both fact texts, the reason, and a confidence score, giving reviewers a direct audit trail.
• Infrastructure reuse via pluggable backends: Cognee runs on Neo4j, Amazon Neptune, PostgreSQL, Kuzu, Redis, LanceDB, pgvector, and others, so existing storage choices carry forward.
• Regulated-industry readiness: Cognee is GDPR-compliant, encrypted at rest and in transit, and supports air-gapped and bring-your-own-cloud deployments for teams in finance, healthcare, and the public sector.

How Cognee Simplifies Adding Memory to Your LLM Application

Cognee ships as an Apache-2.0 Python package installed with pip install cognee, exposing a memory-native API built around four verbs: remember, recall, forget, and improve. Under the hood, the six-stage cognify pipeline classifies documents, checks permissions, extracts chunks, uses an LLM to pull out entities and relationships, generates summaries, then embeds content into the vector store and commits edges to the graph. Only new or updated files are reprocessed on subsequent runs. For IDE and agent integration, the Cognee MCP server runs with a single Docker command and one mcp add or JSON configuration per client, letting Cursor, Claude Code, and Cline share the same endpoint.

Teams can self-host the open-source engine under Apache-2.0 at no cost, or offload operations to Cognee Cloud, which is usage-based at $1.00 per 1M tokens processed, plus $5 per additional workspace. A free tier includes 1M tokens and one workspace, which is enough for prototypes and small internal agents. For regulated workloads, the Enterprise tier adds dedicated support, SLAs, and bring-your-own-cloud deployment, so memory data stays inside an organization's own VPC. Full plan details are available on the Cognee pricing page, and the engine itself is open-sourced by Cognee.

Key Takeaways and How to Get Started with Cognee

LLM memory is the external system that compensates for stateless model calls. A useful taxonomy, drawn from the CoALA framework, breaks it into in-context, episodic, semantic, and procedural memory. Architectures range from simple conversation buffers to vector RAG, knowledge-graph memory, and hybrid graph-plus-vector designs, and should be evaluated on recall quality, latency, cost, staleness, and deployment fit. To get started, self-host the open-source Apache-2.0 Python package or try Cognee Cloud at $1.00 per 1M tokens processed, plus $5 per additional workspace, with 1M free tokens and one workspace on the free tier. See the Cognee platform for details.

Related guides

• AI Agent Memory: What It Is and How It Works
• What Is AI Memory?
• LangChain Memory in 2026: Built-In Options, LangMem, and Persistent Alternatives
• MCP Memory: What It Is and How to Give Any MCP Client Persistent Memory
• Best Mem0 Alternatives in 2026
• The Token Cost of Persistent AI Memory

Frequently Asked Questions

What is LLM memory and how is it different from the context window?+

LLM memory is the persistent store of facts, past interactions, and skills that an application supplies back to a model across calls. Every call to an LLM is a new start with no knowledge of prior conversations unless that information is explicitly stored and re-supplied. The context window is per-call: it is the token budget the model sees in a single request. Memory lives outside the model and is selectively loaded into the window at inference time. Cognee implements this split with session memory for working context and permanent memory for long-term knowledge.

What are the main types of LLM memory?+

The CoALA framework from Princeton (2023) formalises four types: in-context (working memory the model sees inside the prompt), episodic (past interactions and events), semantic (concepts and factual knowledge), and procedural (rules and skills). The long-term triad in cognitive science mirrors three of these: semantic for facts, procedural for how to perform actions, and episodic for raw records of experiences. Cognee organises memory along a session versus permanent split, where session memory loads relevant embeddings and graph fragments into runtime context, and permanent memory stores cross-connected long-term artifacts.

How does Cognee implement LLM memory for agents and applications?+

Cognee builds a knowledge graph plus a vector index from ingested documents, conversations, and code, then serves agent queries through a memory-native API with four verbs: remember, recall, forget, and improve. The core cognify operation runs a six-stage pipeline that classifies documents, checks permissions, extracts chunks, uses an LLM to extract entities and relationships, generates summaries, then embeds everything and commits edges to the graph. Cognee exposes 20 search types, including GRAPH_COMPLETION, RAG_COMPLETION, GRAPH_COMPLETION_COT for multi-hop chain-of-thought traversals, TRIPLET_COMPLETION, and a TEMPORAL mode for time-aware agent workflows.

How much does Cognee cost for LLM memory in production?+

Cognee Cloud is usage-based at $1.00 per 1M tokens processed, plus $5 per additional workspace. A free tier includes 1M tokens and one workspace, which is enough to prototype a small agent or evaluate retrieval modes against a corpus. The Enterprise tier adds dedicated support, SLAs, and bring-your-own-cloud for teams with compliance or data-residency requirements. Teams that prefer to run everything themselves can self-host the open-source Cognee engine for free under the Apache-2.0 license, reusing existing graph and vector backends rather than paying for managed infrastructure.

How do Cognee, Mem0, Zep, Letta, Graphiti, and LangMem compare?+

Mem0 and Letta (formerly MemGPT) are block-style frameworks that store conversation blocks and retrieve them by similarity; integration is simple but the ceiling on multi-hop reasoning is narrower. Zep uses Graphiti as its underlying engine to build temporally ordered knowledge graphs from conversation data. Graphiti is a bi-temporal graph library for conversation history. LangMem is vector-primary and tightly coupled to LangChain. Cognee is graph-plus-vector with 20 search types, and its graph-completion retriever placed ahead on correctness against Mem0, Graphiti, and LightRAG on a HotPotQA subset (Markovic et al., 2025).

How should teams evaluate an LLM memory system?+

Teams should score candidate systems on recall quality using multi-hop benchmarks such as HotPotQA, TwoWikiMultiHop, and MuSiQue, where Cognee published consistent gains on correctness and F1 (Markovic et al., 2025). Beyond recall, measure latency per query, cost per 1M tokens processed, and how the system handles staleness and contradictions over time. Deployment and compliance matter in production: look for GDPR compliance, encryption at rest and in transit, pluggable backends such as Neo4j, Neptune, Postgres, Kuzu, LanceDB, or pgvector, and support for air-gapped or bring-your-own-cloud installs, all of which Cognee provides.