Hermes Agent Memory: How It Works and How to Improve It

Hermes Agent memory is the persistence layer that lets Nous Research's self-improving agent carry context across sessions: a built-in learning loop that creates skills from experience, refines them during use, searches past conversations, and gradually builds a model of the user. This guide is for engineers operating or extending a self-hosted Hermes install. It walks through how the built-in loop works in practice, the MEMORY.md file for environment facts, the USER.md file for user profile, and the skills directory under ~/.hermes/skills/, where that file-based approach starts to strain, and how to extend it with an external provider or MCP server such as Cognee, a memory engine that builds a searchable knowledge graph from agent data.

Core Components That Make Hermes Agent Memory Work

In practical terms, Hermes Agent memory is the set of files, stores, and tools Nous Research's agent uses to remember environment facts, user preferences, and reusable procedures across sessions. The core components are: MEMORY.md in the profile directory for environment facts and lessons; USER.md for the user profile; a skills directory at ~/.hermes/skills/ holding on-demand procedure documents compatible with the agentskills.io standard and managed through the skill_manage tool; a holographic memory store backed by SQLite with FTS5 (SQLite's full-text search extension), with a default memory_char_limit of 2,200 characters (~800 tokens); and a pluggable external memory provider slot where exactly one provider is active at a time while MEMORY.md and USER.md keep running alongside it. Supported external providers include Honcho, OpenViking, Mem0, Hindsight, Holographic, RetainDB, ByteRover, and Supermemory. Cognee fits that external slot as a memory engine that builds a searchable knowledge graph from data (reachable via MCP, the Model Context Protocol, or a direct tool call).

How to Think About Hermes Memory in a Modern Agent Stack

Agents are now expected to carry context across sessions, surfaces, and tools, so persistent memory is table stakes rather than a nice-to-have. Hermes ships a minimum viable implementation: a MEMORY.md file for environment facts and lessons, a USER.md profile, a ~/.hermes/skills/ directory, and a SQLite plus FTS5 holographic store with a 2,200-character default per entry. That design is readable and easy to inspect, but file-based memory hits ceilings when context spans months, teams, or tool chains. A mature setup adds a searchable knowledge graph, time-stamped facts for period-constrained recall, and source references on every retrieved fact, the shape of memory practitioners like Cognee build for agents.

Common Challenges Teams Face With Hermes Agent Memory

Hermes agent memory works well for a single workstation and a handful of sessions, but friction appears once teams deepen usage: longer-running projects, more users, and more skills written to disk. The built-in holographic store is backed by SQLite with FTS5 and ships with a memory_char_limit of 2,200 characters (roughly 800 tokens), with persistent context split between MEMORY.md and USER.md in the profile directory. Those defaults are convenient, but they also set hard limits on what a Hermes agent can remember, how it finds that memory, and how teams reason about what has been stored.

Key Challenges and Failure Modes When Scaling Hermes Memory

• Memory budget forces deletions. With the file-based store, hitting the 2,200-character cap forces the agent to delete something it already knew before it can write something new. Important lessons quietly age out, and there is no graph structure to preserve the relationships between the evicted facts.
• Flat text search limits recall. FTS5 over MEMORY.md and USER.md is keyword-driven. It handles exact phrases well but struggles with paraphrase, cross-document reasoning, or questions that require joining several small facts into one answer.
• No source attribution for stored facts. Entries in MEMORY.md are lessons the agent wrote to itself. Operators cannot easily tell which conversation, file, or skill produced a given line, which complicates review, redaction, and debugging of bad memories.
• Fragmented memory across providers. Hermes exposes a pluggable external provider slot (Honcho, OpenViking, Mem0, Hindsight, Holographic, RetainDB, ByteRover, Supermemory), but only one external provider is active at a time while MEMORY.md and USER.md keep running alongside it. Teams end up with two parallel memory surfaces and no shared schema.

Mitigation starts with standardization: pick one canonical place for durable, queryable memory and treat the local files as a short-term cache. Next, make memory writes an explicit, owned step rather than an implicit side effect, so every stored fact has a known producer. When the 2,200-character budget is exhausted, reach for an external provider instead of accepting silent deletions. Cognee fits this role by building a knowledge graph from ingested data and using its GRAPH_COMPLETION mode to find relevant triplets via vector search and traverse the graph for structured context. Because every entity and relation carries a pointer back to the source document and chunk, recall returns facts with their references attached, restoring the attribution and cross-fact recall the built-in store cannot provide.

How to Define a Winning Strategy for Hermes Memory

A Hermes memory strategy should start from the workload, not from a tool shortlist. Hermes ships with a SQLite-backed holographic memory and a default 2,200-character cap (~800 tokens) per entry, which is enough for short-horizon tasks but forces the agent to delete prior knowledge once the cap is reached. The strategy question is where that limit stops being acceptable: multi-session user state, document-grounded answers, auditable decisions, or long-running research. Decide which of those matter, then pick capabilities that match. Treat the built-in store as scratch memory and plan an external layer for anything that must persist, be cited, or be queried structurally.

Must-Have Capabilities for a Scalable Hermes Memory Strategy

• Persistence beyond the 2,200-character cap. The external memory layer stores facts without evicting older entries to make room, so the agent can accumulate knowledge across sessions instead of overwriting it each time the default limit is hit.
• Structured retrieval that combines graph and vector. Retrieval uses vector similarity to locate relevant entities, then traverses typed relations to assemble multi-hop context, rather than returning a flat list of nearest-neighbor chunks.
• Source attribution on every recall. Each returned fact carries a pointer back to the ingested document and chunk it came from, so the agent and reviewers can verify where an answer originated.
• Time-aware facts. The store extracts events and timestamps so queries can be scoped to a period ("what did we decide last quarter") instead of returning the most semantically similar text regardless of when it was true.
• Self-hostable deployment. The memory layer runs inside the same trust boundary as Hermes, including air-gapped or bring-your-own-cloud environments, so sensitive context never leaves controlled infrastructure.

Cognee covers these requirements directly. Its default GRAPH_COMPLETION mode uses vector search as a hint to locate graph triplets and then traverses the knowledge graph to build structured context before answering, which addresses persistence and structured recall at the same time. Every entity and relation stored in the graph keeps a pointer to its source document and chunk, so recall returns facts alongside their references. Time-aware extraction captures events and timestamps for period-constrained queries. The engine is Apache-2.0 licensed, GDPR-compliant, encrypted at rest and in transit, and supports air-gapped and BYOC deployments, which makes it a workable fit when Hermes has to run in regulated or isolated environments.

How to Choose the Right Memory Architecture for Hermes

Engineering teams running self-hosted Hermes usually hit the architecture question once the built-in memory starts clipping context or once a second agent needs access to what the first one learned. The decision context is practical: Hermes ships with holographic memory on SQLite plus FTS5, a default memory_char_limit of 2,200 characters, MEMORY.md and USER.md files, and a single active external memory provider slot. The right architecture depends on how many agents and developers share state, how strict compliance is, and whether the team wants to maintain a provider plugin or lean on an existing one.

Tool Selection Criteria That Matter Most

Start with headroom beyond the 2,200-character memory_char_limit, since Hermes truncates holographic recall at that boundary and anything larger has to live in an external store. Interoperability comes next: the external store should fit the single provider slot cleanly, or expose itself through Hermes's built-in MCP client introduced in v0.2.0 so it can coexist with other tools. Source attribution matters when answers need to be auditable, so prefer stores that return the records behind a result. Security covers encryption, tenant isolation, and deployment mode (air-gapped, BYOC, or cloud). Maintainability is whether upgrades, migrations, and backups are someone's documented job.

Build vs Buy Tradeoffs

Writing a custom provider plugin for the Hermes memory slot is viable when internal data models are unusual or when an existing storage system has to be reused as-is. The cost is ongoing: schema changes, extraction prompts, eval harnesses, and keeping pace with Hermes releases all become the team's responsibility. Adopting Mem0 trades that work for an opinionated fact-extraction pipeline wired in through hermes memory setup. Adopting Cognee trades it for a knowledge graph plus vector index with pluggable backends and an MCP server. Buying usually wins unless the memory model itself is the product being built.

Reference Architectures by Team Size

A solo developer running Hermes locally is often fine with the built-in stack alone: holographic memory on SQLite with FTS5, MEMORY.md, and USER.md, tuned by raising memory_char_limit if sessions get long. A small team that wants shared user profiles and automatic fact extraction can slot in Mem0 through hermes memory setup, picking Mem0 Cloud, a self-hosted Docker server, or the in-process OSS mode. A larger team coordinating several agents and IDE clients can mount Cognee over MCP so Hermes, Claude Code, and Cursor share one graph through the remember, recall, and forget tools, backed by Postgres with pgvector or Neo4j.

Tool Categories Required for a Complete Hermes Memory Stack

A complete stack covers four categories. Persistent file memory handles identity and project notes; Hermes already provides this through MEMORY.md and USER.md. An external memory provider fills Hermes's single provider slot and holds structured long-term memory, with Mem0, Cognee, and others in the Honcho/OpenViking/Hindsight/RetainDB/ByteRover/Supermemory list as options. An MCP transport exposes memory tools to Hermes and other clients over HTTP, SSE, or stdio. Finally, a graph or vector backend persists the data itself: SQLite for holographic recall, and pgvector, Neo4j, Kuzu, Neptune, LanceDB, Qdrant, or Redis for larger external stores.

Step-by-Step Guide to Improving Hermes Agent Memory in Production

Improving Hermes Agent memory in production works best as a phased rollout rather than a single cutover. The built-in MEMORY.md and USER.md files keep running throughout, so teams can layer external memory in without breaking existing sessions. The sequence below moves from measuring the current limits of Hermes's holographic store, to installing a memory provider, to ingesting real knowledge, and finally to validating recall with source references. Each phase is reversible, which matters when agents are already serving traffic.

Extending Hermes Memory End to End

1. Audit current MEMORY.md and USER.md usage against the 2,200-character cap. Hermes's holographic memory is backed by SQLite with FTS5 full-text search, and the default memory_char_limit is 2,200 characters, roughly 800 tokens. Pull current MEMORY.md and USER.md contents from running agents and measure how often writes approach that cap. Once it fills, the agent has to delete something it already knew before it can store something new, so the audit is really a measurement of how much institutional knowledge is evicted silently each session.

2. Decide between an external memory provider and a raw MCP server. Hermes exposes a single external memory provider slot, with providers including Mem0, Honcho, Hindsight, Supermemory and Cognee, and only one is active at a time. Hermes also ships an MCP client, so any MCP memory server can be attached as a tool set. The provider route is deeper: it hooks the session lifecycle (capture per turn, recall per prompt, bridge at session end) rather than only exposing tools the agent has to remember to call.

3. Install the Cognee provider. The plugin is published on PyPI as cognee-integration-hermes-agent; a Hermes catalog listing (hermes plugins install cognee) follows catalog acceptance. The pip path works today:

pip install cognee-integration-hermes-agent
cognee-hermes-install
hermes memory setup        # pick "cognee", then Mode: local or remote

In local mode the wizard asks for an LLM API key, writes non-secrets to ~/.hermes/cognee.json and secrets to ~/.hermes/.env, and the next hermes session starts a Cognee server on 127.0.0.1:8011 (or attaches to one already running for the Claude Code, Codex or OpenClaw plugins) with storage in ~/.cognee. In remote mode it takes a Cognee Cloud tenant URL and API key instead; nothing runs locally and no LLM key is needed. Re-run the wizard to switch modes; values in cognee.json take precedence over environment variables.

4. Verify the round trip.

hermes cognee status                    # mode, dataset, service URL
curl -s http://127.0.0.1:8011/health    # local mode only

Then, in a hermes chat: "Remember that my favorite editor is Helix." The agent should call cognee_remember. Start a fresh conversation with /new and ask "What's my favorite editor?"; it should answer via cognee_recall.

5. Ingest existing knowledge and index the codebase. The plugin stores each completed turn and its tool calls in session memory under a capture policy that skips calls touching secret files and redacts secrets from what it keeps. For repositories, hermes cognee index-repo builds a deterministic code graph that cognee_code_search answers structural questions from (callers, dependencies, paths). Documents can be added with cognee_remember, which returns as soon as the server has the text; the graph is built in the background.

6. Validate recall with source references and use the recap tools. Every recalled fact carries the graph context it came from, so validation is a matter of asking representative questions and checking both the answer and its sources. cognee_forget handles user-directed deletion ("forget what we said about tennis") with a preview-then-confirm flow, cognee_switch_dataset moves a conversation to another dataset mid-session, and hermes cognee standup, digest and timeline summarise what was worked on, what was left open, and how one topic evolved across recorded sessions. At session end, and before Hermes compresses a long conversation, the plugin runs cognee.improve() so turns that get compressed away still reach the graph.

Best Practices for Operating Hermes Memory Long Term

Hermes agent memory holds up over months of use when teams treat it as a system that needs discipline, periodic review, and clear standards for what belongs where. The two profile files, MEMORY.md and USER.md, drift quickly if every session appends raw notes, and the holographic SQLite store with its 2,200-character default chunk window will surface noise if nothing is pruned. External memory layers deserve the same care: schedule reviews, keep source references on every recalled fact, and document the split between local and external storage. Cognee maintains the open-source graph-plus-vector memory layer used in production agent stacks and recommends the practices below from that work.

Curate what MEMORY.md stores versus what belongs in external memory. Keep MEMORY.md focused on short, stable environment facts and hard-won lessons that Hermes needs in every session, build commands, repo conventions, known gotchas. Push bulky project knowledge, document corpora, and conversation history into an external graph store where it can be queried on demand rather than pasted into context.

Keep source references on every recalled fact. When memory returns a claim, the agent should be able to point to the document and chunk it came from. Cognee's knowledge graph stores every entity and relation with a pointer to the ingested document and chunk, and recall returns the facts together with their source references, which makes it straightforward to audit and correct the memory layer rather than guessing where a stale answer originated.

Schedule periodic cognify re-runs so only new or updated files are processed. The cognify pipeline classifies documents, checks permissions, extracts chunks, uses an LLM to extract entities and relationships, generates summaries, and then embeds everything into the vector store and commits edges to the graph, and on re-runs it processes only new or updated files. A weekly or on-commit schedule keeps the external memory synchronized with the source of truth without rebuilding the whole graph.

Separate user profile from shared project knowledge. Keep USER.md scoped to the individual user's preferences and working style, and keep shared project knowledge in the external graph where multiple agents or teammates can query it. Pair this with time-aware memory, Cognee extracts events and timestamps so retrieval can answer questions constrained to particular periods, so older project state is still reachable without polluting current context.

How Cognee Simplifies and Scales Hermes Agent Memory

Cognee ships a native memory provider for Hermes Agent (cognee-integration-hermes-agent, Apache-2.0, source in topoteretes/cognee-integrations) rather than relying on a generic MCP connection. The provider hooks the Hermes session lifecycle: it captures each completed turn and its tool traces into session memory, injects a per-prompt memory block with the retrieved graph context trimmed to what Hermes does not already have, mirrors explicit Hermes memory writes through on_memory_write, and bridges the session into the permanent knowledge graph at session end from a detached worker, so exiting Hermes never waits on a graph build. Turns the server missed while booting are queued and replayed.

The agent gets six tools: cognee_recall for explicit search, cognee_remember for durable graph memory, cognee_forget for confirmed per-document deletion, cognee_switch_dataset, cognee_code_search over an indexed repository, and cognee_recap. On recall, Cognee's default GRAPH_COMPLETION mode uses vector search to locate relevant graph triplets, traverses the knowledge graph to assemble structured context, and returns the facts with the graph context they came from; temporal search answers questions constrained to a period, and opt-in contradiction detection records a contradicts edge instead of overwriting.

Backends are pluggable, so teams can reuse existing infrastructure: Kuzu, Neo4j, Amazon Neptune or Postgres for the graph, LanceDB or pgvector for vectors. The same local server is shared with the Claude Code, Codex and OpenClaw plugins, which is how one memory ends up serving every agent on a developer's machine. The engine is self-hostable, air-gap capable, and available as Cognee Cloud at $1.00 per 1M tokens processed plus $5 per additional workspace, with a free tier of 1M tokens and one workspace; Enterprise is delivered as a BYOC engagement with SLAs and a dedicated support engineer.

Key Takeaways and How to Get Started

Hermes ships a working holographic memory on SQLite with FTS5 and a default 2,200-character (~800 token) window, which is enough for short sessions but tight for long-running agents. For more capacity, Hermes exposes one external provider slot (Honcho, OpenViking, Mem0, Hindsight, Holographic, RetainDB, ByteRover, Supermemory) while MEMORY.md and USER.md keep running alongside it, or teams can route through an MCP server. Cognee adds a graph layer with source references and temporal recall, reachable through the native cognee-integration-hermes-agent plugin (hermes memory setup) or the Cognee MCP server (remember, recall, forget) over HTTP, SSE, or stdio. Start on the Cognee free tier (1M tokens, one workspace, agentic integrations with Claude Code, Codex, and MCP) or deploy the MCP server directly.

Related guides

• OpenClaw Agents and Cognee
• Codex Memory: How to Give OpenAI Codex Persistent Memory Across Sessions
• Claude Code Memory: CLAUDE.md, Auto Memory, and Persistent Memory Across Sessions
• MCP Memory: What It Is and How to Give Any MCP Client Persistent Memory

Frequently Asked Questions

What is Hermes Agent memory and how does the built-in system work?+

Hermes Agent is Nous Research's self-improving agent with a built-in learning loop that creates skills from experience, searches past conversations, and builds a model of the user across sessions. Persistent memory lives in two files inside the profile directory: MEMORY.md for environment facts and lessons, and USER.md for the user profile, alongside a skills directory under ~/.hermes/skills/. A holographic memory system backed by SQLite with FTS5 full-text search handles retrieval, and the default memory_char_limit is 2,200 characters, roughly 800 tokens.

Why do teams add an external memory layer to Hermes Agent?+

The built-in file store has a hard ceiling. When the agent hits the 2,200-character cap in MEMORY.md, it must delete something it already knew before it can write anything new, which erodes long-horizon recall. Hermes exposes a pluggable external memory provider slot with options like Honcho, Mem0, Hindsight, Holographic, RetainDB, ByteRover, and Supermemory. Only one external provider is active at a time, and MEMORY.md and USER.md continue running alongside it. Teams add an external layer to grow capacity, keep source references, and share memory across agents.

How do I connect Cognee to Hermes Agent over MCP?+

Yes, and the native route is simpler than wiring MCP by hand. Install cognee-integration-hermes-agent, run cognee-hermes-install, then pick "cognee" in hermes memory setup (local mode stores the graph in ~/.cognee; remote mode points at a Cognee Cloud tenant). Hermes gets cognee_recall, cognee_remember, cognee_forget, cognee_switch_dataset, cognee_code_search, and cognee_recap as first-class tools, plus hermes cognee status|index-repo|standup|digest|timeline. Hermes also ships a built-in MCP client, so teams that already run the Cognee MCP server (Docker or pip install cognee-mcp, HTTP/SSE/stdio) can mount that instead and share one graph with Claude Code, Codex, and Cursor.

What's the difference between using Mem0 as a Hermes memory provider and using Cognee over MCP?+

Mem0 installs through hermes memory setup, runs in Mem0 Cloud, self-hosted Docker, or in-process OSS modes, extracts facts server-side in a background thread, and adds mem0_profile, mem0_search, and mem0_conclude tools. Cognee connects over MCP and builds a knowledge graph where every entity and relation keeps a pointer to the ingested document and chunk, so recall returns facts with source references. GRAPH_COMPLETION uses vector search as a hint, then traverses the graph for structured context. Cognee Cloud is $1.00 per 1M tokens processed, plus $5 per additional workspace.

Can Hermes Agent memory be self-hosted and air-gapped end to end with Cognee?+

Yes. Hermes already stores MEMORY.md, USER.md, skills, and its SQLite FTS5 store on the local machine. Cognee's open-source engine is Apache-2.0 licensed, GDPR-compliant, encrypted at rest and in transit, and supports air-gapped and bring-your-own-cloud deployments. The Cognee MCP server runs in Docker over stdio, SSE, or HTTP, so it can sit on the same host as Hermes with no outbound network. Pair that with self-hosted graph and vector stores and a local LLM, and the entire Hermes memory path, including graph recall with citations, stays inside the environment.