Keenable × cognee: Giving Web Retrieval Persistent Memory

Web pages are awkward source material for AI agents. The information is often mixed with navigation, cookie banners, scripts, footer links, and other page furniture, and even a successful fetch is only good for as long as that content stays in the current context.

Keenable can take care of the web side of that process — pass it a URL and it returns the page as clean Markdown, either using an indexed copy or by fetching the page directly. An optional extraction prompt can narrow the result to the content the agent actually needs.

cognee is there for what comes afterward — the returned text can enter persistent graph-backed memory, where it can be processed, connected with other information, and retrieved during later work.

With cognee 1.5 and later, Keenable is available as a built-in web-fetch backend. The basic setup can be as small as one environment variable:

KEENABLE_API_KEY="kn_..."

cognee picks a fetch backend from what is configured: Tavily if TAVILY_API_KEY is set, then Keenable if KEENABLE_API_KEY is set, otherwise the built-in crawler. Tavily wins when both keys are present — so if a Tavily key is already in your environment, unset it to route URL ingestion through Keenable.

Then a URL can go directly into the memory workflow:

await cognee.remember("https://docs.example.com/pricing")
await cognee.recall("what does the enterprise tier cost?")

Keenable gets readable content out of the page and cognee interconnects what the agent learns from it and keeps it available for future work.

You can start from whichever side

If you already build on Keenable, this integration gives fetched pages a path into persistent memory. Instead of separately wiring storage, chunking, embeddings, and graph processing around the content your agent reads, the returned Markdown can enter cognee's existing ingestion pipeline.

If you already use cognee, URLs already work with the default crawler, which Keenable swaps for a service that returns markdown instead of HTML, returning agent-readable content before downstream memory processing begins.

The integration uses the HTTP client already shipped with cognee, so there's no separate Keenable SDK to install.

Which backend runs depends on where you enter. For URLs going into memory through remember() or add(), selection comes from the environment — whichever key is present, with the precedence above. When calling cognee's fetch layer directly, the backend is a parameter:

from cognee.tasks.web_scraper import fetch_page_content

pages = await fetch_page_content(urls, preferred_tool="keenable")

The fetch boundary

The integration connects at cognee's shared web-fetch layer.

URL ingestion and the structured web-scraping workflow use that boundary to retrieve page content before it enters downstream processing. Keenable can be selected there without requiring Keenable-specific behavior throughout the rest of the memory pipeline.

Once the page comes back, the later stages can handle it like other ingested content.

What happens to a URL

1. Guard. Before anything is fetched, the URL goes through cognee's HTTP policy checks. Unsupported schemes, disabled HTTP access, and destinations rejected by the network policy are stopped before a request leaves the application.

With Keenable selected, the request cognee itself makes goes to Keenable's API — so the check is a policy gate on what you ask cognee to ingest, not a constraint on what the fetching service can reach. Worth knowing which side of the call that check falls on if you're treating it as a network boundary.

2. Fetch. cognee sends the URL to Keenable's /v1/fetch endpoint with the API key in the X-API-Key header.

Keenable can serve its indexed copy of a page or go and fetch it live. On the memory path that is an environment setting:

KEENABLE_LIVE_FETCH="true"

The finer controls — including an extraction prompt that narrows the response to the part of the page you care about — travel on a config object, which the fetch layer and the scraper task take as a parameter:

from cognee.tasks.web_scraper import fetch_page_content
from cognee.tasks.web_scraper.config import KeenableConfig

cfg = KeenableConfig(live=True, prompt="Extract the pricing table and plan limits only.")
pages = await fetch_page_content(url, preferred_tool="keenable", keenable_config=cfg)

3. Ingest. The returned Markdown enters cognee's regular processing pipeline. The content can be chunked, embedded, and processed into entities and graph-connected information without the downstream pipeline needing to know which fetch backend produced it.

4. Recall. Later queries can retrieve the information processed from the page along with the entities and relationships created from it. With cognee's recall(), the model receives context derived from the page content instead of having to work through the original HTML again.

Batches fail independently

The Keenable backend can fetch several URLs concurrently, with up to five requests in flight by default.

Failures are isolated to individual URLs. If one request fails, successful pages from the same batch can still continue through processing.

If every URL raises, the call re-raises the first underlying error rather than making an authentication or transport failure look like a legitimate empty batch.

To avoid exposing credentials or other sensitive values in query parameters, the warning path doesn't write the URL itself to the log. Instead, it can identify the failed item by its position in the batch and its exception class.

Track pages over time

For pages that need to be checked repeatedly instead of ingested once, cognee's scheduled scraper can run through the same fetch path (it needs apscheduler):

from cognee.tasks.web_scraper import cron_web_scraper_task
from cognee.tasks.web_scraper.config import KeenableConfig

await cron_web_scraper_task(
    url=["https://docs.example.com/changelog"],
    schedule="0 6 * * *",
    job_name="daily_changelog",
    keenable_config=KeenableConfig(live=True),
    tavily_api_key=None,  # this parameter defaults to TAVILY_API_KEY, which would win
)

Each run writes WebPage nodes into the graph, linked is_part_of to a WebSite, which the ScrapingJob links to in turn with is_scraping.

The fetched page also receives a SHA-256 content hash, giving the scraping workflow a stable value for identifying whether content has changed between runs.

This is a different path from step 3 above: it writes and indexes graph nodes directly rather than running chunking and entity extraction. Both paths still go through the same fetch boundary, though, so with Keenable in place they get identical clean page content — only what happens to that content afterward differs.

Try it yourself

Install cognee:

pip install cognee

Set the Keenable key once in your environment:

KEENABLE_API_KEY="kn_..."

Optional Keenable behavior can also be configured through the environment:

KEENABLE_LIVE_FETCH="true"
KEENABLE_BASE_URL="https://api.keenable.ai"

Then give cognee a URL and retrieve what the agent needs when the information becomes relevant again.