Small Models in AI Memory: What We Learned Making Entity Extraction LLM-Free

Everyone (and their AI agent) wants all the benefits of frontier models with the speed and efficiency of simple ML models. We all want genius-level reasoning and accuracy, but instantly and for free. And we want it to be convenient, as we don’t really want to go back to training our classifiers.

This need for convenience is part of why Jev, TypeSafe AI’s model for structured decisions, is having a bit of a moment right now. At the same time, there’s renewed interest in BERT-style encoders and related modern architectures. And the enthusiasm for small language models never really went away.

What’s changing is that people in the industry are beginning to appreciate specialized models from different perspectives, with hopes of reducing latency and costs. It’s also becoming increasingly evident that these kinds of models can add real value in production.

Here at cognee, we’re on a small-model journey of our own. We identified ways to make our memory for AI agents run reliably and consistently with small models and we observed concrete improvements in speed and efficiency. If you’ve missed it, the open-source version of cognee can already run locally, with entity extraction and retrieval available without an LLM API key.

So, in this post, we want to share a bit about how we got there, what we’ve been working on in parallel, and what we learned along the way.

Rethinking Entity Extraction in cognee

Our journey began with a simple motivation: make entity extraction LLM-free. As we all know by now, cognee builds graphs that contain, among many other things, raw information (chunks) and entities extracted from those chunks. Typically, an LLM would receive a chunk as input and propose the entities and the relationships between them. When ingesting multiple large documents, the time and cost of those LLM calls become noticeable.

In parallel, through practice, our understanding of what makes a graph valuable grew. We recently published a paper on graph-modified embeddings in which we detail the improvements we observed researching multi-stage retrieval in a controlled synthetic setting in which the graphs were built from the same underlying facts as the corpus. Meanwhile, our experience with noisier, less complete graphs taught us that meaningful signal can emerge without a perfect graph, at both retrieval and completion time.

We explored traditional fixed-label named entity recognition (NER) systems, but they were too noisy and incomplete. And fine-tuning a small language model (SLM) as a first step seemed like a bit of overkill. We began researching more traditional bidirectional architectures and were happy to learn that the field is alive, kicking, and providing value.

GLiNER is one of the big success stories out there — an older architecture (the well-known DeBERTa-v3) being repurposed for the modern need of general entity extraction with user-defined types. The newer GLiNER2 models support extracting relationships between entities, too, which was important for our use case. There was a lot of good buzz around the model, which made us more open to investing time and resources into making it work.

And make it work we did.

Taming a Noisy Small Model

In some ways, GLiNER performed both better and worse than we expected. It requires entity types to be provided, which was a bit of a hurdle that you don’t need to jump over with an LLM. But once we figured out how to do that somewhat flexibly, it extracted a lot of relevant entities, though at times it was noisy and inconsistent.

Through systematic and measurable trial and error, we were able to combine several heuristics that made the model perform well in production. That included creating a “document sketch” (a short sample of each document’s most information-dense sentences, which GLiNER scans) to select entity types automatically, along with rules for discarding relationships when their endpoints couldn’t be identified unambiguously.

Instead of just comparing what got extracted to what an LLM would extract, we separately measured what got extracted and how much it helped retrieval. Good retrieval doesn’t require a perfect-looking graph, but the noise had to be tamed. Since we already had continuous evaluations in place, we were able to use the information from those evaluations to make the end-to-end system stable and effective.

We never expected GLiNER to be a plug-and-play replacement, and it indeed required a few modifications to prove itself. Our internal evaluation suggests it was worth it: local extraction was roughly 30× faster per chunk than an estimated LLM baseline.¹ Efficiency was more modest for bulk ingestion on a single CPU, and actual throughput depends on hardware, workload, and parallelism.

Under our estimated token usage, this local LLM-free setup with GLiNER and FastEmbed can save users approximately $570 in API charges per GB of text compared with cognee’s default OpenAI stack, excluding local inference costs.² With encouraging results and positive feedback from production use cases, we’d call that both satisfactory and satisfying.

The open-source GLiNER integration is a demo of cognee’s production extractor, which has higher accuracy and broader label coverage. The production version is available with an Enterprise license.

What Splitting Up LLM Calls Taught Us

Entity extraction was just our starting point. Elsewhere in cognee, a single LLM call can return a complex, structured response that serves several closely related purposes. When moving toward small models, we learned that these tasks could be separated and handled by more specialized components, including classifiers, older encoder architectures, and fine-tuned SLMs.

For example, a call that returns user satisfaction, detected preferences, and lessons learned could be split between a transformer-based classifier for satisfaction and SLMs for extracting the preferences and lessons.

This comes with its costs and benefits. The costs are obvious: you do have to think about the order of things, orchestration, and coordination.

The benefits are a bit surprising: decomposing complex calls made us think about components of our system at a more granular level and identify new opportunities. Maybe two small things could be combined? Now that not everything is part of one response model, maybe we can do something between two components? Could we measure each step more precisely, or let less urgent work run in the background?

The added complexity was generally worth it, as the result is a faster product where the sum of the small (model) parts is somehow greater than what it replaced.

All this doesn’t mean that we’re giving up on LLMs. For one, they’re still the right tool in general settings for the actual completion, once cognee assembles the context. When we’re building a new feature, the quickest way to prototype is still with frontier models. And not every use case requires small models; some deployments are better suited for LLMs for various reasons. cognee’s development through small models is a parallel track with upsides we’re still discovering.

One Small Model Down, More to Come

Integrating one small model was an instructive exercise that showed us its limits and possibilities. The lessons along the way were sometimes surprising, occasionally frustrating, but always valuable. We learned that integrating a small model requires additional logic, and the work led us to rethink how our system could work once we had the opportunity to break it into smaller components. And we plan to keep doing more of that.

On a more general note, since its inception, cognee has relied on those frontier models. The work on small models has been rewarding, and it’s exciting to explore how AI memory can be organized more efficiently for real-world use cases. There is more to come, and we are looking forward to sharing it with you soon.

FAQ

Answers to the most common questions from this guide.

Which GLiNER model does cognee’s open-source extractor use?+

It uses Fastino’s gliner2.5-base-v1, a 194M-parameter checkpoint built on a DeBERTa-v3-base encoder. It’s installed with the cognee[gliner] extra and downloads about 750 MB on first use. When no LLM API key is configured, cognee picks this extractor automatically.

What goes into the document sketch?+

In the open-source extractor, the sketch is a condensed sample of the whole document, capped at 12,000 characters or 3,000 words. It keeps sentences likely to carry entities, such as those with dates, amounts, IDs, contact details, capitalized names, or headings, plus sentences spread evenly through the text. GLiNER reads the sketch once against built-in banks of entity and relation labels, and the labels it detects become the schema for every chunk of that document. If you supply an ontology, its labels are used instead.

Can I extract with GLiNER and still use an LLM for answers?+

Yes. With an API key set, cognee defaults to LLM extraction, but GRAPH_EXTRACTOR=gliner_demo in the environment, or cognify(extractor="gliner_demo") for a single run, keeps extraction local while completion searches use your configured LLM. To keep an LLM for extraction but use a cheaper one, LLM_EXTRACTION_MODEL sends extraction to a separate model, including a local one served through Ollama.

Does GLiNER extraction need a GPU?+

No. It runs on CPU, and by default cognee runs several model batches at once, sized to the machine’s threads and available memory, at about 2 GB per batch in flight. GLINER_INFERENCE_THREADS=1 runs one batch at a time, and the extracted graph is identical either way. For larger ingestion jobs, the GLiNER calls can be offloaded to a separate worker service over HTTP, gRPC, or AMQP, which keeps PyTorch and the model download off the application host.

Footnotes

1. On an Apple M4 Pro, GLiNER processed 2,000-token chunks in a median time of ~1.2 seconds, compared with an estimated ~35 seconds for gpt-5.6-luna at its default medium reasoning setting. ↩

2. Cost estimate methodology. Based on cognee’s default OpenAI configuration: gpt-5.6-luna at $0.20 per 1M input tokens and $1.20 per 1M output tokens, and text-embedding-3-large at $0.13 per 1M tokens. Assuming 250,000 source tokens per MB, 2,000-token chunks (median estimate), and 125 extraction calls per MB, the estimates are approximately $0.08 for input and schema tokens, $0.15 for graph JSON output, $0.30 for reasoning tokens (median estimate), and $0.04 for embeddings.

   Under these assumptions, that comes to $0.57/MB, or $570 per decimal GB (1,000 MB) in potentially avoided API charges if the listed calls are replaced by local extraction and embeddings. This is an illustrative cost estimate, not a measured saving for every deployment. It excludes local compute, hardware, storage, and operating costs, and actual token consumption varies with documents, extraction settings, and model usage. ↩