Does Utopia support NER for the entity extraction phase of the pipeline?

[lexful-shah]

Is it possible to use something like GLiNER instead of an LLM for the entity extraction pipeline?

[WaylandYang]

Thanks for asking, and welcome — GLiNER is exactly the thing to reach for if you've built extraction pipelines before, so this deserves a real answer rather than a one-line no.

Not today, and the reason is structural. Extraction here is one call per chunk that returns four things at once: entities with an ontology class, the relations between them, the time interval each relation holds over (normalised against the document's own date), and a verbatim quote for every fact. That quote becomes the evidence row the bi-temporal ledger and the derivation chain hang off. GLiNER gives you spans and zero-shot labels — the first of those four; GLiREL would add the second. The time normalisation and the quote have no counterpart, and they are what the rest is built on. There's also no seam to plug a different extractor into: utopia-extract builds a prompt and parses JSON, and the model is any OpenAI-compatible endpoint.

If what you're after is keeping documents off a hosted API, that already works — point the chat model at Ollama or vLLM. A local 7B (we've run qwen2.5:7b) does extraction fine, just slower.

Where a NER model would earn its place is a different job from replacement: a cheap span pass alongside the LLM call, as a recall net for entities it walked past or a check on names it invented. No design for that yet. If you want to explore it, the first question to answer is whether GLiNER's labels can be driven from a base's ontology classes at call time — those differ per knowledge base, and that's what would decide whether it's a real seam or a bolt-on.

[lexful-shah]

Thanks for the insightful reply, in my current setup we are facing high unit cost of indexing the knowledge graph using LLM for entity extraction (due to the volume of documents we are indexing) and exploring GLiNER for that phase of our pipeline.

I will circle back to Utopia if we see we don't get deterioration in our KG curation from using open source models instead of NER.

Thanks again for looking into this!

[WaylandYang]

Circling back with numbers, since your question is really about cost per document and I didn't want to answer that from intuition.

I ran GLiNER (multi-v2.1) against our pipeline on the two corpora we use for our own benchmarks, in Chinese and in English, same answer key for both sides. Three label settings:

• Given only the ~20 classes that appear in the answer key, GLiNER types what it finds very well: 17 of 20 right on the English pharma corpus, about a second per corpus on CPU. That is its ceiling.
• Given the real class space of a schema.org base (965 classes, in batches of 25 with the best score kept per span), precision falls to 13 of 20 and 4 of 14. It still knows "this is an organisation"; it stops telling government_organization from medical_organization, and DeepBlue 3.0 becomes a drug. The output also fills with dates and fragments that would all need cleaning.
• All 965 labels in one pass: it finds almost nothing (0 to 4 entities per corpus). The labels fill the encoder window and the text is truncated away, silently.

The numbers, side by side. "Found" is how many answer-key entities were extracted at all; "typed right" counts only entities that were found and have a class in the key, so the two columns have to be read together (a row can be 11/11 on typing while missing three entities).

Chinese corpora

	pharma found	pharma typed right	tech found	tech typed right
Pipeline, at extraction	22/22	11/20	15/16	3/13
Pipeline, after type resolution	22/22	16/20	15/16	10/13
GLiNER, answer-key classes only	18/22	16/17	12/16	10/10
GLiNER, 965 classes in batches	20/22	9/19	13/16	5/11
GLiNER, 965 classes in one pass	0/22	–	0/16	–

English corpora (same documents, translated sentence by sentence)

	pharma found	pharma typed right	tech found	tech typed right
Pipeline, at extraction	21/22	17/20	15/16	9/13
Pipeline, after type resolution	21/22	19/20	15/16	12/13
GLiNER, answer-key classes only	22/22	17/20	13/16	11/11
GLiNER, 965 classes in batches	22/22	13/20	16/16	4/14
GLiNER, 965 classes in one pass	4/22	2/3	2/16	–

Timing per corpus: GLiNER about a second with 20 labels, about 40 s in 39 batches, about 70 s with all 965 in one pass, all on CPU. The pipeline takes 40 to 100 s depending on the model endpoint.

Its recall in English is genuinely good: in the batched setting it found every entity in both corpora. In Chinese it structurally cannot find names longer than 12 characters once you segment by character.

Our pipeline on the same keys: 19 of 20 and 12 of 13 typed right after type resolution, one entity missed across both corpora. And that is only entity typing; relations, validity time and telling two people with one name apart are not something GLiNER attempts.

Where your cost concern and these numbers meet is the first setting. If the candidate classes for a chunk are narrowed first (we already retrieve a few dozen per chunk when the ontology exceeds the prompt budget), GLiNER is a cheap and accurate first pass, and its spans could go to the LLM as hints or replace the type-resolution round. That is the seam from my earlier reply, and now there is a reason to build it. No timeline yet. The comparison script, corpora and tables are in scripts/bench/README.md so anyone can rerun it.

If your deterioration test turns up numbers, I'd genuinely like to see them here. And if you want to think through the design of that pre-pass, this thread is a good place for it: how to pick the candidate classes per chunk, what to do when GLiNER and the LLM disagree on a span, and whether the spans should feed the prompt or only the recall check. Happy to work through it here.

[lexful-shah]

Thanks for running the numbers, this is far more than I expected.

Our setup lands in your first setting by construction. We index IT service provider documentation (runbooks, network diagrams, ticket notes) into a knowledge graph against a small, fixed set of entity types rather than a per-base ontology. So the "which candidate classes for this chunk" question has a trivial answer for us: the full label set, every chunk, well inside the encoder window. Your ~20-label row is the setting we are in, though I have not measured GLiNER on our domain yet, so I would not borrow your accuracy numbers for it. On cost, a GLiNER pass on CPU should be roughly two orders of magnitude below what we pay for an LLM extraction call per chunk, but that is an estimate until we run it.

Two caveats from our side. First, entity typing is not where our cost is. Our extractor is LightRAG-style: one call returns entities, relations, and a natural language description per entity, and the relations plus descriptions are most of the output tokens. So the plan we are evaluating is GLiNER + GLiREL for entities and relations, with the LLM kept only for descriptions, not GLiNER as a drop-in. Second, we do not have a labelled answer key yet. Our current extraction benchmark uses yield per chunk as a proxy, which is exactly the kind of number you declined to answer from. Building the key is the first step; I will likely adapt gliner_compare.py and your truth-file format so the results are comparable, and will post them here.

On the three design questions, from having worked with a closed taxonomy:

1. Candidate classes. For an open ontology I would not hand GLiNER the leaf classes at all. Your data shows it is reliable on anchors (organisation, person, place) and unreliable on siblings (government_organization vs medical_organization). Give it the anchor layer, which is small and fixed per base, and let type resolution refine to the subclass. That also removes the batching and the truncation failure mode entirely.

2. Disagreement. Treat the two extractors asymmetrically. A span GLiNER found and the LLM did not is a recall candidate, not a fact: it only enters the graph if a second LLM pass confirms it and produces the quote. A span the LLM found and GLiNER did not stays, since the LLM has context GLiNER lacks. On type disagreement the LLM wins at extraction time and GLiNER acts as a tiebreaker in type resolution. On boundary disagreement take the longer span.

3. Prompt vs recall check. Recall check first. Feeding spans into the prompt anchors the LLM on GLiNER's false positives, and you already saw the output fill with dates and fragments at 965 classes. Start with the drop signal, measure how often GLiNER finds something real that the LLM walked past, and only then decide whether the spans earn a place in the prompt, filtered by score.

Happy to keep working through it here once we have numbers.

[MikeRecognex]

@lexful-shah — on the cost side, it may be worth separating two routes, since they solve different parts of the bill.

GLiNER + GLiREL replaces entities and relations, but as you say the descriptions are most of your output tokens, and encoders can't generate those. The API call stays, and so does the bulk of the cost.

The other route is distilling the whole extractor into a small local model — entities, relations and descriptions in one call — and dropping the hosted API entirely. The useful property is that you don't need labels to start: you're already running an LLM extractor in production, so its outputs over a few thousand chunks are the training set, and the student learns your exact schema rather than being prompted into it.

For model choice there's a close analogue published in June — a LoRA study on merchant information extraction from noisy transaction strings, 24 variants across Gemma 3, Qwen 3.5, Aya and LLaMA 3.1 ([arXiv:2606.08051](https://arxiv.org/abs/2606.08051)). Four findings that seem likely to transfer:

• Qwen 3.5 4B with JSON-only prompting reached 96.60% F1, within 0.35 points of a LLaMA 3.1-8B baseline at half the parameters.
• Qwen 3.5 0.8B reached 94.75% F1, matching models 2.5–4× larger.
• LoRA rank 8 came within 0.20 points of rank 32, so small adapters are production-ready.
• Chain-of-thought supervision was unnecessary — Think and Nothink templates differed by under 0.004 F1, and the Think-trained models learned on their own to emit an empty reasoning block followed straight by JSON.

So: Qwen 3.5 4B Instruct, LoRA rank 8, JSON-only, no reasoning traces. Then test 0.8B, since at your volume throughput may be worth more than the F1 points. Apache 2.0 either way.

[Unsloth](https://github.com/unslothai/unsloth) is the path of least resistance for the training run. Worth being clear it's a training accelerator and single-GPU, not a serving stack — train with Unsloth, serve with vLLM, and use grammar-constrained decoding at inference so schema validity stops being a model property at all.

One existing checkpoint worth trying before training anything: InternScience/Agents-K1, a 4B Qwen3 fine-tune aimed at NER and RE with structured JSON output, reporting +3.3 average F1 over its base across ten NER/RE benchmarks. Either as a zero-shot baseline to beat, or as a starting point to fine-tune from instead of the raw base.

The caveat is the one you already named. A distilled student is capped by its teacher and inherits its errors, so without the answer key you can't distinguish degradation from drift. Same blocker, same first step.

One domain thought, if useful: runbooks and network diagrams aren't prose. Hostnames, FQDNs, IPs, CIDR blocks, MACs, serials, error codes and ticket refs are regex-able at near-perfect precision. Splitting the taxonomy so patterns handle the identifier classes and the model only handles the fuzzy ones — vendor, service, team names — might take a large fraction of the volume off the model path entirely, and it improves precision rather than trading it away.

[WaylandYang]

Both routes are capped in ways you can't see without the key, so I'd build that first and let it decide between them rather than after.

@lexful-shah — what's in scripts/bench/ if it saves you the reverse-engineering: make-truth.mjs writes the truth files, recall.mjs scores a run against one, gliner_compare.py is what produced the tables above, and corpora/ has pharma and tech in Chinese and English (same documents, translated sentence by sentence, one answer key for both sides). The key format is the part worth copying; the rest is ours and probably won't survive contact with runbooks.

On distillation: the ceiling @MikeRecognex names is the reason a key pays for itself twice. A student capped by its teacher looks fine on yield per chunk — it reproduces the teacher's misses faithfully, and yield can even go up while recall goes down. Recall against a key is the measurement that separates those.

The identifier/pattern split looks right to me and is the cheapest thing on the list. Worth doing before either model route, since it changes what's left for the model to be good at — and therefore what you'd be measuring.

One caution on the LoRA numbers: they're from merchant strings, which is a different shape of text and a much smaller label space than runbooks. I haven't read the paper, so I can't say more than that — treat the model ranking as a place to start, not a result to inherit.