search: graph-wide hybrid discovery with uniform typed results #648

Activity

1. ragnorc

   added

   needs-triageAwaiting maintainer triageAwaiting maintainer triage

   featureFeature proposalFeature proposal

   on Sep 5, 2026
2. ragnorc commented on Oct 18, 2026

   

   ragnorc

   on Oct 18, 2026 – with Cursor

   ContributorAuthor

   More actions

   Summary

   A client that does not know the node type cannot find an entity with one call. BM25, vector search, and RRF exist only inside a typed .gq query. The client must read the schema, choose a field on each node type, and run one query per type. The engine returns no uniform row (type, stable id, label, matched field, score kind). Raw BM25 scores and L2 distances from those calls are not one ranking.

   This gap is still present on main be031fea766fc8b280137f9ad9f34d4d2b3087bc. No other open issue replaces this request. This comment does not close the issue.

   Reproduction

   Verdict: reproduced. The observable gap matches the report. Later search work changed the per-type contract. It did not add a graph-wide call.

   Environment

   • Commit: be031fea766fc8b280137f9ad9f34d4d2b3087bc (origin/main at the time of this run).
   • Binary: omnigraph 0.14.0, internal schema 16, HTTP contract 0.13. The debug binary in this checkout is newer than every Rust source and older sources were not modified.
   • Graph: local file graph.omni, CLI embedded engine, branch main. No server. No Cedar bundle.
   • Text nearest used OMNIGRAPH_EMBEDDINGS_MOCK=1. Other queries used explicit vectors and no provider.

   Seed

   Three node types. Keys are different properties (Note.slug, Ticket.ticket_key, Credential.code). Note.embedding is Vector(4) with @embed("body", model="repro-model"). Ticket.embedding is Vector(8) with @embed("detail", model="other-model"). Credential.secret_note is an indexed string. Note and Ticket both implement Labeled. The interface only copies properties into the concrete types.

   First load: 3 Note, 3 Ticket, 1 Credential. embedding_generation is "unsupported". Null embeddings stay null. Commit hb1.01M575104QPE1XADKSF539YVEB.1.01M575104QPE1XADKSF539YVEB.

   Hand vectors: n-alpha [1,0,0,0], n-beta [0,1,0,0], n-gamma null. t-alpha and t-gamma are the 8-d analogues. t-beta is null. n-alpha and n-beta share the title Quantum ledger design and the body quantum ledger settles accounts. t-alpha and t-gamma share the subject Quantum ledger incident. Credential root stores quantum ledger admin token.

   What one call does

   omnigraph schema show graph.omni returns the full .pg source, including secret_note. GET /graphs/discovery is a different operation: it lists graph ids. It does not search entities. The only entity read route is POST /graphs/{graph_id}/query.

   Attempt	Result
   match { $x } and bm25($x.title, $q)	T6: $x is not bound. Exit 1.
   match { $x: Labeled }	T1: unknown node type Labeled. Exit 1.
   match { $x: Note | Ticket }	Q001 parse error, expected a clause. Exit 1.
   match { $n: Note $t: Ticket } with bm25 on Note and limit 5	Exit 0. This is a cross product. Five rows are pairs (n-alpha/t-alpha, n-alpha/t-beta, …), not one entity per hit.
   limit 1 offset 1	Q001 parse error. The grammar has limit and no page token for a ranked query.
   bm25($n.plain, $q) on a string with no @index	T27. The fix text says to declare @index.
   rrf(nearest($n.embedding, $v), bm25($t.subject, $q))	Typecheck accepts it. Execution returns no rows. The CLI error is Invalid argument error: column types must match schema types, expected Float32 but found Utf8 at column index 6. Exit 1. The compiler test admits this shape ($a: Doc $b: Doc in typecheck_tests.rs).

   What the client must do, and what it gets

   Same query text Quantum ledger, one field per type, commit …1.01M575104QPE1XADKSF539YVEB:

   • Note.title BM25: n-alpha and n-beta, both score 0.89427716. n-gamma is absent. limit 1 returns only n-alpha. There is no second page.
   • Ticket.subject BM25: t-alpha and t-gamma, both score 0.89427716.
   • Credential.secret_note BM25: one row, score 0.57536423, and the full string quantum ledger admin token.

   Equal scores use entity id order. On this graph the id is the key value, so n-alpha precedes n-beta. The JSON row has the columns the query projects. It has no matched-field name, no score kind, and no snippet operator. Each response pins graph_commit_id. Two calls share that id only while no write lands between them.

   A later append of five non-matching Note rows publishes commit hb1.01M575104QPE1XADKSF539YVEB.2.01M5753BAFH5S3GXMG1GW18WP8. On that one snapshot the same query text scores Note at 2.3128743 and Ticket at 0.89427716. The numbers moved because BM25 uses the type's own corpus. A client that sorts the two lists by the raw score ranks the notes above the tickets for a corpus-size reason.

   Vector, same second snapshot:

   • Explicit Vector(4) on Note: n-alpha distance 0.0, n-beta distance 2.0. Six null embeddings are absent. Eight notes exist. The JSON has no coverage count.
   • Explicit Vector(8) on Ticket: t-alpha distance 0.0, t-gamma distance 2.0. t-beta is absent. Distance 2.0 also appears on the 4-d Note search. The two 2.0 values are different geometries.
   • Vector(4) against Ticket.embedding: T15, property is Vector(8).
   • Note RRF of that vector arm and bm25(title, "Quantum ledger"), default k 60: n-alpha 0.03278688524590164 (1/61 + 1/61), n-beta 0.03225806451612903 (1/62 + 1/62).
   • Same RRF with text Payroll: n-alpha 0.01639344262295082 (vector rank 1 only), n-gamma 0.01639344262295082 (text rank 1 only; its embedding is null), n-beta 0.016129032258064516. The tie keeps id order. The row does not say which arm fired, and it does not say that n-gamma has no vector.

   Text nearest with the mock embedder:

   • Note and OMNIGRAPH_EMBED_MODEL=repro-model: n-beta distance 1.9555192, n-alpha distance 2.9086053. The stored vectors are the hand vectors above, not the mock embedding of the body. The note whose body equals the query text is the farther row. Nulls stay omitted.
   • Ticket with model repro-model: the call fails. The error names stored model other-model and query model repro-model.
   • Ticket with model other-model: t-gamma 1.9654107, t-alpha 2.7065465. Those distances are not comparable with the Note distances (different dimension, different recorded model).
   • An explicit vector on Note still returns distance 0.0 and does not read the recorded model.

   Differences from the report

   The report's client story is accurate. Since the cited tree 02d182b6, per-type search gained projectable scores, an id tie-break, T27 (a full-text call needs @index), a model check on text nearest, and the rule that the ranked binding roots its component. interface still does not bind in a query. None of that is one discovery call.

   Root cause

   Each node type is its own Lance dataset. A read binds one type name. The binding rule is variable ~ ":" ~ type_name in query.pest. search, bm25, nearest, and rrf are expressions on a property of that binding (lines 229–234). Full-text typecheck requires an index on that one node property (T27). nearest resolves one property's dimension and @embed model (resolve_nearest_query_vec in crates/omnigraph/src/engine/search.rs). rrf fuses two arms. T59 refuses the arms when one traversal connects the two bindings. Unconnected bindings are admitted and then fail in execution, as above.

   Schema interface expands properties at schema compile time. The query catalog has no type Labeled, so the binder reports T1.

   Cedar resources for a read are Graph, Server, and Cluster (crates/omnigraph-policy/src/lib.rs). ResourceScope is the graph or one branch. The query-layer predicate_for method is not implemented. Authorization states that per-entity and per-property authorization is not available. A caller who can read the branch can project secret_note.

   The affected class is every cross-type lookup: CLI, POST /query, and any agent tool that can run only a typed read. It is a missing contract, not a corrupt commit.

   Recommended fix

   Add one typed read that searches the authorized node types on one captured snapshot. Lower it to the per-type scans the engine already runs. Do not add a second index, a side table, or a string flag that means "search everything".

   The contract:

   • The caller may pass a type filter and a field filter. The default field set is the indexed fields the caller named, or an explicit searchable-field declaration. A default that scans every indexed string will return Credential.secret_note to every reader.
   • Each row carries the node type's stable identity, the entity id (@id), a label the caller selected or a schema-declared display property, the matched field, and a score kind (bm25 on that field, L2 on that vector, or rrf). Do not invent a slug.
   • Text-only, vector-only, and hybrid are three named modes. Hybrid uses rank fusion (today's RRF, k 60 unless named). It must not sort raw BM25 or raw L2 across types, fields, dimensions, or models. Ties break by score kind, then type identity, then entity id, and that order is documented.
   • One response pins one graph_commit_id. Bounds cover type fan-out, candidates per arm, and result rows. limit is the fused cut. Pagination needs its own cursor contract; a bare offset is not in the grammar today.
   • A null embedding drops that row from the vector arm and the response counts the drop. A missing model match or a dimension mismatch excludes that field with a named reason and still returns the other fields. Incompatible models do not share one distance list.
   • Snippets and diagnostics use the same authorization as the rows. Today that authorization is the whole branch. Until a finer view exists, diagnostics must not add a second channel, and snippets must come only from the matched eligible field.

   This keeps invariant 3 (one snapshot), invariant 9 (the meaning lives in the typed plan), invariant 7 (indexes stay derived), invariant 10 (the engine enforces the actor), and invariant 11 (the fan-out is bounded). A patch that merges the raw scores would publish a ranking that later clients will depend on. That liability is larger than a smaller RRF result.

   Tradeoff. This design spends correctness and a clear score identity first. It does not build one global ANN index. A later physical index can accelerate the same rows. Efficiency of a single shared vector structure is the quality this choice defers, because a fast list of incomparable distances is a wrong answer.

   Alternatives

   • Keep client fan-out. The client already can do this. The snapshot can move between calls. The scores are not one order. Coverage is invisible. This is the current behavior.
   • Issue #1055 would run several named queries at one snapshot. The client would still write one query per type and would still fuse the scores.
   • PR #884 would let a query bind an interface and lower it to per-type pieces. That helps types that share a declared type set. It is a draft, it is not on main, and it does not define open discovery or cross-type fusion.
   • PR #793 (RFC 0048) defers "discovery across node types (a typed union whose target identity carries its type)". Reuse its score identity if that RFC lands. Do not hide this operation inside the named arguments that RFC adds to nearest and bm25.

   Liability. Shipping the call before PR #951 (actor views) means every branch reader sees every type. The dangerous default is "all indexed strings", not the existence of the call. A queue of embedding jobs, or a shadow cross-type index, would add a second authority the manifest does not own.

   Lance

   Pinned Lance is 11.0.0 (released 2026-08-30). Latest stable is v13.0.0 (2026-10-07). v12.0.0 is between them. v14 is beta only (beta.19 on 2026-10-16).

   Lance search is one scanner on one dataset. In the v11.0.0 scanner, full_text_search and nearest are two options. When a vector prefilter and a full-text query are both set, fts_rerank ranks the vector hits with BM25 (scanner.rs). That hybrid path does not open a second dataset. OmniGraph already fuses ranks in the query engine because each node type is its own dataset.

   Upstream items that matter, and do not close this issue:

   • PR #7905 BM25F cross-field search, in v13.0.0. It scores several columns of one table. OmniGraph is still on Lance 11, so this API is not in the pin.
   • PR #9847, merged 2026-10-11, after v13.0.0. It lets a distributed planner supply the indexed half of one hybrid FTS query. It is not in a stable release.
   • Issue #9058 and open PR #9243: BM25 corpus statistics can change with the filter and with index coverage. OmniGraph already places the full-text prefilter from coverage for this reason. A Lance fix would change per-type scores. It would not make two types one corpus.
   • Open PR #10001: scalar prefilters scoped to fragments, still one scan.
   • Open PR #9944: highlight byte ranges. Useful later for a snippet. Still one column of one dataset.
   • Open issue #5038: a DataFusion table function for hybrid search, still one table.

   No open or merged Lance change searches across datasets or returns a typed graph entity.

   Related work

   • #606 is closed. Its retrieval-algebra draft continues as PR #793, which defers this discovery operation. Analyzed lexical search is PR #792.
   • #655, #656, and #657 own pending, stale, and mixed-model embeddings. This run shows the current holes: load reports embedding_generation: unsupported, nulls disappear from nearest with no count, and an explicit vector skips the model check. Discovery must report that state. Those issues do not provide the call.
   • #649 is neighborhood expansion from a known entity.
   • #991 would add schema and ad-hoc read tools. Those tools would still run typed queries.
   • #1055 batches named queries at one snapshot.
   • #884 and #951 are the draft directions for a type union and for finer reads.

   Supersession: none. #648 remains the request for one bounded discovery call with a uniform row and an explicit cross-type ranking.

   Limitations

   • The run is the CLI on a local file graph. It is not a served HTTP call and not a Cedar 403.
   • Corpora are small. The BM25 figures (0.89427716, then 2.3128743 on Note only) belong to this seed. They show that the score follows the type corpus.
   • The mock embedder and the hand vectors are deliberate. They show a text nearest over vectors that were not produced by that model.
   • The two-type rrf failure is repeated twice. The backtrace has no source line. This note does not assign a planner function to that Arrow error.
   • Lance pages on lance.org are HTML shells here. The scanner citation is the v11.0.0 source. Release notes and the PRs above are the upstream comparison.