embeddings: enforce model and source provenance beyond vector dimension #657

Activity

1. ragnorc

   added

   needs-triageAwaiting maintainer triageAwaiting maintainer triage

   featureFeature proposalFeature proposal

   on Sep 5, 2026
2. azimafroozeh

   added

   acceptedTriaged and validated; open for a PRTriaged and validated; open for a PR

   needs-rfcSize L: the design must land as an RFC before any codeSize L: the design must land as an RFC before any code

   P-mediumMedium priorityMedium priority

   and removed

   needs-triageAwaiting maintainer triageAwaiting maintainer triage

   on Oct 1, 2026
3. azimafroozeh

   self-assigned this

   on Oct 1, 2026
4. ragnorc commented on Oct 18, 2026

   

   ragnorc

   on Oct 18, 2026 – with Cursor

   ContributorAuthor

   More actions

   Summary

   An embedding field records one optional model name for the whole column. Each stored value is only a list of floats. Equal length does not mean those floats came from one model, one source text, or one preprocessing contract. A text nearest compares the query embedder with that column name. A load, an update, and an explicit vector query do not. A hand vector of the right length is ranked with the other rows, and the result does not say so.

   Eligibility

   This issue is still open. It has no comments and no linked fix pull request. The timeline has three cross-references from #648, #655, and #656. Those notes point here. They are not a reproduction of this issue. #655 and #656 already have reproduction comments on this SHA. This note uses those results and does not repeat their load, coverage, or job matrix.

   Verdict

   Reproduced. The gap in the report is still the gap on current main.

   SHA: be031fea766fc8b280137f9ad9f34d4d2b3087bc (feat(query): the ranked binding roots its component (#1069), 2026-10-18). git fetch origin main returned that SHA. The checkout was that commit.

   Environment

   omnigraph version reports 0.14.0, storage format 16 (serves v14 to v16), and HTTP contract 0.13. The binary was target/debug/omnigraph from this checkout. rustc is 1.97.1. Cargo.lock pins Lance 11.0.0.

   The provider was mock only. The process had OMNIGRAPH_EMBEDDINGS_MOCK=1, OMNIGRAPH_EMBED_PROVIDER=mock, and OMNIGRAPH_EMBED_MODEL=model-a. OPENROUTER_API_KEY, OPENAI_API_KEY, and GEMINI_API_KEY were unset. No provider secret was set.

   Schema for the first graph:

   node Doc {
       slug: String @key
       body: String
       embedding: Vector(4)? @embed("body", model="model-a") @index
   }
   

   The graph was a local init at /tmp/repro-657/graph.omni.

   The mock does not put the model name into the vector. omnigraph embed of the same two rows with OMNIGRAPH_EMBED_MODEL=model-a and again with model-b wrote identical floats. Both summaries report dimension 4. The summary model is the process model (model-a, then model-b). Each output row has keys body, embedding, and slug. It has no model key. The distinct spaces in this run are a hand vector, a zero vector, and a later schema relabel. They are not two different mock geometries.

   What is stored

   schema show prints the column contract:

   embedding: Vector(4)? @embed("body", model="model-a") @index
   

   The accepted schema file on disk stores that source text and an IR embed_source. The IR value has the source property body and "model": "model-a". It has no provider, no model revision, no preprocessing id, and no source hash.

   The string model-a occurs in the schema history file and in __manifest data files, including the side-branch manifest copy. It occurs in none of the files under nodes/. The node data files name the columns body, embedding, and slug, and they carry #lance-schema:unenforced-primary-key. They do not carry a model, a provider, or a source hash on the vector column.

   A read of the three loaded rows returns d.slug, d.body, and d.embedding. The embedding value is four floats. The JSON has no marker beside it.

   Same length, more than one space

   Load of three rows, mode merge, exit 0. total_entities was 3. embedding_generation was "unsupported".

   slug	body	embedding
   from-a	alpha body	mock of alpha body
   hand	hand body from another space	[1.0, 0.0, 0.0, 0.0]
   zero	zero vector	[0.0, 0.0, 0.0, 0.0]

   A length-3 vector for slug short exited 1:

   {"error":"vector property 'embedding' expects 4 dimensions, got 3","code":"bad_request"}

   The next read still names the successful load as graph_commit_id. The rejected row did not publish.

   Text nearest for alpha body, with the embedder resolving to model-a, returned row_count 3. The order was from-a, zero, hand. The JSON keys were query_name, target, row_count, columns, rows, and graph_commit_id. There was no warning and no model field.

   The same text query with OMNIGRAPH_EMBED_MODEL=model-b exited 1:

   nearest() on 'embedding': its stored vectors were embedded with model 'model-a', but the query embedder resolves to 'model-b'. Set OMNIGRAPH_EMBED_MODEL='model-a' (and the matching provider) or re-embed the stored vectors.
   

   That sentence is not true for the stored rows. hand and zero were not produced by model-a. The check never read them.

   An explicit vector query for [1.0, 0.0, 0.0, 0.0], still with OMNIGRAPH_EMBED_MODEL=model-b, exited 0. row_count was 3. The order was hand, zero, from-a. The model mismatch did not apply.

   A query parameter of type Vector(3) exited 1 at typecheck, code T15:

   nearest vector dimension mismatch: property is Vector(4), query is Vector(3)
   

   Length is checked. Model identity is checked only for a text query, and only against the schema string.

   Source edit and model change

   This update publishes a vector that is not the embedding of the body:

   update Doc set { embedding: $v } where slug = $s
   

   $s was from-a and $v was [0.25, 0.25, 0.25, 0.25]. affected_nodes was 1. The receipt has commit and no coverage fields. After the commit, from-a.body was still alpha body. The embedding was [0.25, 0.25, 0.25, 0.25].

   This update changes the source text and leaves the vector:

   update Doc set { body: $b } where slug = $s
   

   hand.body became hand body edited. The embedding stayed [1.0, 0.0, 0.0, 0.0]. A text nearest for hand body edited returned row_count 3. The order was hand, from-a, zero. Nothing in the row or the result says the vector is stale.

   schema plan against a copy of the schema whose @embed model is model-b exited 0 with supported: false and one step:

   kind: unsupported_change
   entity: node:Doc.embedding
   reason: changing @embed source or model for 'Doc.embedding' is not supported in schema migration v1; rebuild the graph so stored vectors and their declared embedding space cannot diverge
   

   schema show after that plan still says model-a. The plan writes nothing and lists no rows. The hand vector and the zero vector stay in the field.

   Load, merge, export, and reopen

   export wrote 3 lines. Top-level keys were type, id, and data. Data keys were body, embedding, and slug. The file does not contain model-a, model-b, provider, source_hash, revision, or preprocess.

   A second graph was initialized with the same shape and model="model-b". load --mode merge of that export published 3 entities. embedding_generation was "unsupported". schema show says model-b. Text nearest for alpha body, with the embedder resolving to model-b, exited 0 and returned row_count 3. The stored floats were the hand vectors from the first graph, including [0.25, 0.25, 0.25, 0.25], [1.0, 0.0, 0.0, 0.0], and the zero vector. The new schema name is now what a text query treats as the space that produced them.

   On the first graph, branch create side succeeded. A load on side added slug branch-hand with embedding [0.0, 1.0, 0.0, 0.0]. branch merge side --into main reported outcome fast_forward. The next read had row_count 4. schema show still says model-a. The new vector has no marker.

   A new process then ran schema show and the text nearest again. The schema was still model-a. The query returned row_count 4, in order from-a, zero, hand, branch-hand. Reopen keeps the schema string and the floats. There is no per-value marker to keep.

   Difference from the report at 02d182b6 and a09176d7

   The reported gap is unchanged. Equal length still does not establish one embedding space. A hand vector of the right length is still stored and searched. The query-time model check still does not read the stored values.

   Three details differ in location or in a later flag. They do not add the missing marker.

   • The check cited at exec/query.rs line 410 on a09176d7 now lives in resolve_nearest_query_vec. The text branch still compares the client model with the schema model and uses the same error sentence. An explicit vector still returns with no model check.
   • validate_and_normalize_embedding on 02d182b6 still checks length, finite components, and nonzero norm, then normalizes. It still does not write a marker. The mock still records OMNIGRAPH_EMBED_MODEL and does not use it to build the vector. This run showed that: the two embed files match.
   • The schema-plan refusal of an @embed model change already exists at 02d182b6. This run shows that the refusal does not see a hand vector. Load JSON now includes embedding_generation. Here it was "unsupported". embeddings: managed population with observable pending, ready and stale coverage #655 records that flag. It is not a marker.

   The load of a zero vector is one more disagreement. The embedding client rejects a zero norm. The load stored [0.0, 0.0, 0.0, 0.0] and text nearest ranked that row.

   Root cause

   @embed is catalog metadata: a source property name and an optional model string (EmbedSource). The accepted IR stores that pair and nothing else (EmbedSourceIR). The Arrow field for the column is built with Field::new and no metadata map (catalog/mod.rs). The stored value is a fixed-size list of floats.

   The text query reads the schema model, not the row (resolve_nearest_query_vec, nearest_property_dim_and_model). The explicit-vector branch returns the list and does not call that check. Load rejects a wrong length (loader/mod.rs). Mutation does the same length check (mutation.rs). Neither path calls validate_and_normalize_embedding (embedding.rs), and neither stores a model or a source hash. The client normalizes and rejects a zero norm. The load path does not.

   schema plan refuses a change of the schema string (schema_plan.rs). A new graph with a different string accepts an export of the old floats. The schema string is then the only identity the text query has.

   The rules this gap breaks, and the class each rule covers:

   • Invariant 8. An integrity failure is loud. The class is every stored value whose meaning depends on a producer. The error text says the stored vectors were embedded with the schema model. This run shows rows that were not.
   • Invariant 7. Derived physical state may lag. It must not change the logical answer in silence. The class is every derived value, not only an index. A vector that no longer matches its source text is still a nearest hit.
   • Invariant 6. Stable identity is not inferred from a name, a length, or a current label. The class is every identity the engine claims. Here the embedding space is inferred from Vector(4) and from the current @embed model string.
   • Invariant 9. Query meaning lives in typed structures. The class is every check a result claims about stored data. The model check is a string compare at the start of a text query. It is not a property of the stored value, and the explicit-vector path does not use it.
   • Invariants 2 and 4. One publication is the door. A marker has to be written in that same publication as the vector. A later reader must not invent it from the current schema.
   • Invariant 12. The schema model and the row marker are different facts. The schema model is the space the field asks for. The marker is the space that produced the value. Using one string for both is how the relabel succeeded.

   Recommended fix

   Write the contract in an RFC before code. This issue is labeled needs-rfc. RFC 0015 is the draft for ingest reconciliation. Its registry status is draft and its implementation is not-started. Extend that RFC. Do not start with a queue. Do not call the provider inside load or mutate.

   The schema string stays the intended space. It is not proof about an existing float list. Each stored vector also carries a marker: the model id and a hash of the source bytes that produced it. Dimension stays on Vector(N). Provider and preprocessing belong in that model id when they change the vector. Gemini already sends a task type and the OpenAI-compatible request does not (embeddings guide). That difference is not stored today. If two providers can share one model string and still differ, the marker needs a provider id too. Do not fold that into the schema string.

   Put these rules in the RFC:

   1. A content publication does not call the provider. A caller-supplied vector is stored with the marker the caller supplies. A missing marker stays unknown. The load does not copy the current schema model onto that row.
   2. A supplied marker that differs from the schema model refuses the load. An external corpus uses an explicit contract: the field, or the load, names that external model, and the stored marker must equal that name. A bare float array is not that contract.
   3. A source-text change clears the vector and the marker in that same publication. Search then does not use the old floats. This is the NULL-on-change rule from RFC 0015. The marker is what makes a row that already has floats observable. embeddings: managed population with observable pending, ready and stale coverage #655 and embeddings: served backfill and re-embedding jobs with progress and resume #656 both need that fact.
   4. Ready means the marker model equals the schema model and the marker hash equals the current source text. Unknown means a non-null vector with no marker. Stale means a hash or model mismatch. Do not report unknown as the schema model.
   5. Text nearest keeps the current client-model check against the schema model. It also applies the marker. Rows that are ready are the verified hits. Unknown and stale rows are not described as that space. The result states the unverified count. An explicit query vector is a query of the declared length. It does not prove the corpus. The result does not claim those hits share the schema model unless their markers match.
   6. Existing graphs open. Default text nearest stops calling mixed or unknown rows one verified space. A documented legacy mode keeps today's ranking until an operator refreshes the rows. A refresh writes the marker only after it embeds the current source, or after an explicit publication that accepts the current floats as one named model. That publication is the external contract for vectors that are already loaded. It is not a silent relabel.
   7. schema plan keeps the refusal of an in-place model change until a migration publication writes new vectors and new markers in one graph commit.
   8. Export writes the marker. Load, branch merge, and a new process read it back. A merge keeps a vector only with its marker and only when the merged source text is the hashed source. Otherwise the merged vector is null and pending. Do not attach one side's floats to the other side's text.
   9. The reconciler in RFC 0015 publishes the vector and the marker together, and only when the live source hash still matches the hash that was embedded. embeddings: served backfill and re-embedding jobs with progress and resume #656 is the served progress surface for that reconciler. It is not a second writer.

   This choice spends correctness first. The row is larger by a small marker, and a NULL-only pending test is no longer enough. That is the right cut. A shorter float list that nearest treats as one verified space is a wrong answer. The cost is a marker format, a legacy mode, and a migration note for graphs that already have vectors. Those costs are smaller than a provider call on every content write, and smaller than a job queue.

   Other shapes, and why not

   • Put the model name only in Lance field metadata. Field metadata is one map for the column. This run already has a column-level model, in the schema contract. The hole is the value. Two rows in one column can still disagree.
   • Treat the current schema model as the marker when the marker is absent. That is the relabel this run already does, by loading an export into a graph whose schema says model-b.
   • Rely on the schema-plan refusal alone. It blocks an in-place rename of the schema string. It does not see the hand vector, and it does not run on a new graph.
   • Call the provider inside load and mutate. The content commit then waits on the provider. The deny-list rejects that shape. RFC 0015 already records the reconciler instead.
   • Let feat(cli)!: derive omnigraph embed vectors from the schema's @embed declarations #952 own this. That draft changes the file command so the schema is the spec. It does not write a marker into the graph.

   Liability

   This removes a false claim in the text-query error and a silent mix of spaces. It also removes the silent relabel on export and load. It adds a marker format and a legacy policy. A crash is safe only when the vector and the marker publish in one commit, and when a late write-back checks the live source hash. Unknown rows need the legacy mode so an old graph does not change ranking with no migration note. The error text must not say that every stored vector used the schema model until the marker says so.

   Lance

   Pinned Lance is 11.0.0 (released 2026-08-30). Later stable releases are v12.0.0 (2026-09-17) and v13.0.0 (2026-10-07). The newest prerelease seen here is v14.0.0-beta.19 (2026-10-16). None of these releases add an embedding model id, a source hash, or a check that equal dimensions share one model.

   Lance 11 already has the physical tools a marker would use. It does not define the marker.

   • A field can carry metadata (schema spec, Field metadata map). Dataset::update_field_metadata is in the pinned Rust source. That map is per field, not per row. OmniGraph already stores the column model in the schema contract. This run's node files do not even put that string on the Lance field.
   • The data evolution guide can add a column and backfill it, including a Python function that writes an embedding column and a checkpoint file for that function. The checkpoint file is not an OmniGraph commit. A new marker column can use this shape. Lance will not decide what the marker means.
   • The vector index spec records index type and distance type in lance:index. The pinned IndexMetadata struct has index_type and distance_type only. It has no model id. This graph's only index directory is the full-text index on slug. This run did not build a vector index.
   • Transactions are per dataset version (transaction spec). OmniGraph graph tables publish detached commits and one __manifest pin. A Lance metadata update is not the graph publication door.

   Later Lance work changes column update and index freshness. It does not close this issue.

   • lance#8351, in v12.0.0 and not in v11.0.0. Vector search drops stale index-segment rows before top-k. That is index entries, not the model that produced the stored floats.
   • lance#9134 and lance#9209, in v13.0.0 and not in v12.0.0. A small table may have an empty vector index, and a bad vector update literal is rejected instead of stored as null. Useful for a later write-back. They do not record a model.
   • lance#9523, in v13.0.0 and not in v12.0.0. A partial-schema fill is one projection.
   • lance#9742 and lance#9765, in v14.0.0-beta.19 and not in v13.0.0. They repair partial blob sources and join columns on horizontal updates.
   • lance#9950 and lance#10043 are merged after v14.0.0-beta.19. They fix UpdateIf struct names on indexed joins, and they prune unread target columns.
   • Open lance#9127 is cross-column vector refinement. Open lance#7954 is a transaction wire format. Neither defines embedding provenance.

   A search of open Lance issues and pull requests for an embedding model id or a source hash returned no match. Compare status above is behind_by == 0 when the tag contains the merge commit.

   Related work

   • #655. Managed population and pending, ready, and stale coverage. Its reproduction on this SHA shows that a source edit keeps the old vector and that nearest omits nulls with no count. The marker here is how "ready" can mean the current source and the current model. That issue does not replace this one.
   • #656. Served backfill. Its reproduction on this SHA shows that fill-missing keeps a hand vector and a stale vector, and that a text query does not read a model from the row. The refresh mode needs this marker. That issue does not replace this one.
   • #874. OpenAI-compatible role hints and batched omnigraph embed. The hint changes vectors for some models. That is another preprocessing fact the marker must be able to name. It is not the stored-vector check.
   • #952. Draft. The file command would read @embed from a schema file and compare the provider model with that name before it reads input. It still writes JSONL. It does not store a marker.
   • #648. Graph-wide discovery. It asks search to show missing embeddings and incompatible models. Those facts belong on the retrieval descriptor. They need this marker before a count can be true.
   • RFC 0015, RFC 0047, and draft PR #793 (RFC 0048). They sketch reconciliation and embedding_coverage. The query JSON from this run has no coverage field.

   Supersession

   No open issue replaces this one. #655 is the coverage contract. #656 is the served job. Both say this issue is the identity check they need. #952 changes the file command only. This issue stays open.

   Limitations

   • The embedder is the mock. There was no second real model and no provider secret. The mock vectors for model-a and model-b are identical. The mixed space is the hand vector, the zero vector, and the schema relabel.
   • The graph is a local file. This run did not use S3, Azure, or a server. The direct engine is the writer for those paths. embeddings: managed population with observable pending, ready and stale coverage #655 and embeddings: served backfill and re-embedding jobs with progress and resume #656 already cover served load and mutate on this SHA.
   • The merge was a fast-forward that added one row. This run did not edit one id's vector on both branches and then merge.
   • Reopen means a new CLI process on the same files. It does not mean a server restart.
   • This run did not build a vector index. The index-metadata claim is from the Lance 11 source and the vector index spec.
   • The set has 4 rows and dimension 4. I did not add a test to the tree. The proof is this CLI run.