Repository navigation
fix(catalog): a publication that never sent its manifest is unpublished - #1092
Conversation
A served write whose publisher failed before its conditional manifest put was reported in doubt, and the server contained itself. On 2026-10-19 a throttled `__manifest/_transactions/*.txn` PUT did this: Lance writes the transaction file before the manifest and returns its error with a bare `?`, so no manifest request was sent, but the publisher's read-back rule called absence definite only for a failure before any store request. Every OmniGraph dataset commits through one configured commit handler. It now re-issues an ambiguous manifest put-if-absent unchanged, up to four times, until the store answers it; a held version goes back to Lance's transaction read-back, which tells this commit from another. A per-commit probe records whether a request was sent and how it ended. The catalog publisher derives each failure's completion evidence from that record and the attempt's phase: no request sent, or one the store refused, is `Unpublished`; a request that may have applied is read back, and only an outcome the read-back cannot settle stays in doubt. Every other publisher failure, candidate writes included, is `Unpublished`; a candidate tag whose create may still land is in doubt. The server's `ApiError` carries the engine failure's redacted class, and a contained owned write logs it before admission closes.
The handler result frame is sized by ApiError, which a guard keeps at four words. The redacted cause is logged where the engine error is translated, and a contained owned write logs its status once.
…uests A slot rule in the Lance-realm store fails, throttles, loses the acknowledgement of, or lands late the graph's publication requests, which the seeded weather cannot target. The scenario requires each to end committed once, unpublished, or in doubt with nothing visible, and the next write on the handle to publish.
The re-issue logs named the dataset's base path and the store's error text, which can carry object URIs. They now record the request's object-store error class, as every other server storage log does.
The deployment page is at its length cap. Storage outcomes get their own operations page, which the deployment page's shutdown section points to.
…me-from-commit # Conflicts: # crates/omnigraph-core/src/lance_clone.rs # docs/dev/lance.md
|
This draft fixes the incident class at the right layer. It should stay a draft until the handler flag and the rebase below are done. A graph commit is visible only when the next The re-issue of an ambiguous manifest create uses the same bytes and the same put-if-absent. At most one copy applies. A lost acknowledgement that comes back as "already exists" is a conflict, and Lance compares the full transaction. Exhaustion stays in doubt. That part matches the contract. Blocking1. An unresolved manifest put still lets Lance delete the transaction file
The probe already says the put may still land. This flag is the hook Lance reads. The two disagree. I reproduced it on this head. A catalog publish answered every manifest create with an ambiguous timeout, and wrote the first create's payload for real five seconds later.
A commit small enough to inline its transaction still opens. The storage scope refuses a later delete of Fix this in the handler. That type owns the request record. When this attempt ended The catalog path can read the per-commit probe.
Suggested inline on 2. Rebase onto current
|
…nce 13 Lance 13's WrappingObjectStore also wraps paginated listings; the fault aims at puts only.
…t may land After the last re-issue the handler returned the store's error and kept reporting a missing version as definitive, so Lance read the version back as absent and deleted the attempt's transaction file. A manifest that lands later then names a file that is gone. Only the server's storage scope held that delete back; embedded and CLI opens did not. The handler now records the task whose request no re-issue settled, the task Lance reads the version back on, and reports a missing version as not definitive there. Lance returns CommitStatusUnknown and keeps the file. A refused request and a commit that sent none keep the inner answer.
|
Both blocking findings are done; head 1. Unresolved manifest put let Lance delete the transaction file. Fixed in the handler (
2. Lance 13
Optional,
Validation on
|
|
Ready to merge. There are no blocking findings. There is one optional finding. A graph commit becomes visible only when the next This change keeps one record per commit of whether a manifest request was sent and how it ended. The handler sends an unanswered request again, with the same bytes and the same put-if-absent, up to four times. At most one copy can apply. No request, or a request the store refused, is The record lives on the commit handler. Every dataset commits through that handler. The catalog publisher reads the record. It does not infer the outcome from the error text. Earlier findings1. An unresolved manifest put let Lance delete the transaction file. Resolved.
On this head I made that method return the inner answer. 2. Update onto Lance 13 The branch contains the merge of 3. The tag is the only address of that commit. BlockingNone. OptionalThe shared dataset handler has The catalog path does not use this queue. A mark that outlives its attempt only makes a later failed commit on that task keep an orphan transaction file. Cleanup can reclaim an orphan file by age. A caller with no task id shares one slot, with the same effect. Keep a task id that may still be inside the read-back. The catalog probe can stay as it is. Inline, Tradeoff and liabilityThe probe is the one record of the manifest request. Later outcome work should extend that record. A second reading in the publisher would drift from it. The re-issue adds up to four identical puts, and about 3 seconds of pause (0.2 s, then double, to 1.6 s), before an unresolved publish stays in doubt. The server still stops for that case. #1093 decides what replaces that stop. #1096 contains this head (5 commits ahead, 0 behind) and should merge only after that RFC. #1097 is based on #1096. #1017 does not change this contract. #1067 stays open for the RFC. #1070 remains the earlier rule that a definite failure before publication does not stop the server. ValidationHead: GQT cannot plant an object-store fault on the transaction-file put, and it cannot land a manifest after give-up. The catalog tests own this contract. No GQT case was added. After, on the head ( 5 catalog tests passed. 5 core tests passed. Before, same head, two temporary source edits. Nothing was pushed.
Limitation: that second test plants a file where Both tests passed again after the source was restored. CI on this exact head is green (Actions run 39458692106, created 2026-10-20T16:34:52Z). That includes Test Workspace, clippy, rustfmt, the DST pinned suite, GQT, RustFS, and Azurite. I did not re-run those jobs. I did not run the engine Lance source checked: crates.io RecommendationReady to merge. |
Lance deletes an attempt's transaction file when the handler calls a missing version definitive, and reads that answer with no reference to the attempt. The per-task marks that stood in for one relied on Lance reading the version back on the task that sent the request. The catalog's per-commit handler now answers from its probe, keeping the inner answer unless the request may still land. A dataset's shared handler holds no record of the attempt Lance reads back, so it never calls a missing version definitive; Lance reports the status unknown and the collector's orphan sweep reclaims a table's leftover transaction file.
|
Follow-up to the flag fix, now at head Why. Rule. Only the commit's own request record may call a missing version definitive.
Validation of the Lance 13 assumptions.
Tests.
Run on
|
|
@aaltshuler thanks: both commits are kept, and both fit the design, which gives each fact about a request one owner.
Both are merged into #1096 ( |
Storage outcomes per operation dropped its pending-publication registry: an unknown publication is recorded nowhere and blocks nothing, and its version key fences it. A control fact follows the same fence, and the one-branch service_unavailable row goes. Read-back follows ModernRelay#1092's split, a not-ready branch's absent listing fields are named, and the access evidence uses RFC 0059's target kinds.
|
Ready to merge. There are no blocking findings and no optional findings. WhyA graph commit is visible only when the next What changed
The rule
Lance 13.0.0 reads this flag only inside A false answer makes Lance return The catalog builds its handler with A shared handler's refused commit also keeps the file. Local errors and candidate tags
After a candidate tag exists, a failed authority probe stays in doubt unless the read set changed or the branch is gone. Dependencies
Earlier findings
LiabilityOne request record owns each fact. The catalog probe owns the catalog answer. A shared handler owns no attempt, so it refuses to call a version absent. Later outcome work should extend that record. A second heuristic on the publisher would read the same put twice. The accepted cost is orphan transaction files on refused table commits, until the seven-day sweep, and a short verification retry. An ambiguous local error can stop the server. That is the safe side of the 2026-10-19 class. ValidationPublication logic was executed at After, clean Exit 0. Catalog: 3 passed. Core: 5 passed. The same command passed again after the temporary edits were restored. Before, same commit, temporary source edits, nothing pushed:
GQT cannot plant the store fault, a late manifest, or completion evidence for an operation that returns no candidate id. No GQT case was added. I did not run CI on RecommendationReady to merge. |
Brings in main through ModernRelay#1076: the ModernRelay#1074 squash, ModernRelay#1092 and ModernRelay#1103.
* docs(rfc): propose local graph sessions and client access Draft RFC for issue ModernRelay#980: `omnigraph serve` local sessions with session records, a thin `omnigraph studio` launcher, `omnigraph mcp` for agents, and `omnigraph credential` for static tokens. Wider-blast-radius changes are listed in Appendix A with their filed issues. * docs(rfc): add installing Studio as a later, signature-gated phase * docs(rfc): link Studio's release prerequisites and correct their current state * docs(rfc): reconcile local sessions with main and open PRs, and define the target writer * docs(rfc): name local servers and settle records, tokens and agent paths Rename the session to a local server and --session to --via-server. Move server records from /tmp, which macOS ages out after three days, to per-user directories resolved from the account, with the machine they belong to. Never write a token to a terminal, list live local servers in targets --json, give each agent context one write path, add platform coverage, and gather every refusal code in one table. * docs(rfc): redesign graph access around seven rules and fold in cross-process writes Rewrite the RFC from first principles: name the target, never the path; correctness in the engine across processes; one set of operations on three surfaces; one serving concept with explicit audiences, folding the omnigraph-server binary into omnigraph serve; --as as a declaration or an assertion; one outcome model and exit table; and introspection through target list and access. Bring cross-process writer safety and the access route into scope, with the engine design on the __manifest version CAS, and land the interfaces before the engine phase. Rename the file to match the new title. * docs(rfc): align outcomes, serving lifecycle and late writes with the storage-uncertainty fix * docs(rfc): elect one local server per destination and close the reuse, handoff and MCP gaps from review * rfc: align graph access with the phase 1 and phase 3 implementations The actor header rides the HTTP contract bump to 0.14 instead of a capability response header, and /mcp compares it after authentication alone. Rule 6 states the outcome table the server implements, with an earlier publication marked on the engine's error and outcome.check waiting for an attempt id. The local embedding API names what the implementation settled: LocalBinding::local for local roots, ReadWriteUnavailable, ClusterGraph instead of promoting a graph to its cluster, start_local and LocalServing::release, and one captured ledger revision per read-only cluster server. * rfc: align graph access with the disposition, the If-Actor header and the records' machine id Phase 1's review renamed the error object and the actor header, and phase 3's review tightened the records' machine id. The RFC now says what the implementations do: - The actor header is Omnigraph-If-Actor, after HTTP's conditional If-* headers, since a mismatch is a failed condition (412). - The error object is `disposition`: `execution` is gone because it only repeated `effects`; `effects` gains `partial` for a refused step after an earlier publication of the same operation; `action` gains `fix` for a request that fails the same way on resend. Each code names its own action, and partial effects exit 2. - A record's machine is 32 lowercase hexadecimal digits that are not all zeros. Any other value, systemd's first-boot `uninitialized` marker included, refuses rather than falling back to another source. - Windows records wait for their own pull request: until they check the directory's access list and read MachineGuid, they refuse. - The CLI's MCP bridge bullet lists what the bridge actually changes: the graph binding, the readOnlyHint filter and local cancellation. * rfc: decide branch control through control facts each publish attempt reads The review of c4641d7 found the engine section did not match the publisher on main: a row change must record a graph commit, and the preconditions read only the head and branch identity, so a head-preserving tombstone or drop record stopped nobody. - Control facts (tombstone, registration, drop record) change no table row and record no commit; check_publish_precondition reads the facts that name an attempt on every attempt, for every precondition. - A create registers on main before it writes its ref, so the registration is the visibility point and cleanup reads it instead of an age. Until PR 963 lands, a registration refuses when main's contract moved past its fork; after it, PR 963's rule wins. - Only the upgrade census raises the new stamp. - RFC 0059 (PR ModernRelay#1090) owns addressing and the target and credential commands; deployments pass their root through --target. - The access route is ModernRelay#1073's; this RFC adds the command. A direct graph inside a cluster refuses and names the root. - Storage outcomes are PR 1093's; storage_unavailable goes, and disposition.check is a later phase. * docs(rfc): discontinue the second serving model * docs(rfc): record MCP bridge closure and defer selective reuse * rfc: treat RFC 0059's implementation as merged ModernRelay#1090 landed on main, so the RFC no longer waits on it: blocked_on is empty, and the Motivation, rollout and Appendix A5 say the target commands build on main. * rfc: accept graph access and cross-process writes Maintainer decision: the actor assertion, the error-code registry and disposition, the exit table, the access command, operation parity, the existing-server MCP bridge and Studio opener, and the engine control-fact design with its qualification gates. Appendix A stays separate decisions. * rfc: give every control fact its attempt id and gate the upgrade on registrations The review of eeb6e12 found two holes in the engine section: - A control fact records no graph commit, so the head-equality read-back cannot tell two facts apart after a lost acknowledgement, and the owner's own retry met its refusal. Each control-fact operation now mints one attempt id that every fact it writes records: the read-back compares it, a pending control fact is keyed by it, and a retry treats its own fact as landed. Deletes and drops converge. - System-column upgrade pins main's head, not its version, so a registration did not stop it. Its attempts now read main's registrations every attempt, until PR 963. Cleanup's registration rule now covers refs as well as trees, under the collector's dependency closure; a create no longer completes a crashed create's registration; the pending-branch service_unavailable covers one branch; branch list shows a registered branch without a ref as not ready; the --as help quote is current. * rfc: name the access command's test owner and target terms * rfc: settle a control fact from the fact, and keep the collector's keep set The review of ef44175 recommended merging, with four optional corrections: read-back has three results; a caller settles a control fact from the fact, since the head does not move; cleanup candidates still pass protected_manifest_branches and retained_merge_bases; a crash loses the attempt id, so only delete and drop converge after one. A not-ready branch's listing fields come from its registration. * rfc: fence an unknown control fact by its version key Storage outcomes per operation dropped its pending-publication registry: an unknown publication is recorded nowhere and blocks nothing, and its version key fences it. A control fact follows the same fence, and the one-branch service_unavailable row goes. Read-back follows ModernRelay#1092's split, a not-ready branch's absent listing fields are named, and the access evidence uses RFC 0059's target kinds. * rfc: align the access view field and disposition.check with actor views * rfc: settle a control fact when its attempted version is readable --------- Co-authored-by: aaltshuler <andrew@collectivelab.io>
What & why
A publisher failure no longer stops the server when no manifest request can land. Every OmniGraph dataset commits through one configured
CommitHandler. That handler now re-issues an ambiguous manifest put-if-absent, unchanged, until the store answers it. It records, per commit, whether a request was sent and how it ended. The catalog publisher uses that record and the attempt's phase to set completion evidence on every failure, instead of inferring it from where the error came from.This closes the publisher half of #1067. It fixes the 2026-10-19 containment, which #1070 did not cover.
0.14.0-edge+0a62bb7served onemutateonmain. Tigris answered the__manifest/_transactions/5268-….txnPUT503 SlowDown. Lance writes that file before the manifest and returns its error with a bare?(lance-11.0.0/src/io/commit.rs:1397), so no manifest request was sent. The publisher read version 5269 back as absent, but its rule called absence definite only for a failure "before any store request" (publisher.rs:1455), so it answeredmanifest_publish_in_doubt. The server then closed admission for every graph, exited 2 with the lock held, and stayed down about 16 hours.Unpublished. The client gets a 500 and the server keeps serving.The full root-cause chain for both incidents, with line references at
main, is in the Motivation of the storage outcomes RFC, #1093.Backing issue / RFC
issue_1067regressions. bug: one storage write with an unknown outcome stops the server until a manual unlock #1067 stays open for the decisions in the storage outcomes RFC (rfc: storage outcomes per operation and restartable ownership #1093). This change needs no RFC, because it computes the existingUnpublishedand in-doubt evidence more precisely under the current owned-operations contract.Changes
omnigraph-core::manifest_request(new).ManifestRequestProberecordsNotSent,Refused,Held,LandedorUnresolved. A request dropped in flight isUnresolved.classify_failed_requestreads the typedobject_storeerror:InMemory, a refusal by the owning storage scope (OWNED_STORE_NAME), not found, an invalid path, an unsupported operation;Genericerrors and a 403. A Windows manifest rename can create the destination before temporary-file cleanup fails.lance_clone. The configured handler re-issues an ambiguous manifest put up to 4 times, pausing 0.2 s and doubling to 1.6 s. Each re-issue keeps the same manifest and precondition. A held version goes back to Lance asCommitConflict, soverify_commit_outcomecompares the complete transaction there and tells this commit from another.probed_commit_handlerattaches a probe for one commit.omnigraph-catalog::commit.overwrite_placedcommits with a probed handler. It marks a failureUnpublishedwhen no request can land.merge_rowsreports whether a failure came before or after the store's acknowledgement.Unpublishedis returned as is, and only an opaque error with no evidence is read back.Unpublished, including preparation and history archiving. Candidate failures after tag creation remain in doubt unless captured authority is proven expired; a successful rollback deletion cannot fence an earlier conditional-create retry.Uncertain.RedactedCausewhere the server maps it, and a contained owned write logs its status before admission closes.ApiErrorstays four words; its size guard keeps the handler result frame small.request_error_class).PublicationFault, aims at the two publication requests the seeded weather cannot target: the manifest put-if-absent and the__manifesttransaction file. It can fail a request unanswered, throttle it, lose its acknowledgement, or land it late.operations/storage-failures.md, covers publication outcomes and logs;deployment.mdis at its 350-line cap and now points to it.writes.mdandcontrol-plane.mdare updated, andlance.mdgets a new fence row.publication-failures-before-the-manifest-are-definite.fixed.md.Tests
On
main(9e2425d2),transaction_file_failure_before_the_manifest_is_unpublished_issue_1067fails with the incident's message (manifest publish outcome is in doubt … _versions/2.manifest was not found); it passes here, as do the rest.manifest_request::tests: an unresolved request is dropped, refusals are told from ambiguous failures, the pauses are bounded, and the logged error class carries no path.lance_clone::tests::manifest_request_is_re_issued_until_the_store_answers_issue_1067.transaction_file_failure_before_the_manifest_is_unpublished_issue_1067lost_manifest_acknowledgement_resolves_by_re_issue_issue_1067abandoned_manifest_create_that_lands_late_publishes_once_issue_1067manifest_create_without_a_definite_answer_stays_in_doubt_issue_1067writes::publisher_failure_before_the_manifest_put_is_definite_issue_1067.boot_settingsdefinite_write_failure_keeps_serving_and_releases_the_lock_issue_1067plants a fault under a real server's__manifest/_transactionsand expects a 500, then a 200 from the same PID.dst_publication_requests_get_a_definite_outcome_issue_1067runs five cases:Unpublished;Review regression coverage
candidate_tag_reappearing_after_rollback_stays_uncertain_issue_1067: reuse the publication fault fixture to hold a duplicate conditional tag create, fail the authority read, confirm rollback deletion, then land the earlier create. Both proposal and unattached variants remain uncertain; the proposal can still publish and the snapshot can still be read and extended.Checklist
__manifestcommit. A re-issue is the same conditional request, logged, bounded and never a replay of a reported outcome. Evidence is only ever narrowed from in doubt to definite where the request record proves it. Explicit uncertainty is never erased.Review-fix verification
At the reviewed base
b017cc39, the new/extended core and candidate tests fail for the predicted reasons: a persisted local manifest is called refused; replayed candidate tags remain publishable/readable but the operation claimsUnpublished; the engine releases merge input holds. With these fixes, on macOS arm64 andRUST_MIN_STACK=16777216:omnigraph-catalog/failpoints: 295 passed (82 core, 213 catalog).merge_as_commit_probe_failure_retains_inputs_issue_1067.cargo fmt --all --check,git diff --check,typos, documentation/link checks, andcheck-ci-cells.py --self-testpass.CI security dependency fix
Fresh CI for
faeef19efound three newly reported TLS advisories in dependencies already in the lockfile. The follow-up changes only these package versions/checksums and adds a release note:rustls-webpki0.103.15 → 0.103.16 (RUSTSEC-2026-0382).webpki-root-certs1.0.8 → 1.0.10 (RUSTSEC-2026-0383).webpki-roots1.0.8 → 1.0.10 (RUSTSEC-2026-0381).Cargo resolved the updates within existing dependency constraints; no Cargo manifests, allowlists, or gate rules change. The dependency-source guard and docs checks pass. The full core/catalog unit suites pass again with this patched lockfile (295 tests), as does focused core/catalog clippy with failpoints. The fresh CI dependency guard passes;
cargo-denyis not installed locally.Earlier local verification
The following records the author's validation before these review fixes, with
RUST_MIN_STACK=16777216on macOS arm64.cargo test -p omnigraph-engine -p omnigraph-server -p omnigraph-cluster -p omnigraph-core -p omnigraph-storage -p omnigraph-catalog --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints --locked:api_error_tests,operations::and theboot_settingsowned_shutdownjourneys;manifest_requestandlance_clone.crates/omnigraph-dst:dst_publication_requests_get_a_definite_outcome_issue_1067,dst_lance_realm_ack_loss_bites_and_oracles_hold,dst_fault_storm_on_one_live_handle_keeps_writing,dst_memory_graph_end_to_end;dst_bench_cost_count_golden, unchanged.cargo fmt --all --check; both workspace clippy graphs (with and without failpoints);scripts/check-docs.py;scripts/check-agents-md.sh;typos.After merging
main(Lance 13.0.0, #1072)main'sManifestLocationCommitHandler; the probe and re-issue are unchanged.lance-13.0.0/src/io/commit.rs). It still writes the transaction file first with a bare?.ConditionalPutCommitHandlernow reports a missing version as definitive at read-back. A failed manifest put therefore ends at its first read-back, while an abandoned put can still land afterwards. So the re-issue still runs before Lance reads back, and thelance.mdfence row says so.WrappingObjectStorealso wraps paginated listings; the catalog test fault passes them through (64e7d339).b017cc39): only a per-commit probe can make a missing version definitive. A shared handler never does, so Lance keeps transaction files that a late manifest may name without per-task state or eviction. Regressions:unresolved_manifest_create_keeps_its_transaction_file_issue_1067anda_shared_handler_never_calls_a_missing_version_definitive_issue_1067.11b391f7, with the review fix below);cargo fmt --all --checkand both clippy graphs.Notes for reviewers
__manifest.ApiError::from_omni, so expect a small conflict there.ApiErroritself does not grow.