Repository navigation
fix(server): scope an unknown storage outcome to its key and branch - #1096
Conversation
A served write whose publisher failed before its conditional manifest put was reported in doubt, and the server contained itself. On 2026-10-19 a throttled `__manifest/_transactions/*.txn` PUT did this: Lance writes the transaction file before the manifest and returns its error with a bare `?`, so no manifest request was sent, but the publisher's read-back rule called absence definite only for a failure before any store request. Every OmniGraph dataset commits through one configured commit handler. It now re-issues an ambiguous manifest put-if-absent unchanged, up to four times, until the store answers it; a held version goes back to Lance's transaction read-back, which tells this commit from another. A per-commit probe records whether a request was sent and how it ended. The catalog publisher derives each failure's completion evidence from that record and the attempt's phase: no request sent, or one the store refused, is `Unpublished`; a request that may have applied is read back, and only an outcome the read-back cannot settle stays in doubt. Every other publisher failure, candidate writes included, is `Unpublished`; a candidate tag whose create may still land is in doubt. The server's `ApiError` carries the engine failure's redacted class, and a contained owned write logs it before admission closes.
The handler result frame is sized by ApiError, which a guard keeps at four words. The redacted cause is logged where the engine error is translated, and a contained owned write logs its status once.
…uests A slot rule in the Lance-realm store fails, throttles, loses the acknowledgement of, or lands late the graph's publication requests, which the seeded weather cannot target. The scenario requires each to end committed once, unpublished, or in doubt with nothing visible, and the next write on the handle to publish.
The re-issue logs named the dataset's base path and the store's error text, which can carry object URIs. They now record the request's object-store error class, as every other server storage log does.
The deployment page is at its length cap. Storage outcomes get their own operations page, which the deployment page's shutdown section points to.
A storage request with no definite answer stopped the whole process: the storage scope kept one process-wide uncertain flag, and an in-doubt publication or an opaque storage error contained the server until an operator unlocked it. - The storage scope tracks unsettled mutable keys instead of one flag. A key whose request has an unknown outcome refuses later requests to that key, including a recursive delete over it, and nothing else. The set is bounded; past the bound every mutation is refused, and a clean release still requires it empty. - Requests route by class. Reads and fresh immutable writes go through a retrying client within Lance's defaults; conditional requests and mutable keys keep one attempt per request, and their owners resolve ambiguity. - A `__manifest` commit whose manifest request no re-issue settles is pending on its branch. The process keeps the base and the exact Lance transaction; the branch's next publication re-issues it once and refuses with the typed `PublicationPending` while storage stays silent. The server re-issues pending commits on a backoff and once more at clean shutdown, which retains admission if any stay pending. - The server contains only uncertainty the engine does not scope: a panic, a lost owner, `RecoveryRequired`, an opaque execution failure and an uncertain activation. A pending publication answers 503. - A guard registers every unconditional request to a reused key across the crates that write storage, with why its late landing is harmless.
|
Reviewed WhyOne storage request with no answer stops the whole server today. This commit keeps that server up. It limits the doubt to the key or the branch that the request touched. WhatA manifest create that stays unanswered is pending on its branch. The process keeps the base dataset and the same Lance transaction. The next publication on that branch re-issues that commit once. While the store stays silent, later writes get HTTP 503 How
ImpactA throttled fresh file or an unanswered manifest no longer takes every graph down. A pending write can still become visible after the client got HTTP 500. A clean shutdown that still has a pending publication or an unsettled key keeps the cluster lock. Restart without an operator still needs the lease. This commit does not add the lease. Fix before this leaves draft
Fail closed. If
Put the resolver in the same drain as admitted writes. Refuse root deletion while that root has a pending publication.
Optional
Tradeoff and liabilityThis removes the failure mode where one request closes every graph. It adds a process-global registry of up to 64 The new liability is the resolver and the ignored TestsGQT cannot express this contract. A Same contract, before and after: a lost acknowledgement on one mutable key must still admit a different mutable key.
I did not run the server operations tests, the DST scenario, clippy, or fmt. I did not run the catalog tests on Recommendation: keep this in draft. Fix the registry failure and the resolver drain, then rebase onto Lance 13. |
…me-from-commit # Conflicts: # crates/omnigraph-core/src/lance_clone.rs # docs/dev/lance.md
…tion-storage-outcomes
…nce 13 Lance 13's WrappingObjectStore also wraps paginated listings; the fault aims at puts only.
…tion-storage-outcomes
…t may land After the last re-issue the handler returned the store's error and kept reporting a missing version as definitive, so Lance read the version back as absent and deleted the attempt's transaction file. A manifest that lands later then names a file that is gone. Only the server's storage scope held that delete back; embedded and CLI opens did not. The handler now records the task whose request no re-issue settled, the task Lance reads the version back on, and reports a missing version as not definitive there. Lance returns CommitStatusUnknown and keeps the file. A refused request and a commit that sent none keep the inner answer.
…tion-storage-outcomes # Conflicts: # crates/omnigraph-catalog/src/tests.rs # docs/dev/writes.md
Lance deletes an attempt's transaction file when the handler calls a missing version definitive, and reads that answer with no reference to the attempt. The per-task marks that stood in for one relied on Lance reading the version back on the task that sent the request. The catalog's per-commit handler now answers from its probe, keeping the inner answer unless the request may still land. A dataset's shared handler holds no record of the attempt Lance reads back, so it never calls a missing version definitive; Lance reports the status unknown and the collector's orphan sweep reclaims a table's leftover transaction file.
A publication whose manifest request no re-issue settles was kept in a process registry, refused its branch with a 503 and was re-issued by a server resolver until storage answered. That registry repeated a guarantee the version key already gives: the manifest version is created only if absent, so the request lands before the branch's next publication or never. The registry also landed a crashed operation's commit under a rerun that had already captured its authority, so the rerun failed instead of converging. The registry, its error, the resolver and the shutdown hold are gone. An unknown publication answers in doubt and the branch publishes on: the next publication finds the late commit or takes its version first. A regression test covers both orders.
|
Design change, now at head Why. CI's DST crash-state enumeration ( The registry was also redundant. The manifest version is created only if absent, so an unresolved request lands before the branch's next publication or never:
What's left. One owner per fact:
Removed: Evidence on
|
…tion-storage-outcomes # Conflicts: # crates/omnigraph-catalog/src/tests.rs # docs/user/operations/storage-failures.md
…t' into fix/per-operation-storage-outcomes
…rage-outcomes # Conflicts: # crates/omnigraph-catalog/src/tests.rs # crates/omnigraph-server/src/lib.rs # crates/omnigraph-storage/src/scoped.rs # docs/dev/writes.md # docs/user/deployment.md # docs/user/operations/storage-failures.md
|
Reviewed WhyOne storage request with no answer used to stop every graph. This commit keeps the server up. The doubt stays on the key the request named, or on the manifest version the publication used. What
HowS3 and Azure use two clients. Reads and fresh immutable writes retry inside Lance's bounds (three client retries and three throttle retries). Conditional requests and mutable keys use one attempt. ImpactA lost acknowledgement on one tag still admits a different mutable key. A client that repeats a non-idempotent write after HTTP 500 can apply it twice while the late manifest can still land. The response text tells the caller to look the commit up first. The structured body Earlier review of
|
Main made ApiError's status a method; the owned-write log line read it as a field.
|
Merged
Verification on
|
Accepted on the maintainer's instruction after review approved the RFC, ModernRelay#1096 and ModernRelay#1097. The owned server operations and engine settlement RFCs get disposition notes for the parts this replaces: unknown storage outcomes no longer close admission, ambiguous storage outcomes are fenced by re-issue and the version key instead of fail-stop, and a server's S3 lock is leased. Section 7 also says, per the approving review, that a claim proves nothing about its writer and that the owner's timer, not the claim, is the fence.
|
Merged
Verification on
Re-queued. |
Stacked on #1092. Implements §§1–6 of the draft RFC #1093 and should merge only after that RFC is accepted.
What & why
A storage request with no definite answer no longer stops the server: its uncertainty is scoped to the key it wrote. This closes steps 1, 2, 3 and 5 of #1067 for the served write path. #1092 closed the publisher half of the 2026-10-19 containment.
The design gives each fact about a request one owner, and every consumer reads it from there:
The changes:
ApiError::contains_processseparates an unknown storage outcome the engine scopes from one it does not.RecoveryRequired, an opaque DataFusion failure, and an uncertain deployment activation.StorageIoScopedrops its process-wideuncertainflag.OBJECT_STORE_CLIENT_MAX_RETRIESstays.forbidden_apis::mutable_key_requests_are_registered_issue_1067registers every unconditional request to a reused key, with why its late landing is harmless. It covers the engine, core, catalog, storage, cluster, server and Azure admission crates.What changed from the first draft. That draft kept each unresolved publication in a process registry. It refused the branch with a 503 and re-issued the publication from a server resolver.
unknown_publication_lands_before_the_next_publication_or_never_issue_1067covers both orders.Backing issue / RFC
Checklist
scoped::tests::*_issue_1067: per-key refusal, bound, routing;unknown_publication_lands_before_the_next_publication_or_never_issue_1067;operations::testscontainment matrix;operations/storage-failures.md: unknown publications, retries, when the server stops, logs;deployment.mdpointers, within its line cap;control-plane.mdandwrites.md;changelog.d/storage-outcomes-per-operation.changed.md.__manifestcommit with no second record of it; retries are class-bounded and logged.Local verification
Head
d1f65c5e, which carries #1092 atb017cc39andmainwith Lance 13. Run withRUST_MIN_STACK=16777216on macOS arm64; all pass.-p omnigraph-engine -p omnigraph-server -p omnigraph-cluster -p omnigraph-core -p omnigraph-storage -p omnigraph-catalog --features omnigraph-engine/failpoints,omnigraph-cluster/failpoints): 2725 passed, 0 failed. That includesforbidden_apiswith the mutable-key guard.dst_crash_state_enumeration_sampled, which failed in CI with the registry;dst_publication_requests_get_a_definite_outcome_issue_1067;check-docs.py,check-agents-md.shandtypos.Notes for reviewers
outcome_unknownon MCP) and the server keeps serving. feat(server)!: say on every error body whether the request ran #1076 and rfc: graph access and cross-process writes #1017 carry the typed disposition on the wire.