Repository navigation
feat(cluster)!: take over a silent server's S3 lock through a lease - #1097
Conversation
A served write whose publisher failed before its conditional manifest put was reported in doubt, and the server contained itself. On 2026-10-19 a throttled `__manifest/_transactions/*.txn` PUT did this: Lance writes the transaction file before the manifest and returns its error with a bare `?`, so no manifest request was sent, but the publisher's read-back rule called absence definite only for a failure before any store request. Every OmniGraph dataset commits through one configured commit handler. It now re-issues an ambiguous manifest put-if-absent unchanged, up to four times, until the store answers it; a held version goes back to Lance's transaction read-back, which tells this commit from another. A per-commit probe records whether a request was sent and how it ended. The catalog publisher derives each failure's completion evidence from that record and the attempt's phase: no request sent, or one the store refused, is `Unpublished`; a request that may have applied is read back, and only an outcome the read-back cannot settle stays in doubt. Every other publisher failure, candidate writes included, is `Unpublished`; a candidate tag whose create may still land is in doubt. The server's `ApiError` carries the engine failure's redacted class, and a contained owned write logs it before admission closes.
The handler result frame is sized by ApiError, which a guard keeps at four words. The redacted cause is logged where the engine error is translated, and a contained owned write logs its status once.
…uests A slot rule in the Lance-realm store fails, throttles, loses the acknowledgement of, or lands late the graph's publication requests, which the seeded weather cannot target. The scenario requires each to end committed once, unpublished, or in doubt with nothing visible, and the next write on the handle to publish.
The re-issue logs named the dataset's base path and the store's error text, which can carry object URIs. They now record the request's object-store error class, as every other server storage log does.
The deployment page is at its length cap. Storage outcomes get their own operations page, which the deployment page's shutdown section points to.
A storage request with no definite answer stopped the whole process: the storage scope kept one process-wide uncertain flag, and an in-doubt publication or an opaque storage error contained the server until an operator unlocked it. - The storage scope tracks unsettled mutable keys instead of one flag. A key whose request has an unknown outcome refuses later requests to that key, including a recursive delete over it, and nothing else. The set is bounded; past the bound every mutation is refused, and a clean release still requires it empty. - Requests route by class. Reads and fresh immutable writes go through a retrying client within Lance's defaults; conditional requests and mutable keys keep one attempt per request, and their owners resolve ambiguity. - A `__manifest` commit whose manifest request no re-issue settles is pending on its branch. The process keeps the base and the exact Lance transaction; the branch's next publication re-issues it once and refuses with the typed `PublicationPending` while storage stays silent. The server re-issues pending commits on a backoff and once more at clean shutdown, which retains admission if any stay pending. - The server contains only uncertainty the engine does not scope: a panic, a lost owner, `RecoveryRequired`, an opaque execution failure and an uncertain activation. A pending publication answers 503. - A guard registers every unconditional request to a reused key across the crates that write storage, with why its late landing is harmless.
A contained or crashed server kept `__cluster/lock.json` with no expiry, so a restart needed an operator's exact-ID unlock. On local and S3 roots a held lock is now version 3 and records `lease_seconds` (30 by default, `OMNIGRAPH_CLUSTER_LEASE_SECONDS`, 2 to 600). Its owner creates `__cluster/lease/<lock_id>/beat-<seq>.json` put-if-absent every third of a lease and checks for a claim before each beat. A timer of its own fences it two thirds of a lease after the send time of its last acknowledged beat, or at once on a claim; the server then contains and exits 2 without releasing. A serving acquisition that finds a leased lock watches the beat listing on its own monotonic clock. A new beat refuses it; a lease and a half of silence lets it create the claim once and replace the lock, by if-match on S3 and by overwrite on a local root, where the claim has serialized successors. Every other acquisition still refuses a held lock at once. A clean release stops beating and removes the lease directory first. Azure roots keep the unleased version-1 record. Older binaries refuse the version-3 record.
…me-from-commit # Conflicts: # crates/omnigraph-core/src/lance_clone.rs # docs/dev/lance.md
…tion-storage-outcomes
…ock-lease # Conflicts: # Cargo.lock
…nce 13 Lance 13's WrappingObjectStore also wraps paginated listings; the fault aims at puts only.
…tion-storage-outcomes
…t may land After the last re-issue the handler returned the store's error and kept reporting a missing version as definitive, so Lance read the version back as absent and deleted the attempt's transaction file. A manifest that lands later then names a file that is gone. Only the server's storage scope held that delete back; embedded and CLI opens did not. The handler now records the task whose request no re-issue settled, the task Lance reads the version back on, and reports a missing version as not definitive there. Lance returns CommitStatusUnknown and keeps the file. A refused request and a commit that sent none keep the inner answer.
…tion-storage-outcomes # Conflicts: # crates/omnigraph-catalog/src/tests.rs # docs/dev/writes.md
Lance deletes an attempt's transaction file when the handler calls a missing version definitive, and reads that answer with no reference to the attempt. The per-task marks that stood in for one relied on Lance reading the version back on the task that sent the request. The catalog's per-commit handler now answers from its probe, keeping the inner answer unless the request may still land. A dataset's shared handler holds no record of the attempt Lance reads back, so it never calls a missing version definitive; Lance reports the status unknown and the collector's orphan sweep reclaims a table's leftover transaction file.
A publication whose manifest request no re-issue settles was kept in a process registry, refused its branch with a 503 and was re-issued by a server resolver until storage answered. That registry repeated a guarantee the version key already gives: the manifest version is created only if absent, so the request lands before the branch's next publication or never. The registry also landed a crashed operation's commit under a rerun that had already captured its authority, so the rerun failed instead of converging. The registry, its error, the resolver and the shutdown hold are gone. An unknown publication answers in doubt and the branch publishes on: the next publication finds the late commit or takes its version first. A regression test covers both orders.
…tion-storage-outcomes # Conflicts: # docs/dev/writes.md
…rage-outcomes # Conflicts: # crates/omnigraph-catalog/src/tests.rs # crates/omnigraph-server/src/lib.rs # crates/omnigraph-storage/src/scoped.rs # docs/dev/writes.md # docs/user/deployment.md # docs/user/operations/storage-failures.md
RecommendationDo not merge A lease is the right tool when the owner is dead. This head treats one retryable storage error as death, and it can leave two processes writing the same cluster. Fix the heartbeat, the first-beat deadline, the local lock update, and the shutdown bound together. Keep one conditional update of RFC 1093 section 7 states two of these rules. Update that RFC in the same change. A beat that already exists must refresh the owner. The operation list must include the S3 if-match replace this code already performs. Blocking
Heartbeats are put-if-absent. The keeper then fences without a second attempt. After one acknowledged beat, the next beat waits one third of a lease. A failure leaves
A temporary test on this commit, Send fresh put-if-absent heartbeats on the retrying client. A failed create of a fresh beat name must stay retryable. Keep sending until the deadline. When a beat and the deadline are both ready, send the beat.
A temporary test paused the ownership read, wrote a successor record, then let the release continue. The release returned Refuse local takeover until the replacement and the release are one conditional update. The S3 if-match on the version the successor first read is the right shape. Add that operation to the list in the
The owner fences two thirds of a lease after its last acknowledged send. The successor may claim a lease and a half after the newest beat it saw. The gap is five sixths of a lease. That is 25 seconds at the default 30 second lease, and about 1.67 seconds at the minimum lease of 2 seconds. On the fence path the server closes HTTP admission and arms the shutdown watchdog. The default grace is 25 seconds ( At the default lease the grace equals the gap. At a 2 second lease the fenced process can still mutate for about 23 seconds after a successor is allowed to claim. The takeover wait must cover the grace this server uses, or the fenced process must stop storage mutations before a successor may claim. Optional
Tests to addA
On ValidationCommit Result on this commit, with only the new tests added: 1 passed ( Result: failed in 0.00 seconds.
CI on this commit is green. Those jobs do not run these four cases. I did not run a live server, S3, Azure, or an injected HTTP 503. The grace overlap is from the constants TradeoffCorrectness comes before a shorter lease or a single storage attempt. One failed heartbeat must leave the owner in place. The cost is more heartbeat retries, and a takeover that waits out real shutdown. Five more special cases around this lock would leave two clocks, a grace period, and a local overwrite as separate sources of truth. One conditional Inline comments
|
…rage scope A heartbeat went through the serving scope's single-attempt client. One unanswered beat left the owner one more chance before its deadline, and it left the beat key unsettled, so the scope refused the owner's re-issue and later its clean release. The lease now writes through the root outside the scope, on the backend's retrying client. A beat already present is this owner's own earlier request and counts from the first send of that beat; an unanswered beat is sent again after a quarter of a beat interval. A successor that finds a claim naming its own lock id holds it. A local takeover overwrote lock.json, and an exact-owner delete the old process had already sent could remove the successor's record. Local roots have no conditional update, so their locks carry no lease and keep the exact-ID unlock; only S3 takes a silent owner over, by if-match.
ragnorc
changed the title
The first beat ran before the fence timer was armed. A hung first request let a successor claim the lock, after which acquisition still returned a guard the server would serve with. It now races the owner's deadline, counted from the send of the request that wrote the lock record; when it lapses, acquisition fails and the process does not serve. A fenced server kept its storage scope open through its shutdown grace, which can outlast the gap between the fence and a successor's claim. The lease now closes the holder's scope before it reports the fence, so no mutation is sent after a successor may act. Only a server's S3 lock is leased: a CLI writer or deployment has nothing that stops its writes when a lease fences, so its lock stays unleased with the exact-ID unlock. A beat and the deadline ready together count the beat. The crate's own test build gets Tokio's paused time.
ragnorc
changed the title
A claim that appears while beat 0 is in flight now keeps the holder from admitting work: the owner reads the claim once more after beat 0 is acknowledged, and acquisition fails if one names its lock.
|
Thanks. Every blocking finding of the review of 1. One failed heartbeat fenced a live owner. Heartbeats, the watch, the claim and the lease-directory removal now go through the root outside the owner's storage scope, on the backend's retrying client ( 2. A landed beat whose acknowledgement was lost. 3. The first beat sat outside the fence. 4. Local takeover and the local release delete. Refused. Only a server's lock on S3 is leased, and takeover is the if-match on the version the successor first read. A local lock, even one whose record carries a lease, refuses a server at once. The if-match is in the operation lists of the
5. Shutdown grace against the fence-to-claim gap. A fence now closes the holder's storage scope before it is reported ( Optional.
Tests. The names you asked for, under the existing owners:
Verification on
|
… more A lease fence closed the holder's storage scope. Closing refuses new requests, but an admitted request kept retrying inside its client and an admitted multipart upload kept sending parts until the process exited, and the default 25 s shutdown grace outlasts the fence-to-claim gap of any lease under 30 s. `StorageIoScope::fence` refuses every later request, including the parts, completion and abort of an upload admitted before it, and cancels every admitted request still running: its future is dropped, so its client sends no further attempt. The cancelled request's caller gets an ambiguous error, since its last attempt may be on the wire. The lease fences the scope instead of closing it, so a fenced owner sends what a crashed owner sends.
Main made ApiError's status a method; the owned-write log line read it as a field.
RecommendationDo not merge A lease is the right tool when an S3 server is dead. A new server can then take the one cluster lock. The five blocking findings on What this change doesA server's S3 lock is version 3 and carries Local roots, Azure, CLI writers, and deployments stay unleased. They have no conditional replace, or nothing in the process stops their writes when a lease fences. Earlier findings on
|
…ped claim A successor created `superseded.json`, then replaced `lock.json` by if-match. If it stopped between the two, every later start found the claim, reported the lock held at once and never watched the silence, so no server could take the lock and an operator had to unlock it. The claim now only fences an owner that is somehow still running. Each successor waits out its own lease and a half of silence, makes sure the claim exists (one already there changes nothing), and replaces `lock.json` by if-match on the version it read. That if-match alone decides the next owner, so a stopped successor leaves the lock for the next start.
|
Thanks. The wedge was real. It is fixed in The claim elects nobody now.
Tests, under the existing owners:
Also since your review, from the RFC review of Verification on
|
RecommendationMerge WhyA successor that stopped after it wrote WhatThe claim no longer chooses the next owner. Each successor waits a lease and a half of its own silence, then makes sure the claim exists. The if-match on the version of How
The claim still fences an owner that is running, at that owner's next beat. The owner also fences itself two thirds of a lease after its last acknowledged beat. That fence still closes the storage scope before the successor may claim. ImpactA later start can finish a takeover that stopped between the claim and the replace. The wait is one lease and a half. That is 3 seconds at the minimum lease and 15 minutes at the maximum. Two servers cannot both hold the lock. A fresh claim does not skip the silence. A leftover claim does not add a wait that grows without a bound. Earlier findingsThe five blockers on BlockingNone. OptionalRFC 1093 at TradeoffRestart has to succeed after a crash in the claim step. That comes before a shorter claim. The claim only fences a running owner. One conditional update of ValidationReviewed head: A Before, on Result: 0 passed, 1 failed. Panic: Result: 0 passed, 1 failed, in 3.01s. Panic: After, on the reviewed head, rebuilt from that commit after removing a stale Result: 17 passed, 0 failed, in 6.02s. CI on this head is green, including Test Workspace, Lint (clippy), Format, GQ Logic Tests, the DST pinned suite, both RustFS shards, and Azurite. I did not re-run those jobs. I did not run a live S3 server or an injected HTTP 503. The acquire test uses the in-memory adapter, whose conditional update is |
Accepted on the maintainer's instruction after review approved the RFC, ModernRelay#1096 and ModernRelay#1097. The owned server operations and engine settlement RFCs get disposition notes for the parts this replaces: unknown storage outcomes no longer close admission, ambiguous storage outcomes are fenced by re-issue and the version key instead of fail-stop, and a server's S3 lock is leased. Section 7 also says, per the approving review, that a claim proves nothing about its writer and that the owner's timer, not the claim, is the fence.
Stacked on #1096. Review the commits after the merge of
fix/per-operation-storage-outcomes. This implements §7 of the draft RFC #1093 and should merge only after that RFC is accepted.What & why
On S3 roots, a server's cluster lock now carries a lease, so an orchestrator can restart a failed server without
cluster force-unlock. This closes step 4 of #1067. On 2026-10-19 the contained process kept__cluster/lock.jsonwith no expiry, Railway's restart policy wasNEVER, and recovery took an operator and about 16 hours.Only a server's S3 lock is leased.
A takeover must replace
lock.jsonwithout racing a release the old owner already sent, and only a conditional update does that. Local roots have none, so a successor's overwrite could be undone by an exact-owner delete the old process had already sent. Local and Azure locks keep the exact-ID unlock, as today.A CLI writer or deployment has nothing that stops its writes when a lease fences, so its lock stays unleased too.
Record. A held S3 lock is version 3 and records
lease_seconds: 30 by default, orOMNIGRAPH_CLUSTER_LEASE_SECONDSwithin 2–600. A record outside those bounds is refused, so a corrupt lease cannot hold a successor off. Older binaries refuse version 3, so they neither take it over nor release it.Where the lease's requests go. Heartbeats, the watch, the claim and the removal of a lease directory go through the root outside the owner's storage scope, on the backend's retrying client (
ClusterStore::lease_adapter). A beat is a key only its owner creates, under a lock id no other process uses, so a copy that lands late only keeps a successor waiting. A failed beat leaves no key unsettled, and beats continue after the scope closes for a release.Owner.
__cluster/lease/<lock_id>/beat-<seq>.jsonput-if-absent every third of a lease, the first before it admits work. It checks forsuperseded.jsonbefore each beat.StorageIoScope::fence, new here) before it is reported. Every later request is refused, including the parts, completion and abort of an admitted multipart upload. Every admitted request is cancelled: its future is dropped, so its client sends no further attempt or retry. A fenced owner therefore sends what a crashed owner sends, however long its shutdown grace lasts. The server then contains: admission closes, and it exits 2 without releasing.Successor. Only serving admission takes over; a CLI writer or deployment still refuses a held lock at once.
state_lock_held. An empty listing starts the silence, and the first beat that appears restarts it, because the owner writes its lock before beat 0 lands.lock.jsonby if-match on the version it first read. The claim elects nobody: it only fences an owner that is somehow still running, and a claim already there, from any successor, changes nothing.lock.jsonunchanged, so the next start takes it over the same way.Clean release. It stops beating and deletes the lease directory, then releases as before. If that release fails, the lock is a silent leased lock, and the next server takes it over.
Timing. Every counted beat became visible no earlier than its first send, so a takeover happens at least 1.5 leases after it. The owner fenced by 2/3 of a lease after it. The gap of 5/6 of a lease covers clock-rate drift and a stalled runtime; no clock sync is needed.
Correctness does not rest on the lease. A request a dead owner sent before it fenced is one of these kinds, and none can corrupt state:
Backing issue / RFC
Checklist
scoped::tests::a_fence_stops_admitted_requests_and_later_parts_issue_1067:lease::testsunder paused time:one_failed_heartbeat_does_not_fence_a_live_owner_issue_1067: one failure, then three in a row, inside the deadline;a_landed_beat_whose_ack_was_lost_refreshes_the_owner_issue_1067: six beats applied with lost answers do not fence the owner, and a successor still sees them;a_hung_first_beat_fences_before_a_successor_can_claim_issue_1067:startreturns the fence and no keeper by 2/3 of a lease after the lock write, the successor claims only after 1.5 leases, and the holder's scope refuses writes;a_claim_during_the_first_beat_keeps_the_holder_from_serving_issue_1067;a_claim_left_by_a_stopped_successor_does_not_block_takeover_issue_1067: the stopped successor and a different one both find the owner silent after their own 1.5 leases (the review's wedge observedAliveafter 0 ns for both);a_successor_finds_a_silent_owner_after_a_lease_and_a_half_issue_1067: two successors both find the owner silent, and the claim is in place.cec09eaa, per the review.state_lock::tests, on an in-memory store with S3's conditional semantics:a_restarted_server_takes_over_a_silent_owners_lock_issue_1067;a_release_and_a_takeover_never_both_apply_issue_1067.only_a_servers_s3_lock_is_leased_issue_1067.a_takeover_survives_a_successor_that_stopped_after_its_claim_issue_1067: with a stopped successor's claim in place, two racing starts each wait out their silence, and the if-match picks exactly one.a_local_lock_is_never_taken_over_issue_1067: a local lock, even one whose record carries a lease, refuses a server at once and stays unchanged.state_lock_is_exclusive_and_releases_on_drop: no lease, and a serving acquisition refuses a held lock at once.boot_settingslocal journeys are back to their pre-lease expectations: a retained lock refuses another server.operations::tests: a fenced lease closes admission and reaches the fail-stop path.state_lockwire test: version 3 needs a bounded lease, a release drops it, and version 4 is a future version.dst_successor_takes_over_a_dead_owners_lease_issue_1067: 16 seeds, with the dead owner's in-flight beat landing before, during or after the watch.operations/storage-failures.md: restart policy, lease, and exact-ID unlock for local, Azure and older locks;deployment.md;control-plane.mdand troubleshooting;changelog.d/cluster-lock-lease.changed.md.lock.json.Local verification
On
8acad43a(#1096 andmainatd111c403merged), withRUST_MIN_STACK=16777216, on macOS arm64:cargo fmt --all --check: clean.crates/omnigraph-dst: 7 passed:dst_crash_state_enumeration_sampled;check-docs.py,check-agents-md.sh, andtyposon the changed paths: clean (run onf5b939d3; later commits touched one docs paragraph, whichcheck-docs.pypassed).Notes for reviewers