Skip to content

Port s3cache to groupnet 30407c0 with managed links - #18

Merged
coconutbird merged 20 commits into
mainfrom
feat/groupnet-main
Nov 12, 2026
Merged

coconutbird merged 20 commits into
mainfrom
feat/groupnet-main

Conversation

@coconutbird

@coconutbird coconutbird commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

This branch pins groupnet to 30407c0bb057a7f12bf1520397f7398b38f97103 and imports the unfinished managed-link work from the local clone: explicit UDP neighbors, every ordinal listed in the chart, and the coherence, sync tests and strong_rw modules split into child files. It then ports s3cache to the API.

Pre-write fence for strong writes

strong_rw::workload::a_crashed_pod_restarts_on_its_warm_tier failed intermittently on main and on this branch (unacked). A writer crashed after the origin applied its PUT and before it published the outcome. The peer kept its lease for up to D and served the old value.

A strong write is now fenced before its origin call:

  • The writer publishes IndexOp::Pending for each key. It waits on the same ack-or-lapse round as the outcome. Only then does it call the origin.
  • A peer that applied the announcement answers the key from the origin. A LIST whose prefix covers the key is also forwarded. The peer stops when the outcome event names the announcement (settles). Every event also carries the writer's oldest open announcement (floor), so a gap that drops the outcome cannot strand the fence.
  • If the writer's life ends with announcements open (a gap into a new epoch, a renewal or a seal), the peer marks each key as uncertain. One origin HEAD then resolves the key.
  • If the fence does not complete in time, the write is answered 503 and is not sent to the origin. If the origin refuses the write, the writer publishes IndexOp::Abandoned.
  • A donor offers no index image while it holds a peer's open announcement. An install carries the follower's open announcements.
  • strong-acks and bounded do not fence. The wire magic moves to 0xFD.

Cost: every strong write publishes two feed events and pays the ack round twice. strong_rw reports the latency of acknowledged writes (debug build, local MinIO):

scenario before: median / p99 after: median / p99
steady, origin jitter 0–40 ms (3 runs each) 89.8–91.9 ms / 120–135 ms 90.7–94.6 ms / 153–183 ms
150 ms gossip delay, 30% loss (2 runs each) 389–390 ms / 553–603 ms 718–723 ms / 982–1007 ms

Verification (local)

  • The crash test passed 20 of 20 runs in a row on c35e1b4, with 0 violations in each run. Later commits change only tests and documentation.
  • I ran the full cargo test --locked --workspace --all-features --test it three times on 56a1a7c. Runs 1 and 3 passed: 95 passed and 10 ignored. Run 2 had 94 passed and 1 failed: fleet_bootstrap::peer_bootstrap_transfers_a_production_scale_index. In that run the joiner stopped waiting for the 116 MB donor image (DonorUnavailable) and listed the origin. That test does no writes, and it passed 3 of 3 runs alone.
  • The unit tests passed: 191 passed and 1 ignored. They include new tests for the three crash points, a stalled fence (503 with no origin connection), a lapse of a partitioned peer, and the settle and floor rules.
  • cargo fmt --check, clippy (all targets, all features) and cargo doc -D warnings are clean.

Old and new pods cannot exchange gossip: groupnet reset every wire version to 1, moved to the routed network, and keys the default feed entry as ~writes/d. The feed event changed again here.

…lit coherence and test modules

Import the unfinished work from the local clone as it stood: the gossip
node starts one managed network over an explicitly admitting UDP link,
the chart lists every ordinal as a neighbor, the planned stop closes the
network after the seal, the apply loop moves to sync/coherence/apply.rs,
the sync test fixtures move to sync/tests/fixtures.rs, and the strong_rw
workload moves to tests/it/strong_rw/workload.rs. Groupnet builds without
default features, with the memory binding for tests only.

It was written against a Groupnet revision between fb97452 and 30407c0
and does not build at either; the following commits port it.
…and ack outcomes

Lease types move to groupnet::consistency::lease and Leases::new returns a
Result. WriteFeed::publish and seal report an unadvertised change instead of
dropping it silently, and publish after a seal is an error rather than a
panic inside Groupnet; s3cache keeps the panic. applied_by_selected takes an
AckWait and returns an AckOutcome; is_complete is the old boolean, empty
selection included. AckLedger::record reports a refused advertisement, now
logged. The settle floor reads gossip_interval_ms, which absorbed
anti_entropy_interval_ms.
… RPITIT recovery adapter

NamedSeeds takes with_* refiners and a NonZeroU32 attempt count, and the UDP
transport reports its bound address directly. A link rejects a repeated
neighbor, so duplicate seed ids collapse to one. RecoveryConfig carries
NonZero bounds, validation names the violated rule, and the driver opens
through RecoveryHandle::builder. The adapter returns AdapterError kinds and
logs operation_failed. with_recovery_config returns RecoveryConfigRejected
and with_recovery_rearm returns RecoveryRearmError.
…bootstrap ports

Bootstrap ports are RPITIT traits, so the adapter implements async fns. Every
journal, transfer, claim, wire and admission bound is NonZero; s3cache's own
image accounting stays usize and converts at the Groupnet boundary. Effects
run under one EffectContext, the journal takes a CapturePlan, ImageSize and
NativeRenewal, NativeClaimSource and BulkDonorPort take option structs, the
session takes a PresenceIdentity, and the driver opens through the recovery
builder. Adapter failures carry an AdapterErrorKind. A refused feed
retirement advertisement stays local until the next publish, as before.
…net port

The synchronous adapter operations return ready futures: an async fn without
an await trips unused_async_trait_impl, and an async block trips
manual_async_fn.
Lease types come from consistency::lease, publish returns a Result, seed
refiners are with_*, RecoveryConfig and journal bounds are NonZero, and the
fault transport fills Inbound::source with None because it keeps no address
book. The test configs that set anti_entropy_interval_ms to 25 next to a
10 ms gossip_interval_ms now set gossip_interval_ms to 25: the engine
scheduled digest rounds on the former, and Groupnet folded the two into one
cadence.
…ed key

Groupnet now keys the default write feed's entry as ~writes/d, so a value
written under the bare ~writes reached no observer and the observation stayed
healthy. The test failed for that reason and passes with the real key.
The single test step passed integration-module filters that also applied to
the lib and bin harnesses, so CI ran none of the 182 unit tests. Unit tests now
run in their own step. Pull requests run the test job only; the image push and
the Fleet bump stay on pushes to main. The MinIO-backed integration suites
still run locally only.
@coconutbird
coconutbird marked this pull request as ready for review November 12, 2026 10:50
@coconutbird
coconutbird merged commit e16dcb9 into main Nov 12, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant