diff --git a/CLAUDE.md b/CLAUDE.md index 97fc09f..6b8b5dc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -36,16 +36,19 @@ recorded as an ADR in `docs/adr/`. Keep them in sync when the design changes. Crate roles are what `ls crates/` + each `Cargo.toml` say; the rules are what matters. Strict dependency direction: top-level adapters `{cli, desktop} → repo → -{vfs, gitio, crypto} → core`, with the separate leaf edge `repo → tlsio` -(`tlsio` is a leaf — it depends on no other workspace crate, not even `core`). -**`core` must never depend on Git, worktrees, or crypto.** **`gix` must stay -quarantined in `gitio`** — if you find yourself reaching for `gix` elsewhere, -add a function to `gitio` instead. **RustCrypto must stay quarantined in -`crypto`** — if you find yourself reaching for it elsewhere, add a function to -`crypto` instead. **rustls/rcgen must stay quarantined in `tlsio`** — if you -find yourself reaching for TLS elsewhere, add a function to `tlsio` instead. -**`repo` must not depend on `gitio`** — `cli` links both and passes imported -snapshots down; `repo` stays Git-agnostic. +{vfs, gitio, crypto} → core`, with the separate leaf edges `repo → tlsio` +**and `repo → objio`** (`tlsio` and `objio` are leaves — each depends on no +other workspace crate, not even `core`). **`core` must never depend on Git, +worktrees, or crypto.** **`gix` must stay quarantined in `gitio`** — if you +find yourself reaching for `gix` elsewhere, add a function to `gitio` +instead. **RustCrypto must stay quarantined in `crypto`** — if you find +yourself reaching for it elsewhere, add a function to `crypto` instead. +**rustls/rcgen must stay quarantined in `tlsio`** — if you find yourself +reaching for TLS elsewhere, add a function to `tlsio` instead. **Object-store +SDKs must stay quarantined in `objio`** — if you find yourself reaching for +S3 elsewhere, add a function to `objio` instead. **`repo` must not depend on +`gitio`** — `cli` links both and passes imported snapshots down; `repo` stays +Git-agnostic. ## Core invariants (do not break) @@ -103,7 +106,7 @@ the project tree if desired. ## Capability map (what's built, by phase) -All 35 phases are built and tested. One line of current fact per phase; the +All 36 phases are built and tested. One line of current fact per phase; the authoritative rationale and full semantics live in the linked ADR, the design in `ARCHITECTURE.md`. The old per-phase narrative log this table replaced is archived verbatim at `docs/archive/claude-md-phase-log-2026-07.md` — do not @@ -147,6 +150,7 @@ the code, those win. | P33 | Randomized protected sealing (fresh DEK + nonce; `RANDOMIZED` perms bit); dual-read of pre-P33 convergent ciphertext; per-checkout keyed stat cache; `sc rewrap` upgrades convergent blobs at the tip | [0043](docs/adr/0043-randomized-protected-encryption.md) | | P34 | Private branches: ref points at a sealed-branch manifest; every commit/tree/blob individually sealed (copy-on-write) under a per-branch KEK wrapped per recipient + escrow; `sc branch --private/grant/revoke/publish`; opaque to non-recipients (content, paths, messages); grant O(1), revoke rotates the KEK; publish replays to public with a scanner gate; git bridge + private→public integration refused; `PROTOCOL_VERSION` 4 | [0044](docs/adr/0044-per-branch-access-control.md) | | P35 | Native Tauri desktop browser: opens `.sc` repositories through `scl-repo`, shows local/remote refs, all-parent snapshot DAG + provenance, public trees and first-parent diffs; protected content is locked and private branches remain opaque; no mutation or identity surface | [0045](docs/adr/0045-native-desktop-read-model.md) | +| P36 | P36a built: bucket WAL remotes (sc+wal://, sc+s3://) — immutable packs + CAS'd manifest, multi-writer safe, no coordinator; checkpoints (P36b) and bucket-backed serve (P36c) pending | [0046](docs/adr/0046-wal-bucket-remotes.md) | ## Standing boundaries & gotchas @@ -165,6 +169,9 @@ transport-adjacent. The rest, imperatively: - **Partial clones refuse merge, cherry-pick/rebase, `sc ws fork`/`harvest`, `sc work`, `sc export`, and `sparse disable`.** `sc backfill --all` converts to a genuine full clone and re-enables them. +- **Bucket remotes hold public content plaintext at rest** — bucket ACL is + the perimeter (sealed content stays ciphertext, unchanged); partial-clone + `filter` against bucket remotes is refused. - **Protected sealing is randomized since P33.** Pre-P33 convergent ciphertext dual-reads forever and stays equality-confirmable forever (rotation ≠ erasure). Identical independent edits on two branches now genuinely diff --git a/Cargo.lock b/Cargo.lock index e02ed42..3cd0edc 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -122,12 +122,6 @@ dependencies = [ "rustversion", ] -[[package]] -name = "arrayref" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" - [[package]] name = "arrayvec" version = "0.7.7" @@ -208,6 +202,490 @@ version = "1.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" +[[package]] +name = "aws-config" +version = "1.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a767267da9e2c2e189b2f9df8b5657e850ecf5352644734ba130d4a57095cf1b" +dependencies = [ + "aws-credential-types", + "aws-runtime", + "aws-sdk-sso", + "aws-sdk-ssooidc", + "aws-sdk-sts", + "aws-smithy-async", + "aws-smithy-http", + "aws-smithy-json", + "aws-smithy-runtime", + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "aws-types", + "bytes", + "fastrand", + "hex", + "http 1.4.2", + "sha1 0.10.6", + "time", + "tokio", + "tracing", + "url", + "zeroize", +] + +[[package]] +name = "aws-credential-types" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e93964ffdaf57857f544be3666a5f57570bb699e934700f11b49708f61bb556e" +dependencies = [ + "aws-smithy-async", + "aws-smithy-runtime-api", + "aws-smithy-types", + "zeroize", +] + +[[package]] +name = "aws-lc-rs" +version = "1.18.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce2b2dcc879c3bae0d371e77c99f2238400ef24ec001394befa67b6e543add9e" +dependencies = [ + "aws-lc-sys", + "zeroize", +] + +[[package]] +name = "aws-lc-sys" +version = "0.44.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f09fae7be8bb3174e05c6afdb34199e6dc0c7c04ba9fa237b1967adfbde27483" +dependencies = [ + "cc", + "cmake", + "dunce", + "fs_extra", + "pkg-config", +] + +[[package]] +name = "aws-runtime" +version = "1.9.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c9007227e10b5fed2f3e0a2beff489211e2b5604c400b7a9d5d81ca9d64c24bb" +dependencies = [ + "aws-credential-types", + "aws-sigv4", + "aws-smithy-async", + "aws-smithy-eventstream", + "aws-smithy-http", + "aws-smithy-runtime", + "aws-smithy-runtime-api", + "aws-smithy-types", + "aws-types", + "bytes", + "bytes-utils", + "fastrand", + "http 0.2.12", + "http 1.4.2", + "http-body 0.4.6", + "http-body 1.1.0", + "percent-encoding", + "pin-project-lite", + "tracing", + "uuid", +] + +[[package]] +name = "aws-sdk-s3" +version = "1.144.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "30dc8bf6baaf7d46336a0ca2c69f223d9b90d7a801fb3e28f7ea17b00dc6b1de" +dependencies = [ + "arc-swap", + "aws-credential-types", + "aws-runtime", + "aws-sigv4", + "aws-smithy-async", + "aws-smithy-checksums", + "aws-smithy-eventstream", + "aws-smithy-http", + "aws-smithy-json", + "aws-smithy-observability", + "aws-smithy-runtime", + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "aws-smithy-xml", + "aws-types", + "bytes", + "fastrand", + "hex", + "hmac 0.13.0", + "http 0.2.12", + "http 1.4.2", + "http-body 1.1.0", + "lru", + "percent-encoding", + "regex-lite", + "sha2 0.11.0", + "tracing", + "url", +] + +[[package]] +name = "aws-sdk-sso" +version = "1.108.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c15301b04372832947916607983b114b3374b9db0be058a00fb7513800de1f05" +dependencies = [ + "arc-swap", + "aws-credential-types", + "aws-runtime", + "aws-smithy-async", + "aws-smithy-http", + "aws-smithy-json", + "aws-smithy-observability", + "aws-smithy-runtime", + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "aws-types", + "bytes", + "fastrand", + "http 0.2.12", + "http 1.4.2", + "regex-lite", + "tracing", +] + +[[package]] +name = "aws-sdk-ssooidc" +version = "1.110.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72cc2c205cb27108183cf1856333f7d584c2ba0f505421b4209ca5828f9ea899" +dependencies = [ + "arc-swap", + "aws-credential-types", + "aws-runtime", + "aws-smithy-async", + "aws-smithy-http", + "aws-smithy-json", + "aws-smithy-observability", + "aws-smithy-runtime", + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "aws-types", + "bytes", + "fastrand", + "http 0.2.12", + "http 1.4.2", + "regex-lite", + "tracing", +] + +[[package]] +name = "aws-sdk-sts" +version = "1.113.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68182ecb449f7537db0f4d5d25917789cf41e32074a9fe47b6a0b847fe1d2032" +dependencies = [ + "arc-swap", + "aws-credential-types", + "aws-runtime", + "aws-smithy-async", + "aws-smithy-http", + "aws-smithy-json", + "aws-smithy-observability", + "aws-smithy-query", + "aws-smithy-runtime", + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "aws-smithy-xml", + "aws-types", + "fastrand", + "http 0.2.12", + "http 1.4.2", + "regex-lite", + "tracing", +] + +[[package]] +name = "aws-sigv4" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "723c2234ad7511ceef63eab016b7ba6ff7c55590fefb96fa8467af014a07309f" +dependencies = [ + "aws-credential-types", + "aws-smithy-eventstream", + "aws-smithy-http", + "aws-smithy-runtime-api", + "aws-smithy-types", + "bytes", + "crypto-bigint", + "form_urlencoded", + "hex", + "hmac 0.13.0", + "http 0.2.12", + "http 1.4.2", + "p256", + "percent-encoding", + "sha2 0.11.0", + "subtle", + "time", + "tracing", + "zeroize", +] + +[[package]] +name = "aws-smithy-async" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f02e407fb3b54891734224b9ffac8a71fdd35f542500fa1af95754a6b2beb316" +dependencies = [ + "futures-util", + "pin-project-lite", + "tokio", +] + +[[package]] +name = "aws-smithy-checksums" +version = "0.65.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b67ecd999972b58e67cab052f5129906c08c25883bd0788ceefc55ef97d61307" +dependencies = [ + "aws-smithy-http", + "aws-smithy-types", + "bytes", + "crc-fast", + "hex", + "http 1.4.2", + "http-body 1.1.0", + "http-body-util", + "md-5", + "pin-project-lite", + "sha1 0.11.0", + "sha2 0.11.0", + "tracing", +] + +[[package]] +name = "aws-smithy-eventstream" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6de526c7b567420a31bc283657a7921b45c4cafe0827fdf2490713dcc770c28f" +dependencies = [ + "aws-smithy-types", + "bytes", + "crc32fast", +] + +[[package]] +name = "aws-smithy-http" +version = "0.64.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "37843d9add67c3aff5856f409c6dc315d3cdff60f9c0cb5b670dab1e9920306d" +dependencies = [ + "aws-smithy-eventstream", + "aws-smithy-runtime-api", + "aws-smithy-types", + "bytes", + "bytes-utils", + "futures-core", + "futures-util", + "http 1.4.2", + "http-body 1.1.0", + "http-body-util", + "percent-encoding", + "pin-project-lite", + "pin-utils", + "tracing", +] + +[[package]] +name = "aws-smithy-http-client" +version = "1.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebfd138fac0337cee7516c352757ea73b9f2266e57d0bcb5bc70e9547e45aef1" +dependencies = [ + "aws-smithy-async", + "aws-smithy-runtime-api", + "aws-smithy-types", + "h2", + "http 1.4.2", + "hyper", + "hyper-rustls", + "hyper-util", + "pin-project-lite", + "rustls", + "rustls-native-certs", + "rustls-pki-types", + "tokio", + "tokio-rustls", + "tower", + "tracing", +] + +[[package]] +name = "aws-smithy-json" +version = "0.63.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3dc65a121adb4b33729919fcfa14fa36fb33c1555a8f06bb0e2188dbfdc1d9ef" +dependencies = [ + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", +] + +[[package]] +name = "aws-smithy-observability" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e86338c869539a581bf161247762a6e87f92c5c075060057b5ed6d06632ed0c" +dependencies = [ + "aws-smithy-runtime-api", +] + +[[package]] +name = "aws-smithy-query" +version = "0.62.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "512346c7212ab7436df2d77a16d976a468ae44a418835511d2a69269810aaf62" +dependencies = [ + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "aws-smithy-xml", + "urlencoding", +] + +[[package]] +name = "aws-smithy-runtime" +version = "1.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b82e438d30e02a825d363bd639a9efaed68a8089d86101054b0081e7e0d3e606" +dependencies = [ + "aws-smithy-async", + "aws-smithy-http", + "aws-smithy-http-client", + "aws-smithy-observability", + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "bytes", + "fastrand", + "http 0.2.12", + "http 1.4.2", + "http-body 0.4.6", + "http-body 1.1.0", + "http-body-util", + "pin-project-lite", + "pin-utils", + "tokio", + "tracing", +] + +[[package]] +name = "aws-smithy-runtime-api" +version = "1.15.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "954c563ce84507722d2679f07a35d21b9c6466b3872d513020d0281fc8112ac9" +dependencies = [ + "aws-smithy-async", + "aws-smithy-runtime-api-macros", + "aws-smithy-types", + "bytes", + "http 0.2.12", + "http 1.4.2", + "pin-project-lite", + "tokio", + "tracing", + "zeroize", +] + +[[package]] +name = "aws-smithy-runtime-api-macros" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "221eaa237ddf1ca79b60d1372aad77e47f9c0ea5b3ce5099da8c61d027dc77b3" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "aws-smithy-schema" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d56e0a4e53127a632224e43633b0fe045fa9e1e3cfc68b9830f1115e103f910" +dependencies = [ + "aws-smithy-runtime-api", + "aws-smithy-types", + "http 1.4.2", +] + +[[package]] +name = "aws-smithy-types" +version = "1.6.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fce83ce9abbb198d25bc7131e468d0f9fe1257125e58c39f3f9fc9f5098c9647" +dependencies = [ + "base64-simd", + "bytes", + "bytes-utils", + "futures-core", + "http 0.2.12", + "http 1.4.2", + "http-body 0.4.6", + "http-body 1.1.0", + "http-body-util", + "itoa", + "num-integer", + "pin-project-lite", + "pin-utils", + "ryu", + "serde", + "time", + "tokio", + "tokio-util", +] + +[[package]] +name = "aws-smithy-xml" +version = "0.62.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce84f71c72fee2cbbadde6e7d082f5fb466e3a84733855295fa7aafd1b31b7d8" +dependencies = [ + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "xmlparser", +] + +[[package]] +name = "aws-types" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eec1cd5469f328c782dc3e33d4153cf118a54e33cbb3356d60d16f89883e1f94" +dependencies = [ + "aws-credential-types", + "aws-smithy-async", + "aws-smithy-runtime-api", + "aws-smithy-schema", + "aws-smithy-types", + "rustc_version", + "tracing", +] + +[[package]] +name = "base16ct" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4c7f02d4ea65f2c1853089ffd8d2787bdbc63de2f0d29dedbcf8ccdfa0ccd4cf" + [[package]] name = "base64" version = "0.21.7" @@ -220,6 +698,22 @@ version = "0.22.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" +[[package]] +name = "base64-simd" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "339abbe78e73178762e23bea9dfd08e697eb3f3301cd4be981c0f78ba5859195" +dependencies = [ + "outref", + "vsimd", +] + +[[package]] +name = "base64ct" +version = "1.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" + [[package]] name = "bisync" version = "0.3.0" @@ -276,11 +770,10 @@ dependencies = [ [[package]] name = "blake3" -version = "1.8.6" +version = "1.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76ae7bad254120e9e4c63bafc385310756f90c484eac0e36b8317cf09cb92a77" +checksum = "6d9e454fc11f76977dc803893aff6304ed33d6a26efae8696573bea74baa27ae" dependencies = [ - "arrayref", "arrayvec", "cc", "cfg-if", @@ -383,6 +876,16 @@ dependencies = [ "serde", ] +[[package]] +name = "bytes-utils" +version = "0.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7dafe3a8757b027e2be6e4e5601ed563c55989fcf1546e933c66c8eb3a058d35" +dependencies = [ + "bytes", + "either", +] + [[package]] name = "bytesize" version = "2.4.0" @@ -597,6 +1100,21 @@ dependencies = [ "hashbrown 0.16.1", ] +[[package]] +name = "cmake" +version = "0.1.58" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678" +dependencies = [ + "cc", +] + +[[package]] +name = "cmov" +version = "0.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a" + [[package]] name = "colorchoice" version = "1.0.5" @@ -613,6 +1131,18 @@ dependencies = [ "memchr", ] +[[package]] +name = "const-oid" +version = "0.9.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" + +[[package]] +name = "const-oid" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" + [[package]] name = "constant_time_eq" version = "0.4.2" @@ -687,6 +1217,16 @@ dependencies = [ "libc", ] +[[package]] +name = "crc-fast" +version = "1.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e75b2483e97a5a7da73ac68a05b629f9c53cff58d8ed1c77866079e18b00dba5" +dependencies = [ + "digest 0.10.7", + "spin", +] + [[package]] name = "crc32fast" version = "1.5.0" @@ -711,6 +1251,18 @@ version = "0.8.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" +[[package]] +name = "crypto-bigint" +version = "0.5.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0dc92fb57ca44df6db8059111ab3af99a63d5d0f8375d9972e319a379c6bab76" +dependencies = [ + "generic-array", + "rand_core 0.6.4", + "subtle", + "zeroize", +] + [[package]] name = "crypto-common" version = "0.1.7" @@ -770,6 +1322,15 @@ version = "0.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "52560adf09603e58c9a7ee1fe1dcb95a16927b17c127f0ac02d6e768a0e25bc1" +[[package]] +name = "ctutils" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e" +dependencies = [ + "cmov", +] + [[package]] name = "curve25519-dalek" version = "4.1.3" @@ -909,6 +1470,17 @@ dependencies = [ "thiserror 2.0.20", ] +[[package]] +name = "der" +version = "0.7.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" +dependencies = [ + "const-oid 0.9.6", + "pem-rfc7468", + "zeroize", +] + [[package]] name = "der-parser" version = "10.0.0" @@ -960,6 +1532,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" dependencies = [ "block-buffer 0.10.4", + "const-oid 0.9.6", "crypto-common 0.1.7", "subtle", ] @@ -971,7 +1544,9 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" dependencies = [ "block-buffer 0.12.1", + "const-oid 0.10.2", "crypto-common 0.2.2", + "ctutils", ] [[package]] @@ -1107,13 +1682,27 @@ version = "1.0.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555" +[[package]] +name = "ecdsa" +version = "0.16.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ee27f32b5c5292967d2d4a9d7f1e0b0aed2c15daded5a60300e4abb9d8020bca" +dependencies = [ + "der", + "digest 0.10.7", + "elliptic-curve", + "rfc6979", + "signature 2.2.0", + "spki", +] + [[package]] name = "ed25519" version = "3.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "29fcf32e6c73d1079f83ab4d782de2d81620346a5f38c6237a86a22f8368980a" dependencies = [ - "signature", + "signature 3.0.0", ] [[package]] @@ -1129,6 +1718,32 @@ dependencies = [ "zeroize", ] +[[package]] +name = "either" +version = "1.18.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "252afb9ae5eaa683babdc6a068b3f5726eb19e05070c731f9b2a23a7c3e8ed34" + +[[package]] +name = "elliptic-curve" +version = "0.13.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b5e6043086bf7973472e0c7dff2142ea0b680d30e18d9cc40f267efbf222bd47" +dependencies = [ + "base16ct", + "crypto-bigint", + "digest 0.10.7", + "ff", + "generic-array", + "group", + "pem-rfc7468", + "pkcs8", + "rand_core 0.6.4", + "sec1", + "subtle", + "zeroize", +] + [[package]] name = "embed-resource" version = "3.0.11" @@ -1210,6 +1825,16 @@ dependencies = [ "simd-adler32", ] +[[package]] +name = "ff" +version = "0.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c0b50bfb653653f9ca9095b427bed08ab8d75a137839d9ad64eb11810d5b6393" +dependencies = [ + "rand_core 0.6.4", + "subtle", +] + [[package]] name = "fiat-crypto" version = "0.2.9" @@ -1306,6 +1931,12 @@ dependencies = [ "percent-encoding", ] +[[package]] +name = "fs_extra" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c" + [[package]] name = "futures-channel" version = "0.3.32" @@ -1484,6 +2115,7 @@ checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" dependencies = [ "typenum", "version_check", + "zeroize", ] [[package]] @@ -2524,6 +3156,17 @@ dependencies = [ "system-deps", ] +[[package]] +name = "group" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0f9ef7462f7c099f518d754361858f86d8a07af53ba9af0fe635bbccb151a63" +dependencies = [ + "ff", + "rand_core 0.6.4", + "subtle", +] + [[package]] name = "gtk" version = "0.18.2" @@ -2576,6 +3219,25 @@ dependencies = [ "syn 2.0.118", ] +[[package]] +name = "h2" +version = "0.4.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ef8e5e5a340588f4452631496976cf8636d4a7ecf600239fdc27615d2530bc16" +dependencies = [ + "atomic-waker", + "bytes", + "fnv", + "futures-core", + "futures-sink", + "http 1.4.2", + "indexmap 2.14.0", + "slab", + "tokio", + "tokio-util", + "tracing", +] + [[package]] name = "hash32" version = "0.3.1" @@ -2653,7 +3315,7 @@ version = "0.12.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7b5f8eb2ad728638ea2c7d47a21db23b7b58a72ed6a38256b8a1849f15fbbdf7" dependencies = [ - "hmac", + "hmac 0.12.1", ] [[package]] @@ -2665,6 +3327,15 @@ dependencies = [ "digest 0.10.7", ] +[[package]] +name = "hmac" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6303bc9732ae41b04cb554b844a762b4115a61bfaa81e3e83050991eeb56863f" +dependencies = [ + "digest 0.11.3", +] + [[package]] name = "html5ever" version = "0.38.0" @@ -2675,6 +3346,17 @@ dependencies = [ "markup5ever", ] +[[package]] +name = "http" +version = "0.2.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "601cbb57e577e2f5ef5be8e7b83f0f63994f25aa94d673e54a92d5c516d101f1" +dependencies = [ + "bytes", + "fnv", + "itoa", +] + [[package]] name = "http" version = "1.4.2" @@ -2685,6 +3367,17 @@ dependencies = [ "itoa", ] +[[package]] +name = "http-body" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ceab25649e9960c0311ea418d17bee82c0dcec1bd053b5f9a66e265a693bed2" +dependencies = [ + "bytes", + "http 0.2.12", + "pin-project-lite", +] + [[package]] name = "http-body" version = "1.1.0" @@ -2692,7 +3385,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ca2a8f2913ee65f60facd6a5905613afaa448497a0230cc41ce022d93290bc2c" dependencies = [ "bytes", - "http", + "http 1.4.2", ] [[package]] @@ -2703,8 +3396,8 @@ checksum = "e9f41fd6a08e4d4ec69df65976da761afd5ad5e58a9d4acb46bd1c953a9e3ff2" dependencies = [ "bytes", "futures-core", - "http", - "http-body", + "http 1.4.2", + "http-body 1.1.0", "pin-project-lite", ] @@ -2739,8 +3432,9 @@ dependencies = [ "bytes", "futures-channel", "futures-core", - "http", - "http-body", + "h2", + "http 1.4.2", + "http-body 1.1.0", "httparse", "itoa", "pin-project-lite", @@ -2749,6 +3443,22 @@ dependencies = [ "want", ] +[[package]] +name = "hyper-rustls" +version = "0.27.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f" +dependencies = [ + "http 1.4.2", + "hyper", + "hyper-util", + "rustls", + "rustls-native-certs", + "tokio", + "tokio-rustls", + "tower-service", +] + [[package]] name = "hyper-util" version = "0.1.20" @@ -2759,8 +3469,8 @@ dependencies = [ "bytes", "futures-channel", "futures-util", - "http", - "http-body", + "http 1.4.2", + "http-body 1.1.0", "hyper", "ipnet", "libc", @@ -3238,6 +3948,15 @@ version = "0.4.33" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" +[[package]] +name = "lru" +version = "0.18.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5d2f2f9b4ba7e6b24d95e7e899329d35be83bcded72c8540cdd5368932d1d90a" +dependencies = [ + "hashbrown 0.17.1", +] + [[package]] name = "markup5ever" version = "0.38.0" @@ -3249,6 +3968,16 @@ dependencies = [ "web_atoms", ] +[[package]] +name = "md-5" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" +dependencies = [ + "cfg-if", + "digest 0.11.3", +] + [[package]] name = "memchr" version = "2.8.2" @@ -3652,12 +4381,36 @@ version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c08d65885ee38876c4f86fa503fb49d7b507c2b62552df7c70b2fce627e06381" +[[package]] +name = "openssl-probe" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe" + [[package]] name = "option-ext" version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "04744f49eae99ab78e0d5c0b603ab218f515ea8cfe5a456d7629ad883a3b6e7d" +[[package]] +name = "outref" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a80800c0488c3a21695ea981a54918fbb37abf04f4d0720c453632255e2ff0e" + +[[package]] +name = "p256" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c9863ad85fa8f4460f9c48cb909d38a0d689dba1f6f6988a5e3e0d31071bcd4b" +dependencies = [ + "ecdsa", + "elliptic-curve", + "primeorder", + "sha2 0.10.9", +] + [[package]] name = "pango" version = "0.18.3" @@ -3716,6 +4469,15 @@ dependencies = [ "serde_core", ] +[[package]] +name = "pem-rfc7468" +version = "0.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88b39c9bfcfc231068454382784bb460aae594343fb030d46e9f50a645418412" +dependencies = [ + "base64ct", +] + [[package]] name = "percent-encoding" version = "2.3.2" @@ -3781,6 +4543,22 @@ version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" +[[package]] +name = "pin-utils" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" + +[[package]] +name = "pkcs8" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f950b2377845cebe5cf8b5165cb3cc1a5e0fa5cfa3e1f7f55707d8fd82e0a7b7" +dependencies = [ + "der", + "spki", +] + [[package]] name = "pkg-config" version = "0.3.33" @@ -3882,6 +4660,15 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "925383efa346730478fb4838dbe9137d2a47675ad789c546d150a6e1dd4ab31c" +[[package]] +name = "primeorder" +version = "0.13.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "353e1ca18966c16d9deb1c69278edbc5f194139612772bd9537af60ac231e1e6" +dependencies = [ + "elliptic-curve", +] + [[package]] name = "proc-macro-crate" version = "1.3.1" @@ -4172,6 +4959,12 @@ dependencies = [ "regex-syntax", ] +[[package]] +name = "regex-lite" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cab834c73d247e67f4fae452806d17d3c7501756d98c8808d7c9c7aa7d18f973" + [[package]] name = "regex-syntax" version = "0.8.11" @@ -4188,8 +4981,8 @@ dependencies = [ "bytes", "futures-core", "futures-util", - "http", - "http-body", + "http 1.4.2", + "http-body 1.1.0", "http-body-util", "hyper", "hyper-util", @@ -4212,6 +5005,16 @@ dependencies = [ "web-sys", ] +[[package]] +name = "rfc6979" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dd2a808d456c4a54e300a23e9f5a67e122c3024119acbfd73e3bf664491cb2" +dependencies = [ + "hmac 0.12.1", + "subtle", +] + [[package]] name = "rfd" version = "0.16.0" @@ -4293,6 +5096,7 @@ version = "0.23.43" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0283386ce02abc0151e1761d08802dfe86c173b0b494af5cbc086574e453da06" dependencies = [ + "aws-lc-rs", "log", "once_cell", "ring", @@ -4302,6 +5106,18 @@ dependencies = [ "zeroize", ] +[[package]] +name = "rustls-native-certs" +version = "0.8.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dab5152771c58876a2146916e53e35057e1a4dfa2b9df0f0305b07f611fdea4d" +dependencies = [ + "openssl-probe", + "rustls-pki-types", + "schannel", + "security-framework", +] + [[package]] name = "rustls-pki-types" version = "1.15.1" @@ -4317,6 +5133,7 @@ version = "0.103.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" dependencies = [ + "aws-lc-rs", "ring", "rustls-pki-types", "untrusted", @@ -4340,6 +5157,12 @@ dependencies = [ "wait-timeout", ] +[[package]] +name = "ryu" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" + [[package]] name = "same-file" version = "1.0.6" @@ -4349,6 +5172,15 @@ dependencies = [ "winapi-util", ] +[[package]] +name = "schannel" +version = "0.1.29" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939" +dependencies = [ + "windows-sys 0.61.2", +] + [[package]] name = "schemars" version = "0.8.22" @@ -4471,6 +5303,18 @@ dependencies = [ "thiserror 2.0.20", ] +[[package]] +name = "scl-objio" +version = "0.1.0" +dependencies = [ + "aws-config", + "aws-sdk-s3", + "blake3", + "hex", + "thiserror 2.0.20", + "tokio", +] + [[package]] name = "scl-repo" version = "0.1.0" @@ -4481,6 +5325,7 @@ dependencies = [ "regex", "scl-core", "scl-crypto", + "scl-objio", "scl-tlsio", "scl-vfs", "serde", @@ -4514,6 +5359,43 @@ version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" +[[package]] +name = "sec1" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3e97a565f76233a6003f9f5c54be1d9c5bdfa3eccfb189469f11ec4901c47dc" +dependencies = [ + "base16ct", + "der", + "generic-array", + "pkcs8", + "subtle", + "zeroize", +] + +[[package]] +name = "security-framework" +version = "3.7.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d" +dependencies = [ + "bitflags 2.13.0", + "core-foundation", + "core-foundation-sys", + "libc", + "security-framework-sys", +] + +[[package]] +name = "security-framework-sys" +version = "2.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3" +dependencies = [ + "core-foundation-sys", + "libc", +] + [[package]] name = "selectors" version = "0.36.1" @@ -4712,6 +5594,17 @@ dependencies = [ "digest 0.10.7", ] +[[package]] +name = "sha1" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "digest 0.11.3", +] + [[package]] name = "sha1-checked" version = "0.10.0" @@ -4719,7 +5612,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "89f599ac0c323ebb1c6082821a54962b839832b03984598375bff3975b804423" dependencies = [ "digest 0.10.7", - "sha1", + "sha1 0.10.6", ] [[package]] @@ -4776,6 +5669,16 @@ dependencies = [ "libc", ] +[[package]] +name = "signature" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de" +dependencies = [ + "digest 0.10.7", + "rand_core 0.6.4", +] + [[package]] name = "signature" version = "3.0.0" @@ -4864,6 +5767,22 @@ dependencies = [ "system-deps", ] +[[package]] +name = "spin" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "023a211cb3138dbc438680b32560ad89f699977624c9f8dbb95a47d5b4c07dd3" + +[[package]] +name = "spki" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d91ed6c858b01f942cd56b37a94b3e0a1798290327d1236e4d9cf4eaca44d29d" +dependencies = [ + "base64ct", + "der", +] + [[package]] name = "stable_deref_trait" version = "1.2.1" @@ -5055,7 +5974,7 @@ dependencies = [ "glob", "gtk", "heck 0.5.0", - "http", + "http 1.4.2", "jni", "libc", "log", @@ -5219,7 +6138,7 @@ dependencies = [ "cookie", "dpi", "gtk", - "http", + "http 1.4.2", "jni", "objc2", "objc2-ui-kit", @@ -5242,7 +6161,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4e6fac707727b7a2f48e4ded90976324267371073edbb415ffb73bb0458d203f" dependencies = [ "gtk", - "http", + "http 1.4.2", "jni", "log", "objc2", @@ -5274,7 +6193,7 @@ dependencies = [ "dom_query", "dunce", "glob", - "http", + "http 1.4.2", "infer", "json-patch", "log", @@ -5429,18 +6348,29 @@ checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" [[package]] name = "tokio" -version = "1.52.3" +version = "1.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe" +checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed" dependencies = [ "bytes", "libc", "mio", "pin-project-lite", + "signal-hook-registry", "socket2", "windows-sys 0.61.2", ] +[[package]] +name = "tokio-rustls" +version = "0.26.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" +dependencies = [ + "rustls", + "tokio", +] + [[package]] name = "tokio-util" version = "0.7.18" @@ -5598,8 +6528,8 @@ dependencies = [ "bitflags 2.13.0", "bytes", "futures-util", - "http", - "http-body", + "http 1.4.2", + "http-body 1.1.0", "pin-project-lite", "tower", "tower-layer", @@ -5626,9 +6556,21 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" dependencies = [ "pin-project-lite", + "tracing-attributes", "tracing-core", ] +[[package]] +name = "tracing-attributes" +version = "0.1.31" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.118", +] + [[package]] name = "tracing-core" version = "0.1.36" @@ -5790,6 +6732,12 @@ dependencies = [ "serde_derive", ] +[[package]] +name = "urlencoding" +version = "2.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "daf8dba3b7eb870caf1ddeed7bc9d2a049f3cfdfae7cb521b087cc33ae4c49da" + [[package]] name = "urlpattern" version = "0.3.0" @@ -5838,6 +6786,12 @@ version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" +[[package]] +name = "vsimd" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c3082ca00d5a5ef149bb8b555a72ae84c9c59f7250f013ac822ac2e49b19c64" + [[package]] name = "vswhom" version = "0.1.0" @@ -6567,7 +7521,7 @@ dependencies = [ "dunce", "gdkx11", "gtk", - "http", + "http 1.4.2", "javascriptcore-rs", "jni", "libc", @@ -6646,6 +7600,12 @@ dependencies = [ "time", ] +[[package]] +name = "xmlparser" +version = "0.13.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "66fee0b777b0f5ac1c69bb06d361268faafa61cd4682ae064a171c16c433e9e4" + [[package]] name = "yasna" version = "0.6.0" diff --git a/Cargo.toml b/Cargo.toml index 742e6e0..907cbe3 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,6 +1,6 @@ [workspace] resolver = "2" -members = ["crates/core", "crates/vfs", "crates/gitio", "crates/crypto", "crates/repo", "crates/cli", "crates/tlsio", "apps/desktop/src-tauri"] +members = ["crates/core", "crates/vfs", "crates/gitio", "crates/crypto", "crates/repo", "crates/cli", "crates/tlsio", "crates/objio", "apps/desktop/src-tauri"] exclude = ["vendor/glib-0.18.5-patched", "fuzz"] # Tauri 2's Linux GTK3 stack is pinned to glib 0.18.5, whose VariantStrIter diff --git a/ROADMAP.md b/ROADMAP.md index 0187d50..ad758ed 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -928,6 +928,54 @@ scale-&-reach horizon): stale pre-revoke manifest — the same gittuf-shaped signed-ref effort the P22 provenance boundary already defers. +- **Bucket compaction/gc (P36a follow-on).** The WAL log (`log/` keys, + each a hand-rolled versioned binary `walfmt` entry, not protobuf) grows + unboundedly with no compaction — every reader walks the full parent chain + back from `head_seq`, and no entry or superseded pack is ever removed. + Deferred until checkpointing (below) gives a safe compaction cutoff. +- **Leases (P36a follow-on).** The only cross-writer coordination today is + the manifest's compare-and-swap; there is no lease/TTL primitive for + operations that need to hold exclusive intent across more than one bucket + round-trip (e.g. a long-running compaction). Deferred. +- **Checkpoint fold (P36b, next).** `walfmt::Manifest.checkpoint_seq` + already reserves a field for a future compaction cutoff, but nothing + writes a `checkpoints//` object yet — cold start is always a full + log walk from `head_seq`. Folding the log into a periodic ref snapshot so + cold start becomes snapshot + short tail is the next bucket-remote phase. +- **Bucket-backed `sc serve` (P36c, next).** `sc serve` cannot host a + bucket as its backing store today — a bucket remote is written to + directly by every client's `sc push`/`sc fetch`, not brokered through a + server process. Wiring `BucketTransport` in as an `sc serve` backend + (so a bucket remote can also sit behind access control / resource limits + the way `.sc/` repos do via P29/P31) is deferred to a follow-on phase. +- **Partial clone from bucket remotes (P36a follow-on).** `sc clone + --filter` against `sc+wal://`/`sc+s3://` is refused outright + (`BucketTransport` has no per-prefix negotiation); teaching the WAL format + to support filtered reads is deferred. +- **Static-bundle clone offload (P36a follow-on).** walgit's bundle-uri + design (`docs/research/walgit-evaluation.md`) offloads full-repo clone + traffic to a CDN-servable static bundle instead of the WAL log/pack path; + an analogous static-bundle clone fast path for bucket remotes is deferred. +- **Native GCS backend (P36a follow-on).** `objio` ships `DirBucket` and + `S3Bucket` (S3-compatible); a native Google Cloud Storage backend behind + the same `Bucket` trait is deferred until a concrete need surfaces. +- **`S3Bucket` streaming (P36a follow-on).** `S3Bucket`'s bodies are + currently buffered fully in RAM — each object is `MAX_OBJECT_SIZE`-bounded + individually, but a transfer moves a whole pack at a time, so memory use + is pack-sized, not object-sized. Streaming the S3 request/response bodies + instead of buffering them whole is deferred. +- **Incremental `refresh()` for bucket remotes (P36a follow-on).** + `BucketTransport::refresh` short-circuits on an unchanged manifest tag, + but whenever the manifest *has* changed it re-walks the full parent chain + from `head_seq` and re-fetches every `idx_key` on that chain from + scratch — O(chain) GETs per manifest change, which gets expensive under + fleet-frequency pushes. An incremental refresh that picks up from the + last-seen manifest/seq instead of re-walking from scratch is deferred. +- **Bucket-aware push negotiation (P36a follow-on).** Push negotiation today + issues a `has_object` round trip per object over S3; batching those probes + into one `refresh()` plus local index lookups (instead of one S3 round + trip per object) is deferred. + ## How a phase gets built 1. Focused brainstorm for the phase (this skill) → phase spec. diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs index 2161136..8674691 100644 --- a/crates/cli/src/main.rs +++ b/crates/cli/src/main.rs @@ -3647,6 +3647,9 @@ fn run_remote(op: RemoteOp) -> Result<()> { if url.starts_with("ssh://") { scl_repo::SshUrl::parse(&url)?; // fail fast on malformed URLs } + if url.starts_with("sc+wal://") || url.starts_with("sc+s3://") { + scl_repo::BucketUrl::parse(&url)?; // fail fast on malformed URLs + } repo.remote_add(&name, &url)?; println!("added remote {name} -> {url}"); } diff --git a/crates/cli/tests/bucket_remote.rs b/crates/cli/tests/bucket_remote.rs new file mode 100644 index 0000000..94f5079 --- /dev/null +++ b/crates/cli/tests/bucket_remote.rs @@ -0,0 +1,53 @@ +//! `sc+wal://` bucket remotes through the real binary. Wire/WAL correctness +//! is proven in scl-repo's bucket_transport tests; this exercises CLI +//! plumbing: remote add validation, push, clone, fetch. + +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +fn sc(dir: &Path, args: &[&str]) -> Output { + let mut cmd = Command::new(env!("CARGO_BIN_EXE_sc")); + cmd.args(args).current_dir(dir); + cmd.output().expect("sc runs") +} + +fn tmp(tag: &str) -> PathBuf { + let d = std::env::temp_dir().join(format!("scl-cli-bucket-{tag}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + d +} + +#[test] +fn bucket_clone_push_fetch_round_trip_and_url_validation() { + let a = tmp("a"); + let bucket = tmp("bucket"); + let url = format!("sc+wal://{}", bucket.display()); + + let out = sc(&a, &["init"]); + assert!(out.status.success(), "{out:?}"); + std::fs::write(a.join("f.txt"), b"via cli").unwrap(); + assert!(sc(&a, &["commit", "-m", "c1"]).status.success()); + // malformed bucket URL is refused at add time + let bad = sc(&a, &["remote", "add", "borig", "sc+s3://"]); + assert!(!bad.status.success()); + assert!(sc(&a, &["remote", "add", "origin", &url]).status.success()); + assert!(sc(&a, &["push", "origin"]).status.success()); + + let parent = tmp("bparent"); + let b = parent.join("b"); + let out = sc(&parent, &["clone", &url, b.to_str().unwrap()]); + assert!(out.status.success(), "{out:?}"); + assert_eq!(std::fs::read(b.join("f.txt")).unwrap(), b"via cli"); + + std::fs::write(b.join("g.txt"), b"round trip").unwrap(); + assert!(sc(&b, &["commit", "-m", "c2"]).status.success()); + assert!(sc(&b, &["push", "origin"]).status.success()); + let out = sc(&a, &["fetch", "origin"]); + assert!(out.status.success(), "{out:?}"); + + for d in [&a, &bucket, &parent] { + std::fs::remove_dir_all(d).unwrap(); + assert!(!d.exists()); + } +} diff --git a/crates/core/src/pack.rs b/crates/core/src/pack.rs index e3e8f40..87119bf 100644 --- a/crates/core/src/pack.rs +++ b/crates/core/src/pack.rs @@ -198,13 +198,63 @@ pub fn parse_index(idx: &[u8]) -> Result> { } /// Read the record at `offset` from `pack`, decompress, verify it hashes to -/// `id`, and decode it. +/// `id`, and decode it. **Trusted-path, unbounded by design** (ADR-0039's +/// explicit scoping: "the cap is transfer-path only, not a local-commit +/// limit... `read_object_at`'s local already-verified on-disk pack is +/// untouched"). `Store::read_pack_object` is this function's one caller +/// (`crates/core/src/store.rs`), reading packs `sc gc`'s own repack wrote to +/// this process's `.sc/objects/`: `write_pack` performs no size validation, +/// so a locally-committed blob legitimately larger than `MAX_OBJECT_SIZE` +/// ("committable but not transferable", per ADR-0039's final-review +/// addendum) must still round-trip through a repack, or `sc gc` would +/// silently make that blob permanently unreadable. Do **not** add a length +/// or decompressed-size cap here. Reading pack bytes from an UNTRUSTED +/// source (a bucket remote, a wire peer) must go through +/// [`read_object_at_bounded`] instead. pub fn read_object_at(pack: &[u8], offset: u64, id: &ObjectId) -> Result { + let (start, end) = record_bounds(pack, offset, None)?; + decompress_and_decode(&pack[start..end], id, None) +} + +/// Like [`read_object_at`], but for pack bytes read from an **untrusted** +/// source (P36a's bucket remotes are the one caller today, +/// `crates/repo/src/bucket_transport.rs`): caps both the compressed record +/// length and the decompressed output at `max`, mirroring +/// `parse_pack_reader`'s decode-WITH-LIMIT guard (ADR-0039) — a hostile pack +/// cannot claim an oversized record, nor mount a "zstd bomb" (a small +/// compressed payload that decompresses to gigabytes), against a caller that +/// has no other reason to trust the bytes it was handed. Callers reading a +/// pack this process wrote and already verified itself (a local on-disk +/// repack) should use [`read_object_at`] instead — capping that path too +/// would make a legitimately-committed `>max` blob permanently unreadable +/// after `sc gc` (see that function's doc comment). +pub fn read_object_at_bounded( + pack: &[u8], + offset: u64, + id: &ObjectId, + max: usize, +) -> Result { + let (start, end) = record_bounds(pack, offset, Some(max))?; + decompress_and_decode(&pack[start..end], id, Some(max)) +} + +/// Shared bounds-check for one record's `[compressed_len:4][data:N]` framing. +/// `max`, when `Some`, rejects a compressed length prefix over the cap +/// before it's ever used to slice `pack` — shared by [`read_object_at`] +/// (`max = None`) and [`read_object_at_bounded`] (`max = Some(cap)`). +fn record_bounds(pack: &[u8], offset: u64, max: Option) -> Result<(usize, usize)> { let off = offset as usize; if off + 4 > pack.len() { return Err(Error::PackCorrupt(format!("offset {offset} past end"))); } let len = u32::from_le_bytes(pack[off..off + 4].try_into().unwrap()) as usize; + if let Some(max) = max { + if len > max { + return Err(Error::PackCorrupt(format!( + "record compressed length {len} exceeds MAX_OBJECT_SIZE (256 MiB) transfer limit" + ))); + } + } let start = off + 4; let end = start + len; if end > pack.len() { @@ -212,13 +262,38 @@ pub fn read_object_at(pack: &[u8], offset: u64, id: &ObjectId) -> Result "record at {offset} runs past end" ))); } - decompress_and_decode(&pack[start..end], id) + Ok((start, end)) } -/// Decompress one record payload, verify against `id`, decode. -fn decompress_and_decode(payload: &[u8], id: &ObjectId) -> Result { - let canonical = zstd::decode_all(std::io::Cursor::new(payload)) - .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; +/// Decompress one record payload, verify against `id`, decode. `max`, when +/// `Some`, bounds the decompressed output: a small compressed payload can +/// still decompress to an enormous plaintext (a "zstd bomb"), so this reads +/// at most `max + 1` bytes from the decoder — enough to detect and reject an +/// over-cap output without ever materializing it in full (same +/// `take`-bounded pattern as `parse_pack_reader`). `max = None` (the +/// trusted [`read_object_at`] path) decompresses in one unbounded shot, by +/// design — see that function's doc comment. +fn decompress_and_decode(payload: &[u8], id: &ObjectId, max: Option) -> Result { + let canonical = match max { + None => zstd::decode_all(std::io::Cursor::new(payload)) + .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?, + Some(max) => { + let mut decoder = zstd::stream::read::Decoder::new(std::io::Cursor::new(payload)) + .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; + let mut canonical = Vec::new(); + decoder + .by_ref() + .take(max as u64 + 1) + .read_to_end(&mut canonical) + .map_err(|e| Error::PackCorrupt(format!("zstd decode failed: {e}")))?; + if canonical.len() > max { + return Err(Error::PackCorrupt( + "decompressed object exceeds MAX_OBJECT_SIZE (256 MiB) transfer limit".into(), + )); + } + canonical + } + }; if ObjectId::of(&canonical) != *id { return Err(Error::Malformed(format!( "packed object {id} failed hash verification" @@ -544,6 +619,71 @@ mod tests { assert!(matches!(err, Error::PackCorrupt(_)), "got {err:?}"); } + #[test] + fn read_object_at_bounded_rejects_over_cap_compressed_length() { + // `read_object_at_bounded` is the random-access counterpart to + // `pack_record_over_cap_rejected` above: its own compressed-length + // prefix must be capped too, since `BucketTransport::object_bytes` + // (P36a) feeds it bytes fetched straight from a remote bucket — + // untrusted input, unlike `read_object_at`'s one caller (`Store`'s + // own self-written packs, ADR-0039's explicit unbounded carve-out). + let id = ObjectId::of(b"whatever"); + let over = (crate::MAX_OBJECT_SIZE + 1) as u32; + let mut pack = Vec::new(); + pack.extend_from_slice(&over.to_le_bytes()); // the length prefix read at offset 0 + let err = read_object_at_bounded(&pack, 0, &id, crate::MAX_OBJECT_SIZE).unwrap_err(); + assert!(matches!(err, Error::PackCorrupt(_)), "got {err:?}"); + } + + #[test] + fn read_object_at_bounded_zstd_bomb_rejected() { + // Mirrors `zstd_bomb_rejected`, but through `read_object_at_bounded`'s + // decode path — the bounded sibling `BucketTransport::object_bytes` + // (P36a) calls, since a hostile bucket-served pack could otherwise + // OOM it on a single `get_object` call. `read_object_at` itself (the + // trusted, unbounded path) deliberately does NOT get this guard — + // see `read_object_at_reads_a_legitimately_oversized_local_record` + // below for the regression that pins the other side of that split. + let bomb_plain = vec![0u8; crate::MAX_OBJECT_SIZE + 1024]; + let compressed = + zstd::encode_all(std::io::Cursor::new(&bomb_plain[..]), COMPRESSION_LEVEL).unwrap(); + assert!( + compressed.len() < bomb_plain.len() / 10, + "expected the all-zero payload to compress small" + ); + + // A wrong id is fine here — the size cap must fire before hash + // verification ever runs. + let id = ObjectId::of(b"whatever"); + let mut pack = Vec::new(); + pack.extend_from_slice(&(compressed.len() as u32).to_le_bytes()); + pack.extend_from_slice(&compressed); + + let err = read_object_at_bounded(&pack, 0, &id, crate::MAX_OBJECT_SIZE).unwrap_err(); + assert!(matches!(err, Error::PackCorrupt(_)), "got {err:?}"); + } + + #[test] + fn read_object_at_reads_a_legitimately_oversized_local_record() { + // ADR-0039's explicit scope: "the cap is transfer-path only, not a + // local-commit limit... read_object_at's local already-verified + // on-disk pack is untouched" — a locally-committed blob legitimately + // larger than MAX_OBJECT_SIZE ("committable but not transferable") + // must still round-trip through `sc gc`'s repack, since `write_pack` + // performs no size validation and a cap here would surface as + // silent, permanent unreadability on the next read. This is the + // regression pin: if `read_object_at` ever regains a cap (as it + // briefly did in an earlier, incorrect fix for the zstd-bomb + // finding), this test catches it directly, without relying only on + // the doc-comment contract. + let big = Object::blob(vec![7u8; crate::MAX_OBJECT_SIZE + 4096]); + let (id, bytes) = enc(&big); + let (pack, idx) = build_pack(&[(id, bytes.clone())]).unwrap(); + let e = parse_index(&idx).unwrap().pop().unwrap(); + let got = read_object_at(&pack, e.offset, &id).unwrap(); + assert_eq!(got.encode(), bytes); + } + #[test] fn parse_index_rejects_overflowing_count() { // Magic b"SCIX", version 1 LE, count = 2^60 — count * 48 wraps to 0 on u64, diff --git a/crates/objio/Cargo.toml b/crates/objio/Cargo.toml new file mode 100644 index 0000000..e05e18f --- /dev/null +++ b/crates/objio/Cargo.toml @@ -0,0 +1,17 @@ +[package] +name = "scl-objio" +version.workspace = true +edition.workspace = true +license.workspace = true +publish.workspace = true + +[dependencies] +aws-config = "1.11.0" +aws-sdk-s3 = { version = "1.144.0", default-features = false, features = ["sigv4a", "http-1x", "default-https-client", "rt-tokio"] } +blake3 = "1.8.7" +hex = "0.4.3" +thiserror = "2.0.18" +tokio = { version = "1.53.1", features = ["rt"] } + +[lints] +workspace = true diff --git a/crates/objio/src/dir.rs b/crates/objio/src/dir.rs new file mode 100644 index 0000000..6ea3f28 --- /dev/null +++ b/crates/objio/src/dir.rs @@ -0,0 +1,154 @@ +//! Local-directory `Bucket` — the test/demo backend behind `sc+wal://`. + +use crate::{validate_key, Bucket, Error, Fetched, Result}; +use std::path::{Path, PathBuf}; + +/// A local filesystem bucket implementation using a directory as the backing +/// store. Tags are blake3 hashes of content; compare-and-swap writes are +/// serialized via a spin-lock file to simulate S3 precondition semantics. +pub struct DirBucket { + root: PathBuf, +} + +fn tag_of(bytes: &[u8]) -> String { + hex::encode(blake3::hash(bytes).as_bytes()) +} + +impl DirBucket { + /// Open (creating if needed) a directory as a bucket. + pub fn open(root: impl Into) -> Result { + let root = root.into(); + std::fs::create_dir_all(&root)?; + Ok(DirBucket { root }) + } + + fn key_path(&self, key: &str) -> Result { + validate_key(key)?; + Ok(self.root.join(key)) + } + + /// Spin-acquire `/.cas-lock`; bounded so a crashed holder surfaces + /// as a loud error, not a hang. + fn lock(&self) -> Result { + let path = self.root.join(".cas-lock"); + for _ in 0..2000 { + match std::fs::OpenOptions::new() + .write(true) + .create_new(true) + .open(&path) + { + Ok(_) => return Ok(CasLock { path }), + Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => { + std::thread::sleep(std::time::Duration::from_millis(1)); + } + Err(e) => return Err(e.into()), + } + } + Err(Error::Backend(format!( + "cas lock stuck (stale {} ?)", + path.display() + ))) + } + + fn write_via_tmp(&self, path: &Path, bytes: &[u8]) -> Result<()> { + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent)?; + } + let tmp = path.with_extension(format!("tmp-{}", std::process::id())); + std::fs::write(&tmp, bytes)?; + std::fs::rename(&tmp, path)?; + Ok(()) + } +} + +struct CasLock { + path: PathBuf, +} +impl Drop for CasLock { + fn drop(&mut self) { + let _ = std::fs::remove_file(&self.path); + } +} + +impl Bucket for DirBucket { + fn get(&self, key: &str, cached_tag: Option<&str>) -> Result { + let path = self.key_path(key)?; + let bytes = match std::fs::read(&path) { + Ok(b) => b, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(Fetched::Absent), + Err(e) => return Err(e.into()), + }; + let tag = tag_of(&bytes); + if cached_tag == Some(tag.as_str()) { + return Ok(Fetched::Unchanged); + } + Ok(Fetched::New { bytes, tag }) + } + + fn put_new(&self, key: &str, bytes: &[u8]) -> Result { + let path = self.key_path(key)?; + let _lock = self.lock()?; + if path.exists() { + return Ok(false); + } + self.write_via_tmp(&path, bytes)?; + Ok(true) + } + + fn put_if_tag( + &self, + key: &str, + bytes: &[u8], + expected_tag: Option<&str>, + ) -> Result> { + let path = self.key_path(key)?; + let _lock = self.lock()?; + let current = match std::fs::read(&path) { + Ok(b) => Some(tag_of(&b)), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => None, + Err(e) => return Err(e.into()), + }; + if current.as_deref() != expected_tag { + return Ok(None); + } + self.write_via_tmp(&path, bytes)?; + Ok(Some(tag_of(bytes))) + } + + fn list(&self, prefix: &str) -> Result> { + fn walk(dir: &Path, root: &Path, out: &mut Vec) -> std::io::Result<()> { + let rd = match std::fs::read_dir(dir) { + Ok(rd) => rd, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(()), + Err(e) => return Err(e), + }; + for entry in rd { + let entry = entry?; + let p = entry.path(); + if p.file_name().is_some_and(|n| { + let n = n.to_string_lossy(); + n == ".cas-lock" || n.contains(".tmp-") + }) { + continue; + } + if p.is_dir() { + walk(&p, root, out)?; + } else { + out.push( + p.strip_prefix(root) + .unwrap() + .to_string_lossy() + .replace('\\', "/"), + ); + } + } + Ok(()) + } + validate_key(prefix.trim_end_matches('/'))?; + let mut out = Vec::new(); + walk(&self.root, &self.root, &mut out)?; + out.retain(|k| k.starts_with(prefix)); + out.sort(); + Ok(out) + } +} diff --git a/crates/objio/src/lib.rs b/crates/objio/src/lib.rs new file mode 100644 index 0000000..9900051 --- /dev/null +++ b/crates/objio/src/lib.rs @@ -0,0 +1,135 @@ +//! Object-storage access for sc bucket remotes (P36). +//! +//! Quarantine rule: object-store SDKs live here and only here — the rest of +//! the workspace sees the [`Bucket`] trait. Leaf crate: depends on no other +//! workspace crate (like `tlsio`). + +mod dir; +pub use dir::DirBucket; +mod s3; +pub use s3::S3Bucket; + +/// Errors from the object-storage bucket layer. Variants indicate key +/// validation failure, I/O errors from the backing store, or backend-specific +/// errors (e.g., lock starvation in the local directory backend). +#[derive(Debug, thiserror::Error)] +pub enum Error { + #[error("bucket key rejected: {0}")] + BadKey(String), + #[error("bucket io: {0}")] + Io(#[from] std::io::Error), + #[error("bucket backend: {0}")] + Backend(String), +} + +pub type Result = std::result::Result; + +/// Outcome of a conditional read. +pub enum Fetched { + /// The caller's cached tag still matches — no bytes transferred. + Unchanged, + /// The key does not exist. + Absent, + /// Current value plus its tag (ETag / content hash). + New { bytes: Vec, tag: String }, +} + +/// A flat key/value object store with the three primitives the WAL needs: +/// conditional read, create-if-absent, and compare-and-swap overwrite. +pub trait Bucket: Send { + /// Conditional read. `cached_tag` matching the current value returns + /// [`Fetched::Unchanged`] without transferring bytes. + fn get(&self, key: &str, cached_tag: Option<&str>) -> Result; + /// Create-only write (if-none-match). `Ok(false)` = key already exists; + /// the existing value is never touched. + fn put_new(&self, key: &str, bytes: &[u8]) -> Result; + /// Compare-and-swap overwrite. `expected_tag: None` = create-new. + /// `Ok(None)` = precondition failed (someone else won); `Ok(Some(tag))` + /// = committed, with the new value's tag. + fn put_if_tag( + &self, + key: &str, + bytes: &[u8], + expected_tag: Option<&str>, + ) -> Result>; + /// Keys under `prefix`, sorted. + fn list(&self, prefix: &str) -> Result>; +} + +/// Reject traversal and absolute keys before any backend touches them. +pub(crate) fn validate_key(key: &str) -> Result<()> { + if key.is_empty() + || key.starts_with('/') + || key.contains('\\') + || key + .split('/') + .any(|c| c.is_empty() || c == "." || c == "..") + || key.chars().any(|c| c.is_whitespace() || c.is_control()) + { + return Err(Error::BadKey(format!("{key:?}"))); + } + Ok(()) +} + +/// Conformance checks every `Bucket` implementation must pass. +/// Panics on violation (test helper). +pub fn contract_suite(b: &dyn Bucket) { + // absent key + assert!(matches!(b.get("manifest", None).unwrap(), Fetched::Absent)); + // put_if_tag with expected None = create; returns the new tag + let t1 = b + .put_if_tag("manifest", b"v1", None) + .unwrap() + .expect("create succeeds"); + // create again must fail (precondition) + assert!(b.put_if_tag("manifest", b"v1x", None).unwrap().is_none()); + // conditional get: matching tag => Unchanged; stale/no tag => New with same tag + assert!(matches!( + b.get("manifest", Some(&t1)).unwrap(), + Fetched::Unchanged + )); + let Fetched::New { bytes, tag } = b.get("manifest", None).unwrap() else { + panic!("expected New") + }; + assert_eq!(bytes, b"v1"); + assert_eq!(tag, t1); + // CAS: wrong tag refused, right tag succeeds and returns a new tag + assert!(b + .put_if_tag("manifest", b"v2", Some("bogus")) + .unwrap() + .is_none()); + let t2 = b + .put_if_tag("manifest", b"v2", Some(&t1)) + .unwrap() + .expect("cas succeeds"); + assert_ne!(t1, t2); + // put_new: first write true, second false, content untouched + assert!(b.put_new("log/1", b"entry-one").unwrap()); + assert!(!b.put_new("log/1", b"entry-two").unwrap()); + let Fetched::New { bytes, .. } = b.get("log/1", None).unwrap() else { + panic!() + }; + assert_eq!(bytes, b"entry-one"); + // list is prefix-scoped and sorted + assert!(b.put_new("log/2", b"x").unwrap()); + assert_eq!( + b.list("log/").unwrap(), + vec!["log/1".to_string(), "log/2".to_string()] + ); + assert_eq!(b.list("packs/").unwrap(), Vec::::new()); +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn dir_bucket_passes_contract() { + let root = std::env::temp_dir().join(format!("scl-objio-dir-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + let b = DirBucket::open(&root).unwrap(); + contract_suite(&b); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } +} diff --git a/crates/objio/src/s3.rs b/crates/objio/src/s3.rs new file mode 100644 index 0000000..2eb9d36 --- /dev/null +++ b/crates/objio/src/s3.rs @@ -0,0 +1,193 @@ +//! S3-compatible `Bucket` backend (AWS, MinIO, R2 via AWS_ENDPOINT_URL_S3). + +use crate::{validate_key, Bucket, Error, Fetched, Result}; + +/// A `Bucket` backed by an S3-compatible object store. Credentials, region, +/// and endpoint come entirely from the SDK's standard chain (`AWS_*` env, +/// shared config/credentials files, `AWS_ENDPOINT_URL_S3` for MinIO/R2/etc.) +/// — this crate adds no credential surface of its own. Runs its own +/// current-thread tokio runtime internally so the trait stays synchronous. +pub struct S3Bucket { + rt: tokio::runtime::Runtime, + client: aws_sdk_s3::Client, + bucket: String, + prefix: String, +} + +impl S3Bucket { + /// Connect to `bucket`, scoping every key under `prefix`, using the + /// SDK's standard credential/region/endpoint resolution chain. + pub fn open(bucket: &str, prefix: &str) -> Result { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .map_err(|e| Error::Backend(format!("tokio runtime: {e}")))?; + let conf = rt.block_on(aws_config::load_defaults( + aws_config::BehaviorVersion::latest(), + )); + Ok(S3Bucket { + client: aws_sdk_s3::Client::new(&conf), + rt, + bucket: bucket.to_string(), + prefix: prefix.trim_matches('/').to_string(), + }) + } + + /// Validate `key` and join it onto this bucket's prefix. + fn full_key(&self, key: &str) -> Result { + validate_key(key)?; + Ok(if self.prefix.is_empty() { + key.to_string() + } else { + format!("{}/{key}", self.prefix) + }) + } +} + +/// HTTP status of a failed S3 SDK call, if the error carries a raw response. +fn http_status(err: &aws_sdk_s3::error::SdkError) -> Option { + err.raw_response().map(|r| r.status().as_u16()) +} + +impl Bucket for S3Bucket { + fn get(&self, key: &str, cached_tag: Option<&str>) -> Result { + let k = self.full_key(key)?; + let mut req = self.client.get_object().bucket(&self.bucket).key(&k); + if let Some(tag) = cached_tag { + req = req.if_none_match(tag); + } + let send_result = self.rt.block_on(req.send()); + match send_result { + Ok(out) => { + let tag = out.e_tag().unwrap_or_default().to_string(); + let bytes = self + .rt + .block_on(out.body.collect()) + .map_err(|e| Error::Backend(format!("s3 get body {k}: {e}")))? + .into_bytes() + .to_vec(); + Ok(Fetched::New { bytes, tag }) + } + Err(e) => { + // 304 => Unchanged (if_none_match precondition); NoSuchKey/404 => Absent. + let status = http_status(&e); + let code = e + .as_service_error() + .and_then(aws_sdk_s3::error::ProvideErrorMetadata::code) + .unwrap_or_default(); + match (code, status) { + (_, Some(304)) => Ok(Fetched::Unchanged), + // Checked before the bare-404 fallback below because a + // missing bucket surfaces as 404 too. When the backend + // supplies the body error code, a missing bucket is a + // hard config error, not an absent object — without this + // arm it would fall through to `Fetched::Absent` and make + // a typo'd/nonexistent bucket look like a real, empty + // remote (`sc fetch` "succeeds" against nothing). Some + // S3-compatibles omit the code on a 404 entirely; those + // still classify as `Absent` here (bare-404 fallback, + // unchanged) — distinguishing that case needs a + // `HeadBucket` probe, not attempted here. + ("NoSuchBucket", _) => Err(Error::Backend(format!( + "s3 bucket {} does not exist (check the sc+s3:// url)", + self.bucket + ))), + ("NoSuchKey", _) | (_, Some(404)) => Ok(Fetched::Absent), + _ => Err(Error::Backend(format!("s3 get {k}: {e}"))), + } + } + } + } + + fn put_new(&self, key: &str, bytes: &[u8]) -> Result { + let k = self.full_key(key)?; + let req = self + .client + .put_object() + .bucket(&self.bucket) + .key(&k) + .if_none_match("*") + .body(bytes.to_vec().into()); + match self.rt.block_on(req.send()) { + Ok(_) => Ok(true), + Err(e) if http_status(&e) == Some(412) => Ok(false), + Err(e) => Err(Error::Backend(format!("s3 put_new {k}: {e}"))), + } + } + + fn put_if_tag( + &self, + key: &str, + bytes: &[u8], + expected_tag: Option<&str>, + ) -> Result> { + let k = self.full_key(key)?; + let mut req = self + .client + .put_object() + .bucket(&self.bucket) + .key(&k) + .body(bytes.to_vec().into()); + req = match expected_tag { + Some(tag) => req.if_match(tag), + None => req.if_none_match("*"), + }; + match self.rt.block_on(req.send()) { + Ok(out) => Ok(Some(out.e_tag().unwrap_or_default().to_string())), + Err(e) if http_status(&e) == Some(412) => Ok(None), + Err(e) => Err(Error::Backend(format!("s3 put_if_tag {k}: {e}"))), + } + } + + fn list(&self, prefix: &str) -> Result> { + let full = self.full_key(prefix.trim_end_matches('/'))?; + let mut out = Vec::new(); + let mut cont: Option = None; + loop { + let mut req = self + .client + .list_objects_v2() + .bucket(&self.bucket) + .prefix(format!("{full}/")); + if let Some(c) = &cont { + req = req.continuation_token(c); + } + let resp = self + .rt + .block_on(req.send()) + .map_err(|e| Error::Backend(format!("s3 list: {e}")))?; + for obj in resp.contents() { + if let Some(k) = obj.key() { + let rel = k + .strip_prefix(&self.prefix) + .unwrap_or(k) + .trim_start_matches('/'); + out.push(rel.to_string()); + } + } + match resp.next_continuation_token() { + Some(c) => cont = Some(c.to_string()), + None => break, + } + } + out.sort(); + Ok(out) + } +} + +#[cfg(test)] +mod tests { + /// Live-backend parity: set SC_OBJIO_S3_BUCKET (and standard AWS_* env, + /// e.g. AWS_ENDPOINT_URL_S3 for MinIO) to run; skipped otherwise so CI + /// stays hermetic on DirBucket. + #[test] + fn s3_bucket_passes_contract_when_configured() { + let Ok(bucket) = std::env::var("SC_OBJIO_S3_BUCKET") else { + eprintln!("skipped: SC_OBJIO_S3_BUCKET not set"); + return; + }; + let prefix = format!("scl-objio-contract-{}", std::process::id()); + let b = super::S3Bucket::open(&bucket, &prefix).unwrap(); + crate::contract_suite(&b); + } +} diff --git a/crates/repo/Cargo.toml b/crates/repo/Cargo.toml index d30fa3b..d75f549 100644 --- a/crates/repo/Cargo.toml +++ b/crates/repo/Cargo.toml @@ -15,6 +15,7 @@ toml = "1" serde = { version = "1", features = ["derive"] } libc = "0.2.186" scl-tlsio = { version = "0.1.0", path = "../tlsio" } +scl-objio = { version = "0.1.0", path = "../objio" } hex = "0.4.3" blake3 = "1.8.5" diff --git a/crates/repo/src/bucket_transport.rs b/crates/repo/src/bucket_transport.rs new file mode 100644 index 0000000..7a0fdce --- /dev/null +++ b/crates/repo/src/bucket_transport.rs @@ -0,0 +1,1170 @@ +//! A [`Transport`] over an object-store bucket (P36a): immutable packs + +//! parent-linked log entries, one CAS'd manifest as the sole commit point. +//! See ADR-0046 and docs/superpowers/specs/2026-08-26-wal-bucket-backend-design.md. + +use crate::error::{Error, Result}; +use crate::transport::Transport; +use crate::walfmt::{idx_key, log_key, pack_key, LogEntry, Manifest, RefUpdate}; +use scl_core::pack::{parse_index, read_object_at_bounded, IndexEntry, PackWriter}; +use scl_core::{Object, ObjectId}; +use scl_objio::{Bucket, Fetched}; +use std::cell::RefCell; +use std::collections::BTreeMap; +use std::io::Write; + +/// Reconstructed state of the WAL at one manifest tag: the manifest itself, +/// the flattened branch tips after replaying the parent chain oldest-first, +/// and a lookup from object id to the pack that holds it. +struct WalView { + /// The bucket's `ETag`/content-hash for the manifest this view was built + /// from — handed back to `Bucket::get` on the next `refresh` so an + /// unchanged manifest costs one conditional GET, not a full rebuild. + tag: String, + manifest: Manifest, + /// branch -> tip, after replaying the parent chain oldest-first. + refs: BTreeMap, + /// object id -> (pack hash, offset, length), from every on-chain pack's idx. + index: BTreeMap, +} + +/// A [`Transport`] whose object graph and refs live entirely in an +/// object-storage bucket (S3-compatible or a local directory), read through +/// the parent-linked WAL log format `walfmt` defines. Readers walk the log +/// backward from the manifest's `head_seq` via `parent_seq` links — a log +/// entry's own claimed `seq` is never trusted for reachability, only for +/// self-consistency (it must match the slot it was read from and its parent +/// must strictly precede it). An entry not on that chain (e.g. a losing +/// racer's orphaned append) is invisible to every read method here, by +/// construction. +pub struct BucketTransport { + bucket: Box, + view: RefCell>, + /// Body bytes of the most recently used pack (walk locality: `get_pack` + /// and repeated `get_object` calls tend to hit the same pack back to + /// back, so caching the last one avoids re-fetching it byte for byte). + pack_cache: RefCell)>>, + /// Objects staged by `put_object`, flushed into one pack the next time + /// `update_ref` is called (mirrors `LocalTransport`'s contract: `put_pack` + /// writes objects up front, but `put_object` callers stage one at a time + /// and expect the ref move to be what makes them durable-and-visible). + staged: RefCell)>>, + /// Hashes of packs already uploaded (via `put_pack` or a `put_object` + /// flush) that the *next* `update_ref` call's log entry must reference. + /// `update_ref` takes this field's contents at the top of the call into + /// a call-local copy — it survives that one call's internal CAS retries, + /// but never leaks into a later, unrelated `update_ref` call on the same + /// (possibly long-lived, e.g. `wire::serve`-hosted) transport regardless + /// of whether this call succeeds or fails. + pending_packs: RefCell>, +} + +/// Untrusted-length guard (P28 parity): refuse any WAL metadata value — +/// manifest, log entry, idx — larger than `MAX_OBJECT_SIZE` before decoding. +/// Pack bodies are exempt from this particular guard (they may legitimately +/// exceed it), but are not unguarded: `object_bytes` reads a pack body via +/// `scl_core::pack::read_object_at_bounded` (not the unbounded +/// `read_object_at` — that path is reserved for `Store`'s own +/// already-verified on-disk packs per ADR-0039's explicit trust split), +/// which caps both the compressed record length and the decompressed output +/// at `MAX_OBJECT_SIZE`, mirroring `parse_pack_reader`'s bounded decode. A +/// hostile bucket pack therefore cannot mount a decompression-bomb DoS +/// against `get_object`/`get_pack`. +fn capped(what: &str, bytes: Vec) -> Result> { + if bytes.len() > scl_core::MAX_OBJECT_SIZE { + return Err(Error::Wal(format!( + "{what} exceeds MAX_OBJECT_SIZE (256 MiB)" + ))); + } + Ok(bytes) +} + +/// Which bucket backend a [`BucketUrl`] names. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum BucketScheme { + /// `sc+wal://` — a local directory used as the test/demo backend. + Wal, + /// `sc+s3://` — an S3-compatible object store. + S3, +} + +/// A parsed `sc+wal://` or `sc+s3:///` remote URL. +/// `sc+wal` treats everything after the scheme as a directory path (so +/// `sc+wal:///abs/path` and `sc+wal://rel/path` both work); `sc+s3` splits +/// the first path component off as the bucket name and keeps the remainder +/// (possibly empty) as the key prefix. +pub struct BucketUrl { + pub scheme: BucketScheme, + pub bucket: String, + pub prefix: String, +} + +impl BucketUrl { + /// Parse a bucket URL; anything malformed is `InvalidArgument` with a + /// message naming the URL, so `remote add` can fail fast — including + /// embedded CR/LF, which would otherwise smuggle extra "lines" into + /// anything that later logs or shells out with the raw URL. + pub fn parse(url: &str) -> Result { + let (scheme, rest) = if let Some(r) = url.strip_prefix("sc+wal://") { + (BucketScheme::Wal, r) + } else if let Some(r) = url.strip_prefix("sc+s3://") { + (BucketScheme::S3, r) + } else { + return Err(Error::InvalidArgument(format!( + "not an sc+wal:// or sc+s3:// url: {url}" + ))); + }; + if rest.is_empty() || rest.chars().any(|c| c == '\r' || c == '\n') { + return Err(Error::InvalidArgument(format!("bad bucket url: {url}"))); + } + Ok(match scheme { + BucketScheme::Wal => BucketUrl { + scheme, + bucket: rest.to_string(), + prefix: String::new(), + }, + BucketScheme::S3 => { + let (bucket, prefix) = rest.split_once('/').unwrap_or((rest, "")); + if bucket.is_empty() { + return Err(Error::InvalidArgument(format!("bad bucket url: {url}"))); + } + BucketUrl { + scheme, + bucket: bucket.to_string(), + prefix: prefix.trim_matches('/').to_string(), + } + } + }) + } +} + +impl BucketTransport { + /// Open the right bucket backend for a bucket URL (`sc+wal://` opens a + /// local-directory test/demo backend, `sc+s3://` opens the real + /// S3-compatible backend), then wrap it as a `Transport` the same way + /// [`BucketTransport::from_bucket`] does. + pub fn open(url: &str) -> Result { + let parsed = BucketUrl::parse(url)?; + let bucket: Box = match parsed.scheme { + BucketScheme::Wal => Box::new(scl_objio::DirBucket::open(&parsed.bucket)?), + BucketScheme::S3 => { + Box::new(scl_objio::S3Bucket::open(&parsed.bucket, &parsed.prefix)?) + } + }; + BucketTransport::from_bucket(bucket) + } + + /// Open a transport directly over an already-constructed bucket backend. + /// Fails only if the initial `refresh` (a conditional GET of the + /// manifest plus a walk of its parent chain) errors — an entirely empty + /// bucket is a valid, successfully-opened remote with no refs yet. + pub fn from_bucket(bucket: Box) -> Result { + let t = BucketTransport { + bucket, + view: RefCell::new(None), + pack_cache: RefCell::new(None), + staged: RefCell::new(Vec::new()), + pending_packs: RefCell::new(Vec::new()), + }; + t.refresh()?; + Ok(t) + } + + /// One conditional GET of the manifest; on change, rebuild refs + index + /// by walking parent links head -> 0 (seq numbers are claims; the chain + /// is the truth — off-chain entries are garbage). + fn refresh(&self) -> Result<()> { + let cached_tag = self.view.borrow().as_ref().map(|v| v.tag.clone()); + match self.bucket.get("manifest", cached_tag.as_deref())? { + Fetched::Unchanged => Ok(()), + Fetched::Absent => { + *self.view.borrow_mut() = None; + Ok(()) + } + Fetched::New { bytes, tag } => { + let bytes = capped("manifest", bytes)?; + let manifest = Manifest::decode(&bytes)?; + let mut entries = Vec::new(); + let mut seq = manifest.head_seq; + while seq != 0 { + let Fetched::New { bytes, .. } = self.bucket.get(&log_key(seq), None)? else { + return Err(Error::Wal(format!( + "log entry {seq} referenced by chain but absent" + ))); + }; + let e = LogEntry::decode(&capped("log entry", bytes)?)?; + if e.seq != seq { + return Err(Error::Wal(format!( + "log entry at {} claims seq {}", + seq, e.seq + ))); + } + if e.parent_seq >= e.seq { + return Err(Error::Wal(format!( + "log entry {} has non-decreasing parent {}", + e.seq, e.parent_seq + ))); + } + seq = e.parent_seq; + entries.push(e); + } + entries.reverse(); // oldest first + let mut refs = BTreeMap::new(); + let mut index = BTreeMap::new(); + for e in &entries { + for u in &e.updates { + crate::refs::validate_branch_name(&u.branch)?; + refs.insert(u.branch.clone(), u.new); + } + for hash in &e.packs { + let Fetched::New { bytes, .. } = self.bucket.get(&idx_key(hash), None)? + else { + return Err(Error::Wal(format!("pack {hash} on chain but idx absent"))); + }; + let bytes = capped("pack idx", bytes)?; + for IndexEntry { id, offset, length } in parse_index(&bytes)? { + index.insert(id, (hash.clone(), offset, length)); + } + } + } + *self.view.borrow_mut() = Some(WalView { + tag, + manifest, + refs, + index, + }); + Ok(()) + } + } + } + + /// Canonical `encode()` bytes of one object, fetched via its pack + /// (downloaded once per distinct pack hash and cached for walk locality). + fn object_bytes(&self, id: &ObjectId) -> Result> { + let (hash, offset, _len) = { + let view = self.view.borrow(); + let view = view + .as_ref() + .ok_or_else(|| Error::Wal("bucket remote is empty".into()))?; + view.index + .get(id) + .cloned() + .ok_or(Error::CorruptObject(*id))? + }; + let mut cache = self.pack_cache.borrow_mut(); + if cache.as_ref().map(|(h, _)| h.as_str()) != Some(hash.as_str()) { + let Fetched::New { bytes, .. } = self.bucket.get(&pack_key(&hash), None)? else { + return Err(Error::Wal(format!("pack {hash} on chain but body absent"))); + }; + *cache = Some((hash.clone(), bytes)); + } + let (_, pack) = cache.as_ref().unwrap(); + Ok(read_object_at_bounded(pack, offset, id, scl_core::MAX_OBJECT_SIZE)?.encode()) + } + + /// Upload one pack (+ its idx) content-addressed by BLAKE3 of the pack + /// bytes, so two callers who happen to pack the identical object set + /// converge on the same key. `put_new`'s "key already exists" outcome is + /// success here, not a conflict — identical content, nothing to redo. + fn upload_pack(&self, objects: &[(ObjectId, Vec)]) -> Result<(String, Vec)> { + let (pack, idx) = scl_core::pack::build_pack(objects)?; + let hash = hex::encode(blake3::hash(&pack).as_bytes()); + self.bucket.put_new(&pack_key(&hash), &pack)?; + self.bucket.put_new(&idx_key(&hash), &idx)?; + Ok((hash, objects.iter().map(|(id, _)| *id).collect())) + } + + /// Pack and upload everything `put_object` has staged since the last + /// flush, recording the resulting pack hash as pending. A no-op when + /// nothing is staged (the common case for a `put_pack`-only caller). + fn flush_staged(&self) -> Result<()> { + let staged = std::mem::take(&mut *self.staged.borrow_mut()); + if staged.is_empty() { + return Ok(()); + } + // On upload failure (a real bucket can fail transiently: network, + // 5xx, auth), put the objects back rather than dropping them — a + // caller that retries `update_ref` must still see them staged, or + // they'd be silently lost (CLAUDE.md: never silently drop data). + match self.upload_pack(&staged) { + Ok((hash, _)) => { + self.pending_packs.borrow_mut().push(hash); + Ok(()) + } + Err(e) => { + *self.staged.borrow_mut() = staged; + Err(e) + } + } + } +} + +/// `ObjectSource` over the bucket for reachability walks (`get_pack`'s +/// want/have closures). +struct BucketSource<'a>(&'a BucketTransport); +impl crate::reachable::ObjectSource for BucketSource<'_> { + fn get(&mut self, id: &ObjectId) -> Result { + let bytes = self.0.object_bytes(id)?; + Ok(Object::decode(&bytes)?) + } +} + +impl Transport for BucketTransport { + fn list_refs(&self) -> Result> { + self.refresh()?; + Ok(self + .view + .borrow() + .as_ref() + .map(|v| v.refs.iter().map(|(b, id)| (b.clone(), *id)).collect()) + .unwrap_or_default()) + } + + fn head_branch(&self) -> Result { + self.refresh()?; + self.view + .borrow() + .as_ref() + .map(|v| v.manifest.head_branch.clone()) + .ok_or_else(|| Error::Remote("bucket remote is empty (no manifest)".into())) + } + + fn has_object(&self, id: &ObjectId) -> Result { + self.refresh()?; + Ok(self + .view + .borrow() + .as_ref() + .is_some_and(|v| v.index.contains_key(id))) + } + + fn get_object(&self, id: &ObjectId) -> Result> { + self.refresh()?; + self.object_bytes(id) + } + + fn get_pack( + &self, + wants: &[ObjectId], + haves: &[ObjectId], + filter: Option<&[String]>, + out: &mut dyn Write, + ) -> Result<()> { + if filter.is_some() { + return Err(Error::InvalidArgument( + "partial clone from bucket remotes is not supported yet; clone via a served remote" + .into(), + )); + } + self.refresh()?; + let mut src = BucketSource(self); + // haves the bucket doesn't know can't shrink the pack — skip them. + let known_haves: Vec = { + let view = self.view.borrow(); + haves + .iter() + .copied() + .filter(|h| view.as_ref().is_some_and(|v| v.index.contains_key(h))) + .collect() + }; + let have_set = crate::reachable::reachable_objects(&mut src, &known_haves)?; + let want_set = crate::reachable::reachable_objects(&mut src, wants)?; + let ids: Vec = want_set.difference(&have_set).copied().collect(); + let mut writer = PackWriter::new(out, ids.len() as u32)?; + for id in &ids { + let bytes = self.object_bytes(id)?; + writer.write_object(id, &bytes)?; + } + writer.finish()?; // idx discarded — transfer needs the body only + Ok(()) + } + + fn put_object(&self, id: &ObjectId, bytes: &[u8]) -> Result<()> { + if ObjectId::of(bytes) != *id { + return Err(Error::CorruptObject(*id)); + } + self.staged.borrow_mut().push((*id, bytes.to_vec())); + Ok(()) + } + + fn put_pack(&self, src: &mut dyn std::io::Read) -> Result> { + // Never trust a live incoming stream (P25/ADR-0039 pattern): verify + // every record's hash as it streams in via the bounded reader parser, + // then rebuild our own pack from the verified objects rather than + // uploading the caller's bytes verbatim. `build_pack`'s output is + // pinned byte-stable for a given object set (core), so the rebuilt + // pack still content-addresses identically for identical input. + let mut objects: Vec<(ObjectId, Vec)> = Vec::new(); + scl_core::pack::parse_pack_reader(src, |id, obj| { + objects.push((id, obj.encode())); + Ok(()) + })?; + let (hash, ids) = self.upload_pack(&objects)?; + self.pending_packs.borrow_mut().push(hash); + Ok(ids) + } + + fn update_ref( + &self, + branch: &str, + id: &ObjectId, + expected_old: Option<&ObjectId>, + ) -> Result<()> { + crate::refs::validate_branch_name(branch)?; + // Everything staged since the last flush becomes one more pending + // pack before we even look at the manifest, so a retry below never + // has to re-stage or re-upload it. + self.flush_staged()?; + // From here, the packs this call may commit are call-local: take them + // out of the shared field entirely rather than clearing it only on + // the success paths. A long-lived transport (e.g. `wire::serve` keeps + // one instance per session) can see this call fail — NonFastForward, + // or CAS exhaustion — and then be reused for an unrelated ref update; + // if `pending_packs` were only cleared on success, that later, + // unrelated call would commit a log entry citing packs this call + // never landed, violating the invariant that a log entry's `packs` + // list is exactly the packs its own ref updates need. Every exit from + // this function — success, failure, or the loop below — simply lets + // `pending` go out of scope; the packs themselves stay uploaded in + // the bucket regardless (content-addressed, inert orphan garbage if + // never referenced — correct per spec, compaction is a later phase). + let pending: Vec = std::mem::take(&mut *self.pending_packs.borrow_mut()); + const MAX_CAS_RETRIES: u32 = 16; + for _ in 0..MAX_CAS_RETRIES { + self.refresh()?; + let (current, prev_tag, head_seq, checkpoint_seq, head_branch) = { + let view = self.view.borrow(); + match view.as_ref() { + Some(v) => ( + v.refs.get(branch).copied(), + Some(v.tag.clone()), + v.manifest.head_seq, + v.manifest.checkpoint_seq, + v.manifest.head_branch.clone(), + ), + None => (None, None, 0, 0, branch.to_string()), + } + }; + // Trait doc: setting the ref to the value it already has + // succeeds regardless of `expected_old` — check this before the + // fast-forward comparison below. + if current.as_ref() == Some(id) { + return Ok(()); + } + if current.as_ref() != expected_old { + return Err(Error::NonFastForward); + } + // Claim a log slot for this attempt. `put_new` on `log/` is + // the claim: if another writer already landed that seq, we didn't + // win it and try the next one — the entry we didn't win becomes + // permanent off-chain garbage (never referenced by any manifest, + // never read by `refresh`), which is fine, it costs one object. + let entry = LogEntry { + seq: 0, // overwritten per candidate in the claim loop below + parent_seq: head_seq, + packs: pending.clone(), + updates: vec![RefUpdate { + branch: branch.to_string(), + old: current, + new: *id, + }], + }; + let mut try_seq = head_seq + 1; + let seq = loop { + let mut candidate = entry.clone(); + candidate.seq = try_seq; + if self + .bucket + .put_new(&log_key(try_seq), &candidate.encode())? + { + break try_seq; + } + try_seq += 1; + }; + let manifest = Manifest { + head_seq: seq, + checkpoint_seq, + head_branch, + }; + if self + .bucket + .put_if_tag("manifest", &manifest.encode(), prev_tag.as_deref())? + .is_some() + { + // Committed. The log entry we just claimed is now on-chain; + // pull the fresh view (cheap: one conditional GET, since our + // own write just changed the tag). + self.refresh()?; + return Ok(()); + } + // Lost the manifest CAS: someone else's append won the race. Our + // just-claimed log entry is now off-chain garbage too (it chains + // from a parent that's no longer the head). Loop back to + // `refresh()`: if this branch's tip moved off `expected_old` we + // hit `NonFastForward` above; otherwise (a different branch or an + // unrelated pack landed) we retry with a fresh entry claimed + // under the new parent — `pending` (this call's local copy) + // still lists our packs, so nothing is re-uploaded. + } + Err(Error::Remote( + "manifest cas contention: gave up after 16 attempts".into(), + )) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::transport::Transport; + use crate::walfmt::{idx_key, log_key, pack_key, LogEntry, Manifest, RefUpdate}; + use scl_core::ObjectId; + use scl_objio::{Bucket, DirBucket}; + + /// Build pack+idx bytes for the given objects (reuses core's builder). + fn pack_of(objects: &[(ObjectId, Vec)]) -> (String, Vec, Vec) { + let (pack, idx) = scl_core::pack::build_pack(objects).unwrap(); + let hash = hex::encode(blake3::hash(&pack).as_bytes()); + (hash, pack, idx) + } + + /// A minimal one-commit object set: blob → tree → snapshot, exactly as a + /// real repo would store them. Reuse the object constructors the repo + /// crate already uses in its own tests (see sync.rs tests for the + /// canonical way to mint a commit); returns (tip, objects). `tag` + /// disambiguates the scratch repo path between the two call sites below + /// — both share this process id, and `cargo test` runs tests on parallel + /// threads within one process by default, so a bare pid-keyed path would + /// let two callers race on the same directory (house pattern: see + /// `transport.rs`'s `tmp_remote(tag)`). + fn tiny_history(tag: &str) -> (ObjectId, Vec<(ObjectId, Vec)>) { + let root = std::env::temp_dir().join(format!("scl-bt-hist-{tag}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + std::fs::create_dir_all(&root).unwrap(); + let repo = crate::repo::Repo::init(&root).unwrap(); + std::fs::write(root.join("f.txt"), b"hello wal").unwrap(); + let tip = repo.commit("t", "c1").unwrap(); + let store_arc = repo.vfs().store(); + let mut store = store_arc.lock().unwrap(); + let ids = crate::reachable::reachable_objects(&mut *store, &[tip]).unwrap(); + let objects = ids + .iter() + .map(|id| (*id, store.get(id).unwrap().encode())) + .collect(); + drop(store); + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + (tip, objects) + } + + /// Like `tiny_history`, but with caller-chosen file content. `tiny_history` + /// hardcodes content/author/message, so two calls close enough in time to + /// land the same commit-timestamp second produce byte-identical Snapshot + /// objects (same root, same empty parents) — fine for the read-half tests + /// above, which only need *a* valid history, but wrong for a test that + /// needs two genuinely distinct object sets (and thus distinct pack + /// hashes) to tell "referenced" apart from "coincidentally identical". + fn tiny_history_distinct(tag: &str, content: &[u8]) -> (ObjectId, Vec<(ObjectId, Vec)>) { + let root = std::env::temp_dir().join(format!("scl-bt-hist-{tag}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + std::fs::create_dir_all(&root).unwrap(); + let repo = crate::repo::Repo::init(&root).unwrap(); + std::fs::write(root.join("f.txt"), content).unwrap(); + let tip = repo.commit("t", "c1").unwrap(); + let store_arc = repo.vfs().store(); + let mut store = store_arc.lock().unwrap(); + let ids = crate::reachable::reachable_objects(&mut *store, &[tip]).unwrap(); + let objects = ids + .iter() + .map(|id| (*id, store.get(id).unwrap().encode())) + .collect(); + drop(store); + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + (tip, objects) + } + + #[test] + fn reads_refs_objects_and_packs_from_a_hand_built_wal() { + let broot = std::env::temp_dir().join(format!("scl-bt-read-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&broot); + let bucket = DirBucket::open(&broot).unwrap(); + + let (tip, objects) = tiny_history("read"); + let (hash, pack, idx) = pack_of(&objects); + assert!(bucket.put_new(&pack_key(&hash), &pack).unwrap()); + assert!(bucket.put_new(&idx_key(&hash), &idx).unwrap()); + let entry = LogEntry { + seq: 1, + parent_seq: 0, + packs: vec![hash.clone()], + updates: vec![RefUpdate { + branch: "main".into(), + old: None, + new: tip, + }], + }; + assert!(bucket.put_new(&log_key(1), &entry.encode()).unwrap()); + let m = Manifest { + head_seq: 1, + checkpoint_seq: 0, + head_branch: "main".into(), + }; + bucket + .put_if_tag("manifest", &m.encode(), None) + .unwrap() + .unwrap(); + + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + assert_eq!(t.list_refs().unwrap(), vec![("main".to_string(), tip)]); + assert_eq!(t.head_branch().unwrap(), "main"); + assert!(t.has_object(&tip).unwrap()); + let bytes = t.get_object(&tip).unwrap(); + assert_eq!(ObjectId::of(&bytes), tip); + // get_pack: full closure with no haves reproduces every object + let mut out = Vec::new(); + t.get_pack(&[tip], &[], None, &mut out).unwrap(); + let got = scl_core::pack::parse_pack(&out).unwrap(); + assert_eq!(got.len(), objects.len()); + // filter is refused loudly, not ignored + let filt = vec!["src/".to_string()]; + assert!(t + .get_pack(&[tip], &[], Some(&filt), &mut Vec::new()) + .is_err()); + drop(t); + std::fs::remove_dir_all(&broot).unwrap(); + assert!(!broot.exists()); + } + + #[test] + fn empty_bucket_lists_no_refs_and_head_branch_errors() { + let broot = std::env::temp_dir().join(format!("scl-bt-empty-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&broot); + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + assert_eq!(t.list_refs().unwrap(), Vec::<(String, ObjectId)>::new()); + assert!(t.head_branch().is_err()); + drop(t); + std::fs::remove_dir_all(&broot).unwrap(); + assert!(!broot.exists()); + } + + #[test] + fn off_chain_log_entries_are_ignored() { + // manifest head=1; a stray log/2 (orphan from a crashed/losing pusher) + // must not affect refs. + let broot = std::env::temp_dir().join(format!("scl-bt-orphan-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&broot); + let bucket = DirBucket::open(&broot).unwrap(); + let (tip, objects) = tiny_history("orphan"); + let (hash, pack, idx) = pack_of(&objects); + bucket.put_new(&pack_key(&hash), &pack).unwrap(); + bucket.put_new(&idx_key(&hash), &idx).unwrap(); + let e1 = LogEntry { + seq: 1, + parent_seq: 0, + packs: vec![hash], + updates: vec![RefUpdate { + branch: "main".into(), + old: None, + new: tip, + }], + }; + bucket.put_new(&log_key(1), &e1.encode()).unwrap(); + let orphan = LogEntry { + seq: 2, + parent_seq: 1, + packs: vec![], + updates: vec![RefUpdate { + branch: "evil".into(), + old: None, + new: tip, + }], + }; + bucket.put_new(&log_key(2), &orphan.encode()).unwrap(); + let m = Manifest { + head_seq: 1, + checkpoint_seq: 0, + head_branch: "main".into(), + }; + bucket + .put_if_tag("manifest", &m.encode(), None) + .unwrap() + .unwrap(); + + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + assert_eq!(t.list_refs().unwrap(), vec![("main".to_string(), tip)]); + drop(t); + std::fs::remove_dir_all(&broot).unwrap(); + assert!(!broot.exists()); + } + + #[test] + fn push_via_trait_round_trips_into_a_fresh_bucket() { + let broot = std::env::temp_dir().join(format!("scl-bt-write-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&broot); + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + + let (tip, objects) = tiny_history("write"); + // exactly what sync::push does: pack, then CAS'd ref update + let (pack, _idx) = scl_core::pack::build_pack(&objects).unwrap(); + let ids = t.put_pack(&mut std::io::Cursor::new(pack)).unwrap(); + assert_eq!(ids.len(), objects.len()); + t.update_ref("main", &tip, None).unwrap(); + + // a second transport sees it + let t2 = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + assert_eq!(t2.list_refs().unwrap(), vec![("main".to_string(), tip)]); + assert_eq!(t2.head_branch().unwrap(), "main"); + assert!(t2.has_object(&tip).unwrap()); + drop((t, t2)); + std::fs::remove_dir_all(&broot).unwrap(); + assert!(!broot.exists()); + } + + #[test] + fn update_ref_honors_expected_old_semantics() { + let broot = std::env::temp_dir().join(format!("scl-bt-cas-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&broot); + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + let (tip, objects) = tiny_history("cas"); + let (pack, _) = scl_core::pack::build_pack(&objects).unwrap(); + t.put_pack(&mut std::io::Cursor::new(pack)).unwrap(); + t.update_ref("main", &tip, None).unwrap(); + // stale expected_old (None while the branch exists) => NonFastForward + let other = ObjectId::of(b"not the tip"); + assert!(matches!( + t.update_ref("main", &other, None), + Err(Error::NonFastForward) + )); + // setting to the value it already has succeeds regardless of expected_old (trait doc) + t.update_ref("main", &tip, None).unwrap(); + t.update_ref("main", &tip, Some(&other)).unwrap(); + drop(t); + std::fs::remove_dir_all(&broot).unwrap(); + assert!(!broot.exists()); + } + + #[test] + fn put_object_stages_and_update_ref_commits_them() { + let broot = std::env::temp_dir().join(format!("scl-bt-stage-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&broot); + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + let (tip, objects) = tiny_history("stage"); + for (id, bytes) in &objects { + t.put_object(id, bytes).unwrap(); + } + // corrupt bytes are rejected at staging time + assert!(t.put_object(&tip, b"garbage").is_err()); + t.update_ref("main", &tip, None).unwrap(); + let t2 = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + assert!(t2.has_object(&tip).unwrap()); + drop((t, t2)); + std::fs::remove_dir_all(&broot).unwrap(); + assert!(!broot.exists()); + } + + #[test] + fn abandoned_pending_packs_do_not_leak_into_a_later_unrelated_update_ref() { + // Regression for a review finding: `pending_packs` must not survive + // a failed `update_ref` call into a later, unrelated `update_ref` on + // the same (long-lived, e.g. `wire::serve`-hosted) transport + // instance — the WAL's per-entry invariant is that a log entry's + // `packs` list is exactly the packs its own ref updates need. + let broot = std::env::temp_dir().join(format!("scl-bt-noleak-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&broot); + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + + // Land "main" -> tip1 normally, so a later push to "main" can fail + // fast-forward. + let (tip1, objects1) = tiny_history_distinct("noleak-main", b"main content"); + let (pack1, _) = scl_core::pack::build_pack(&objects1).unwrap(); + t.put_pack(&mut std::io::Cursor::new(pack1)).unwrap(); + t.update_ref("main", &tip1, None).unwrap(); + + // Stage a second, unrelated pack, then attempt an update to "main" + // that must fail NonFastForward *with pending packs non-empty* — the + // exact condition the existing tests never exercised (they only hit + // NonFastForward with nothing pending). + let (_tip2, objects2) = tiny_history_distinct("noleak-abandoned", b"abandoned content"); + let (pack2, _) = scl_core::pack::build_pack(&objects2).unwrap(); + let abandoned_hash = hex::encode(blake3::hash(&pack2).as_bytes()); + t.put_pack(&mut std::io::Cursor::new(pack2)).unwrap(); + let bogus = ObjectId::of(b"not the real tip"); + assert!(matches!( + t.update_ref("main", &bogus, None), + Err(Error::NonFastForward) + )); + + // A different, unrelated branch, on the SAME instance, with its own + // freshly-staged pack — this call must succeed and must commit only + // its own pack, never the one abandoned by the failed call above. + let (tip3, objects3) = tiny_history_distinct("noleak-other", b"other content"); + let (pack3, _) = scl_core::pack::build_pack(&objects3).unwrap(); + let other_hash = hex::encode(blake3::hash(&pack3).as_bytes()); + t.put_pack(&mut std::io::Cursor::new(pack3)).unwrap(); + t.update_ref("other", &tip3, None).unwrap(); + + // Decode the committed head log entry straight from the bucket and + // check its `packs` list directly, independent of the transport's + // own (already-passing) read half. + let inspect = DirBucket::open(&broot).unwrap(); + let Fetched::New { bytes, .. } = inspect.get("manifest", None).unwrap() else { + panic!("manifest must exist after two successful update_ref calls") + }; + let manifest = Manifest::decode(&bytes).unwrap(); + let Fetched::New { bytes, .. } = inspect.get(&log_key(manifest.head_seq), None).unwrap() + else { + panic!("head log entry must exist") + }; + let head_entry = LogEntry::decode(&bytes).unwrap(); + assert!( + !head_entry.packs.contains(&abandoned_hash), + "committed entry must not reference the pack abandoned by the failed call" + ); + assert_eq!( + head_entry.packs, + vec![other_hash], + "committed entry must reference exactly this call's own pack" + ); + + drop(t); + std::fs::remove_dir_all(&broot).unwrap(); + assert!(!broot.exists()); + } + + #[test] + fn clone_push_fetch_round_trip_over_sc_wal_url() { + let pid = std::process::id(); + let broot = std::env::temp_dir().join(format!("scl-bt-e2e-bucket-{pid}")); + let a_root = std::env::temp_dir().join(format!("scl-bt-e2e-a-{pid}")); + let b_root = std::env::temp_dir().join(format!("scl-bt-e2e-b-{pid}")); + for d in [&broot, &a_root, &b_root] { + let _ = std::fs::remove_dir_all(d); + } + std::fs::create_dir_all(&a_root).unwrap(); + let url = format!("sc+wal://{}", broot.display()); + + // A: init, commit, add bucket remote, push (creates the bucket repo) + let a = crate::repo::Repo::init(&a_root).unwrap(); + std::fs::write(a_root.join("f.txt"), b"one").unwrap(); + let tip1 = a.commit("t", "c1").unwrap(); + a.remote_add("origin", &url).unwrap(); + assert_eq!(a.push("origin").unwrap(), tip1); + + // B: clone from the bucket + let b = crate::repo::Repo::clone_url(&url, &b_root).unwrap(); + assert_eq!(b.head_tip().unwrap(), Some(tip1)); + assert_eq!(std::fs::read(b_root.join("f.txt")).unwrap(), b"one"); + + // B commits and pushes; A fetches and sees it + std::fs::write(b_root.join("g.txt"), b"two").unwrap(); + let tip2 = b.commit("t", "c2").unwrap(); + b.push("origin").unwrap(); + drop(b); + let fetched = a.fetch("origin").unwrap(); + assert!(fetched.iter().any(|(br, id)| br == "main" && *id == tip2)); + + // stale push from A (still at tip1 + its own commit) => NonFastForward + std::fs::write(a_root.join("h.txt"), b"three").unwrap(); + a.commit("t", "c3").unwrap(); + assert!(matches!(a.push("origin"), Err(Error::NonFastForward))); + drop(a); + for d in [&broot, &a_root, &b_root] { + std::fs::remove_dir_all(d).unwrap(); + } + } + + #[test] + fn bucket_url_parses_and_rejects() { + let u = BucketUrl::parse("sc+s3://mybucket/team/repo").unwrap(); + assert!(matches!(u.scheme, BucketScheme::S3)); + assert_eq!(u.bucket, "mybucket"); + assert_eq!(u.prefix, "team/repo"); + let u = BucketUrl::parse("sc+s3://mybucket").unwrap(); + assert_eq!(u.prefix, ""); + let u = BucketUrl::parse("sc+wal:///tmp/x").unwrap(); + assert!(matches!(u.scheme, BucketScheme::Wal)); + assert!(BucketUrl::parse("sc+s3://").is_err()); + assert!(BucketUrl::parse("sc+s3://b\nad/x").is_err()); + assert!(BucketUrl::parse("http://nope").is_err()); + } + + /// P36a acceptance proof #1: two threads race `update_ref("main", ..)` + /// from the same `expected_old`, targeting different (fake) tips. The + /// bucket's `put_if_tag` CAS is the only serialization point — exactly + /// one racer must win and the other must see a clean `NonFastForward`, + /// never a deadlock, a double-commit, or a corrupt manifest. + #[test] + fn racing_pushes_same_branch_one_wins_one_gets_non_fast_forward() { + let pid = std::process::id(); + let broot = std::env::temp_dir().join(format!("scl-bt-race-{pid}")); + let _ = std::fs::remove_dir_all(&broot); + // seed: one commit on main + let t0 = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + let (base, objects) = tiny_history("race-seed"); + let (pack, _) = scl_core::pack::build_pack(&objects).unwrap(); + t0.put_pack(&mut std::io::Cursor::new(pack)).unwrap(); + t0.update_ref("main", &base, None).unwrap(); + drop(t0); + + // two threads race an update from the same expected_old to different tips. + // A barrier forces both threads into `update_ref` at the same instant — + // without it, the OS could simply run racer-a to completion before + // racer-b's thread is even scheduled, and the (1, 1) result below + // would hold trivially with the manifest CAS never actually contended. + let mk_tip = |tag: &[u8]| { + let obj = Object::blob(tag.to_vec()); // any distinct object works as a fake tip + (obj.id(), obj.encode()) + }; + let gate = std::sync::Barrier::new(2); + let results: Vec> = std::thread::scope(|s| { + let handles: Vec<_> = [b"racer-a".as_slice(), b"racer-b".as_slice()] + .into_iter() + .map(|tag| { + let broot = broot.clone(); + let gate = &gate; + s.spawn(move || { + let t = BucketTransport::from_bucket(Box::new( + DirBucket::open(&broot).unwrap(), + )) + .unwrap(); + let (tip, bytes) = mk_tip(tag); + t.put_object(&tip, &bytes).unwrap(); + gate.wait(); + t.update_ref("main", &tip, Some(&base)) + }) + }) + .collect(); + handles.into_iter().map(|h| h.join().unwrap()).collect() + }); + let wins = results.iter().filter(|r| r.is_ok()).count(); + let nffs = results + .iter() + .filter(|r| matches!(r, Err(Error::NonFastForward))) + .count(); + assert_eq!( + (wins, nffs), + (1, 1), + "exactly one winner and one clean refusal: {results:?}" + ); + std::fs::remove_dir_all(&broot).unwrap(); + assert!(!broot.exists()); + } + + /// P36a acceptance proof #2: 8 threads push to 8 *distinct* branches + /// concurrently with no external coordinator. Contention is at the + /// manifest-CAS level only (every push races the same `manifest` key even + /// though the branches don't conflict), so every push must eventually + /// land via the retry loop. + #[test] + fn fleet_hammer_distinct_branches_all_land_without_coordinator() { + let pid = std::process::id(); + let broot = std::env::temp_dir().join(format!("scl-bt-fleet-{pid}")); + let _ = std::fs::remove_dir_all(&broot); + const N: usize = 8; + // A barrier forces all N threads to call `update_ref` at essentially + // the same instant — without it, short-lived threads could finish + // one at a time with the OS never actually overlapping them, and + // "all N land" would hold even against a broken retry loop that was + // never exercised under real contention. + let gate = std::sync::Barrier::new(N); + std::thread::scope(|s| { + for i in 0..N { + let broot = broot.clone(); + let gate = &gate; + s.spawn(move || { + let t = + BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())) + .unwrap(); + let obj = Object::blob(format!("agent-{i}").into_bytes()); + t.put_object(&obj.id(), &obj.encode()).unwrap(); + gate.wait(); + // distinct branches: contention is manifest-level only, so + // every one must eventually land via CAS retry. + t.update_ref(&format!("work-{i}"), &obj.id(), None).unwrap(); + }); + } + }); + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + let refs = t.list_refs().unwrap(); + assert_eq!(refs.len(), N, "all {N} agent branches present: {refs:?}"); + drop(t); + std::fs::remove_dir_all(&broot).unwrap(); + assert!(!broot.exists()); + } + + /// P36a acceptance proof #3: a pusher that died after writing its pack + + /// log entry, but before the manifest CAS, leaves debris that must be + /// invisible to every reader (the manifest never points at it) — and a + /// later, live push must step over the claimed seq rather than colliding + /// with it. + #[test] + fn crash_debris_before_the_cas_is_invisible_and_later_pushes_step_over_it() { + let pid = std::process::id(); + let broot = std::env::temp_dir().join(format!("scl-bt-crash-{pid}")); + let _ = std::fs::remove_dir_all(&broot); + let bucket = DirBucket::open(&broot).unwrap(); + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + let (tip, objects) = tiny_history("crash-seed"); + let (pack, _) = scl_core::pack::build_pack(&objects).unwrap(); + t.put_pack(&mut std::io::Cursor::new(pack)).unwrap(); + t.update_ref("main", &tip, None).unwrap(); + + // simulate a pusher that died after pack + log entry, before the CAS: + let orphan_obj = Object::blob(b"never committed".to_vec()); + let (opack, oidx) = + scl_core::pack::build_pack(&[(orphan_obj.id(), orphan_obj.encode())]).unwrap(); + let ohash = hex::encode(blake3::hash(&opack).as_bytes()); + bucket.put_new(&pack_key(&ohash), &opack).unwrap(); + bucket.put_new(&idx_key(&ohash), &oidx).unwrap(); + bucket + .put_new( + &log_key(2), + &LogEntry { + seq: 2, + parent_seq: 1, + packs: vec![ohash], + updates: vec![RefUpdate { + branch: "doomed".into(), + old: None, + new: orphan_obj.id(), + }], + } + .encode(), + ) + .unwrap(); + + // invisible to readers… + let t2 = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + assert_eq!(t2.list_refs().unwrap(), vec![("main".to_string(), tip)]); + assert!(!t2.has_object(&orphan_obj.id()).unwrap()); + // …and a live push steps over the claimed seq 2 (lands at 3+) and works. + let next = Object::blob(b"after crash".to_vec()); + t2.put_object(&next.id(), &next.encode()).unwrap(); + t2.update_ref("recovered", &next.id(), None).unwrap(); + let t3 = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + let refs = t3.list_refs().unwrap(); + assert!(refs.contains(&("recovered".to_string(), next.id()))); + assert!(!refs.iter().any(|(b, _)| b == "doomed")); + + // Prove the "steps over" part directly, not just its visible effect: + // the live push's claim loop must have found `log/2` already taken + // by the orphan (a real collision) and landed at seq 3+ instead of + // silently overwriting it — inspect the bucket straight, independent + // of the transport's own (already-passing) read half. + let inspect = DirBucket::open(&broot).unwrap(); + let Fetched::New { bytes, .. } = inspect.get("manifest", None).unwrap() else { + panic!("manifest must exist after the live push") + }; + let manifest = Manifest::decode(&bytes).unwrap(); + assert!( + manifest.head_seq >= 3, + "live push must claim a seq past the crashed pusher's seq 2, got {}", + manifest.head_seq + ); + let Fetched::New { bytes, .. } = inspect.get(&log_key(2), None).unwrap() else { + panic!( + "the orphan's log/2 entry must still be present, untouched, as off-chain garbage" + ) + }; + let untouched = LogEntry::decode(&bytes).unwrap(); + assert_eq!( + untouched.updates, + vec![RefUpdate { + branch: "doomed".into(), + old: None, + new: orphan_obj.id(), + }], + "the live push must never overwrite the claimed-but-uncommitted log/2 slot" + ); + + drop((t, t2, t3)); + std::fs::remove_dir_all(&broot).unwrap(); + assert!(!broot.exists()); + } + + fn walkdir_files(dir: &std::path::Path) -> Vec { + let mut out = Vec::new(); + if let Ok(entries) = std::fs::read_dir(dir) { + for entry in entries.flatten() { + let path = entry.path(); + if path.is_dir() { + out.extend(walkdir_files(&path)); + } else { + out.push(path); + } + } + } + out + } + + /// True if `marker` appears as a contiguous byte run anywhere under `dir`. + fn any_bucket_file_contains(dir: &std::path::Path, marker: &[u8]) -> bool { + walkdir_files(dir).into_iter().any(|p| { + std::fs::read(&p) + .map(|bytes| bytes.windows(marker.len()).any(|w| w == marker)) + .unwrap_or(false) + }) + } + + /// Pins the headline security claim bucket-specifically: sealed content + /// pushed to a `sc+wal://` bucket remote must never appear in plaintext + /// among the raw files the bucket backend writes to disk. `protect` + /// convergently encrypts matching working-tree files before `commit` + /// snapshots them, so the pushed pack/log/manifest bytes should carry + /// only ciphertext for `secret/a.txt` — walk every file the DirBucket + /// wrote and assert the plaintext marker appears nowhere. + /// + /// Alongside the protected file, an unprotected `public.txt` carrying a + /// second, distinct marker is committed and pushed too. That marker + /// MUST be found by the exact same walk-and-search: without that + /// positive control, a negative result on the secret marker would be + /// equally consistent with "sealing worked" and with "the search can't + /// see plaintext in bucket files at all" (e.g. because pack bodies are + /// compressed, or the walk misses the relevant files) — either of which + /// would make the assertion vacuous. + #[test] + fn protected_content_ciphertext_never_appears_in_bucket_files() { + let pid = std::process::id(); + let broot = std::env::temp_dir().join(format!("scl-bt-protect-bucket-{pid}")); + let a_root = std::env::temp_dir().join(format!("scl-bt-protect-a-{pid}")); + for d in [&broot, &a_root] { + let _ = std::fs::remove_dir_all(d); + } + std::fs::create_dir_all(&a_root).unwrap(); + let url = format!("sc+wal://{}", broot.display()); + + let a = crate::repo::Repo::init(&a_root).unwrap(); + let (_alice_sk, alice_pk) = scl_crypto::generate_keypair(); + a.protect("secret/", &[alice_pk], None).unwrap(); + std::fs::create_dir_all(a_root.join("secret")).unwrap(); + let secret_marker = b"SC-PLAINTEXT-MARKER-3b8f1c2a9d47"; + let public_marker = b"SC-PUBLIC-CONTROL-MARKER-7e91a0c5"; + std::fs::write(a_root.join("secret/a.txt"), secret_marker).unwrap(); + std::fs::write(a_root.join("public.txt"), public_marker).unwrap(); + a.commit("me", "protect secret/a.txt, add public.txt") + .unwrap(); + a.remote_add("origin", &url).unwrap(); + a.push("origin").unwrap(); + drop(a); + + // Positive control first: if this fails, the walk-and-search can't + // see plaintext in bucket files at all, and the negative assertion + // below would be meaningless. + assert!( + any_bucket_file_contains(&broot, public_marker), + "positive control failed: unprotected public.txt's marker was not \ + found anywhere under the bucket root, so this search method \ + cannot detect plaintext in bucket files — the negative assertion \ + below would be vacuous" + ); + assert!( + !any_bucket_file_contains(&broot, secret_marker), + "plaintext marker for protected secret/a.txt leaked into a bucket file" + ); + + std::fs::remove_dir_all(&broot).unwrap(); + std::fs::remove_dir_all(&a_root).unwrap(); + assert!(!broot.exists() && !a_root.exists()); + } +} diff --git a/crates/repo/src/error.rs b/crates/repo/src/error.rs index 274fca0..d9dbe27 100644 --- a/crates/repo/src/error.rs +++ b/crates/repo/src/error.rs @@ -147,6 +147,13 @@ pub enum Error { /// `--max-connections` was reached (P31). Retryable. #[error("server busy (connection limit reached); retry later")] ServerBusy, + /// P36a: the bucket WAL is untrusted input; decode/consistency failures + /// are their own variant so callers can distinguish "bucket corrupt or + /// newer-format" from transport errors. + #[error("bucket wal: {0}")] + Wal(String), + #[error("bucket: {0}")] + ObjIo(#[from] scl_objio::Error), } pub type Result = std::result::Result; diff --git a/crates/repo/src/lib.rs b/crates/repo/src/lib.rs index 68b54c4..68b5175 100644 --- a/crates/repo/src/lib.rs +++ b/crates/repo/src/lib.rs @@ -1,6 +1,7 @@ //! `scl-repo` — the durable on-disk repository: `.sc/` layout, refs/HEAD, //! named branches, a git-like working tree, and commit/secret orchestration. +pub mod bucket_transport; pub(crate) mod cache; pub(crate) mod conflicts; pub mod diff3; @@ -38,11 +39,13 @@ pub mod textdiff; pub mod tls_pins; pub mod transcripts; pub mod transport; +pub mod walfmt; pub mod wire; pub mod workspace; pub mod worktree; pub(crate) mod ws; +pub use bucket_transport::{BucketScheme, BucketTransport, BucketUrl}; pub use conflicts::{ActiveOp, ConflictKind, ConflictVersions, ResolveSide, Side}; pub use error::{Error, Result}; pub use gc::GcStats; diff --git a/crates/repo/src/stdio_transport.rs b/crates/repo/src/stdio_transport.rs index 1a45788..c5614d0 100644 --- a/crates/repo/src/stdio_transport.rs +++ b/crates/repo/src/stdio_transport.rs @@ -372,6 +372,10 @@ pub fn open_transport(url: &str) -> Result> { Ok(Box::new(crate::http_transport::HttpTransport::connect( &parsed, )?)) + } else if url.starts_with("sc+wal://") || url.starts_with("sc+s3://") { + Ok(Box::new(crate::bucket_transport::BucketTransport::open( + url, + )?)) } else { Ok(Box::new(crate::transport::LocalTransport::open(url)?)) } diff --git a/crates/repo/src/walfmt.rs b/crates/repo/src/walfmt.rs new file mode 100644 index 0000000..a472d4d --- /dev/null +++ b/crates/repo/src/walfmt.rs @@ -0,0 +1,323 @@ +//! On-bucket WAL encoding (P36a). Versioned, strict, fail-closed: readers +//! refuse unknown versions and any length that overruns the buffer. + +use crate::error::{Error, Result}; +use scl_core::ObjectId; + +const MANIFEST_MAGIC: &[u8; 4] = b"SCWM"; +const ENTRY_MAGIC: &[u8; 4] = b"SCWE"; +const VERSION: u32 = 1; +const MAX_NAME: usize = 4096; +const MAX_LIST: usize = 65536; +const MAX_HASH: usize = 128; + +/// A bounds-checked cursor over a decode buffer. Every read either advances +/// `at` by exactly what it consumed or returns an error — callers never see +/// a partially-advanced cursor after a failure. +struct Cursor<'a> { + buf: &'a [u8], + at: usize, +} + +impl<'a> Cursor<'a> { + fn take(&mut self, n: usize) -> Result<&'a [u8]> { + if self.buf.len() - self.at < n { + return Err(Error::Wal(format!("truncated at byte {}", self.at))); + } + let s = &self.buf[self.at..self.at + n]; + self.at += n; + Ok(s) + } + + fn u8(&mut self) -> Result { + Ok(self.take(1)?[0]) + } + + fn u32(&mut self) -> Result { + Ok(u32::from_le_bytes(self.take(4)?.try_into().unwrap())) + } + + fn u64(&mut self) -> Result { + Ok(u64::from_le_bytes(self.take(8)?.try_into().unwrap())) + } + + fn string(&mut self, cap: usize) -> Result { + let n = self.u32()? as usize; + if n > cap { + return Err(Error::Wal(format!("length {n} exceeds cap {cap}"))); + } + String::from_utf8(self.take(n)?.to_vec()).map_err(|_| Error::Wal("non-utf8 name".into())) + } + + fn id(&mut self) -> Result { + let raw: [u8; 32] = self.take(32)?.try_into().unwrap(); + Ok(ObjectId::from_bytes(raw)) + } + + fn done(&self) -> Result<()> { + if self.at != self.buf.len() { + return Err(Error::Wal(format!( + "{} trailing bytes", + self.buf.len() - self.at + ))); + } + Ok(()) + } +} + +/// Parse and validate the 8-byte magic+version header, returning a cursor +/// positioned right after it. Unknown versions are refused (fail closed). +fn header<'a>(bytes: &'a [u8], magic: &[u8; 4], what: &str) -> Result> { + let mut c = Cursor { buf: bytes, at: 0 }; + if c.take(4)? != magic { + return Err(Error::Wal(format!("not a {what} (bad magic)"))); + } + let v = c.u32()?; + if v != VERSION { + return Err(Error::Wal(format!( + "{what} version {v} not supported (this build speaks {VERSION})" + ))); + } + Ok(c) +} + +fn push_string(out: &mut Vec, s: &str) { + out.extend_from_slice(&(s.len() as u32).to_le_bytes()); + out.extend_from_slice(s.as_bytes()); +} + +/// The bucket-remote WAL's root pointer: current head sequence, the last +/// checkpoint sequence (a future compaction cutoff), and which branch the +/// head points at. One manifest object per remote, overwritten on every +/// successful append. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Manifest { + pub head_seq: u64, + pub checkpoint_seq: u64, + pub head_branch: String, +} + +impl Manifest { + /// Serialize to the on-bucket wire format: magic, version, then fields + /// in declaration order, all little-endian. + pub fn encode(&self) -> Vec { + let mut out = Vec::new(); + out.extend_from_slice(MANIFEST_MAGIC); + out.extend_from_slice(&VERSION.to_le_bytes()); + out.extend_from_slice(&self.head_seq.to_le_bytes()); + out.extend_from_slice(&self.checkpoint_seq.to_le_bytes()); + push_string(&mut out, &self.head_branch); + out + } + + /// Strictly decode a manifest: bad magic, unknown version, any length + /// that overruns the buffer, or trailing bytes are all refused. + pub fn decode(bytes: &[u8]) -> Result { + let mut c = header(bytes, MANIFEST_MAGIC, "manifest")?; + let head_seq = c.u64()?; + let checkpoint_seq = c.u64()?; + let head_branch = c.string(MAX_NAME)?; + c.done()?; + Ok(Manifest { + head_seq, + checkpoint_seq, + head_branch, + }) + } +} + +/// One branch-ref move recorded in a [`LogEntry`]. `old` is `None` for a +/// branch that was unborn before this entry. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct RefUpdate { + pub branch: String, + pub old: Option, + pub new: ObjectId, +} + +/// One append-only WAL record: the pack(s) it introduces and the ref moves +/// that became visible once those packs landed. `parent_seq` chains entries +/// so a reader can detect a gap (a missing predecessor) without listing the +/// whole bucket. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct LogEntry { + pub seq: u64, + pub parent_seq: u64, + pub packs: Vec, + pub updates: Vec, +} + +impl LogEntry { + /// Serialize to the on-bucket wire format: magic, version, seq fields, + /// then length-prefixed pack names and ref updates, all little-endian. + pub fn encode(&self) -> Vec { + let mut out = Vec::new(); + out.extend_from_slice(ENTRY_MAGIC); + out.extend_from_slice(&VERSION.to_le_bytes()); + out.extend_from_slice(&self.seq.to_le_bytes()); + out.extend_from_slice(&self.parent_seq.to_le_bytes()); + out.extend_from_slice(&(self.packs.len() as u32).to_le_bytes()); + for pack in &self.packs { + push_string(&mut out, pack); + } + out.extend_from_slice(&(self.updates.len() as u32).to_le_bytes()); + for update in &self.updates { + push_string(&mut out, &update.branch); + match &update.old { + Some(id) => { + out.push(1); + out.extend_from_slice(id.as_bytes()); + } + None => out.push(0), + } + out.extend_from_slice(update.new.as_bytes()); + } + out + } + + /// Strictly decode a log entry: every length is bounds-checked against + /// the cap and the remaining buffer; unknown version or trailing bytes + /// are refused. + pub fn decode(bytes: &[u8]) -> Result { + let mut c = header(bytes, ENTRY_MAGIC, "log entry")?; + let seq = c.u64()?; + let parent_seq = c.u64()?; + + let npacks = c.u32()? as usize; + if npacks > MAX_LIST { + return Err(Error::Wal(format!("{npacks} packs exceeds cap {MAX_LIST}"))); + } + let mut packs = Vec::with_capacity(npacks); + for _ in 0..npacks { + packs.push(c.string(MAX_HASH)?); + } + + let nupdates = c.u32()? as usize; + if nupdates > MAX_LIST { + return Err(Error::Wal(format!( + "{nupdates} updates exceeds cap {MAX_LIST}" + ))); + } + let mut updates = Vec::with_capacity(nupdates); + for _ in 0..nupdates { + let branch = c.string(MAX_NAME)?; + let has_old = c.u8()?; + let old = match has_old { + 0 => None, + 1 => Some(c.id()?), + other => return Err(Error::Wal(format!("bad has_old flag: {other}"))), + }; + let new = c.id()?; + updates.push(RefUpdate { branch, old, new }); + } + + c.done()?; + Ok(LogEntry { + seq, + parent_seq, + packs, + updates, + }) + } +} + +/// `log/` zero-padded so lexical order == numeric order. +pub fn log_key(seq: u64) -> String { + format!("log/{seq:020}") +} + +/// `packs/.pack` — the packfile object for a given content hash. +pub fn pack_key(hash: &str) -> String { + format!("packs/{hash}.pack") +} + +/// `packs/.idx` — the index object accompanying [`pack_key`]'s pack. +pub fn idx_key(hash: &str) -> String { + format!("packs/{hash}.idx") +} + +#[cfg(test)] +mod tests { + use super::*; + use scl_core::ObjectId; + + fn some_id(byte: u8) -> ObjectId { + // Any real id: hash a one-byte payload. Use whatever constructor + // refs::read_branch_tip uses to parse hex tips (check refs.rs:35 and + // reuse the identical call) — or simplest: ObjectId::of(&[byte]). + ObjectId::of(&[byte]) + } + + #[test] + fn manifest_round_trips_and_rejects_garbage() { + let m = Manifest { + head_seq: 7, + checkpoint_seq: 0, + head_branch: "main".into(), + }; + let bytes = m.encode(); + let back = Manifest::decode(&bytes).unwrap(); + assert_eq!(back.head_seq, 7); + assert_eq!(back.checkpoint_seq, 0); + assert_eq!(back.head_branch, "main"); + // wrong magic, truncated, future version, trailing junk: all refused + assert!(Manifest::decode(b"XXXX").is_err()); + assert!(Manifest::decode(&bytes[..bytes.len() - 1]).is_err()); + let mut future = bytes.clone(); + future[4] = 0xFF; // bump version byte + assert!(Manifest::decode(&future).is_err()); + let mut junk = bytes.clone(); + junk.push(0); + assert!(Manifest::decode(&junk).is_err()); + } + + #[test] + fn log_entry_round_trips_with_and_without_old_tips() { + let e = LogEntry { + seq: 3, + parent_seq: 2, + packs: vec!["ab12".into()], + updates: vec![ + RefUpdate { + branch: "main".into(), + old: Some(some_id(1)), + new: some_id(2), + }, + RefUpdate { + branch: "feat".into(), + old: None, + new: some_id(3), + }, + ], + }; + let back = LogEntry::decode(&e.encode()).unwrap(); + assert_eq!(back.seq, 3); + assert_eq!(back.parent_seq, 2); + assert_eq!(back.packs, vec!["ab12".to_string()]); + assert_eq!(back.updates.len(), 2); + assert_eq!(back.updates[0].old, Some(some_id(1))); + assert_eq!(back.updates[1].old, None); + assert_eq!(back.updates[1].new, some_id(3)); + } + + #[test] + fn decode_caps_hostile_lengths() { + // a length prefix claiming 1 GiB must fail fast, not allocate + let mut evil = Manifest { + head_seq: 1, + checkpoint_seq: 0, + head_branch: "m".into(), + } + .encode(); + let n = evil.len(); + evil[n - 2..].copy_from_slice(&[0xFF, 0xFF]); // corrupt branch length tail + assert!(Manifest::decode(&evil).is_err()); + } + + #[test] + fn keys_are_stable() { + assert_eq!(log_key(7), "log/00000000000000000007"); + assert_eq!(pack_key("abcd"), "packs/abcd.pack"); + assert_eq!(idx_key("abcd"), "packs/abcd.idx"); + } +} diff --git a/docs/THREAT-MODEL.md b/docs/THREAT-MODEL.md index f7044e8..3c44e22 100644 --- a/docs/THREAT-MODEL.md +++ b/docs/THREAT-MODEL.md @@ -308,6 +308,57 @@ too, not just the src-control-side metadata. - The ssh:// transport delegates authentication entirely to ssh (ADR-0022); `--max-pack-size` is the one P31 bound that also applies to `--stdio`. +## Bucket remotes (`sc+wal://` / `sc+s3://`) — ADR-0046 + +- **The bucket is untrusted storage with cooperative writers, not a trusted + peer.** Unlike `sc serve`, no process authenticates a push or enforces + policy at write time — the bucket vendor's own IAM/ACL is the only gate on + who may call `put`/`get`/`list` against it. sc treats every byte read back + from the bucket as untrusted input, symmetrically with how it already + treats bytes arriving over `ssh://`/`sc+http(s)://`. +- **Write access is full authority over refs and history.** Anyone who can + write to the bucket's manifest key can move any branch to any commit — + fast-forward-only enforcement (the manifest compare-and-swap plus the + `Error::NonFastForward` check) is **cooperative**, honored by the `sc` + client, not cryptographically enforced by the bucket. A writer with raw + bucket access (outside the `sc` client) can bypass it entirely, the same + way a user with raw filesystem access to a git bare repo can force-move a + ref underneath git's own client-side checks. Scope bucket write credentials + accordingly — they are equivalent to `sc serve --http` write-token + authority, not read-only authority. +- **Readers verify every object independently of the bucket's own + guarantees.** Every pack object's `BLAKE3` id is checked on read, exactly + as for local and served objects — a bucket that returns corrupted or + substituted bytes for a given key is caught, not trusted. The WAL metadata + (manifest, log entries, idx) is strict-decoded fail-closed + (`walfmt.rs`): unknown versions, bad magic, and any length that would + overrun the buffer are refused outright rather than best-effort parsed. + Untrusted-length caps apply throughout: manifest/log/idx bytes are capped + at `MAX_OBJECT_SIZE` before decode, and pack bodies are read through + `scl_core::pack::read_object_at_bounded` (not the unbounded + `read_object_at` the trusted local `Store` uses for its own already- + verified on-disk packs, per ADR-0039's explicit trust split) — capping + both the compressed record length and the decompressed output, so a + hostile or corrupted bucket cannot mount a decompression-bomb DoS via + `get_object`/`get_pack`. +- **Sealed content stays end-to-end sealed, unchanged.** Protected-path + ciphertext (ADR-0014/0043) and committed secrets travel through a bucket + remote exactly as through any other transport: a bucket reader without the + recipient key receives intact ciphertext it cannot decrypt. Bucket + read/write access confers no additional decryption capability. +- **Public (unprotected) content sits in the bucket as plaintext at rest.** + This is not a new exposure relative to a served remote's `.sc/objects/` — + both put unprotected object bytes on a disk or store the operator + controls — but a bucket has no `sc`-native access-control layer at all + (no bearer tokens, no `--read-only`, no loopback-bind gate). **Bucket ACL + is the entire confidentiality perimeter for public content on a bucket + remote**; an operator who needs `sc`-native read/write scoping for public + content should use `sc serve --http`/`--https` instead. +- **Partial clone (`--filter`) against a bucket remote is refused**, not + silently ignored — the WAL format has no per-prefix negotiation yet, so + there is no partial-fetch code path to reason about for a bucket remote at + all. + ## Untrusted-input hardening (DoS) — ADR-0039 - A single `MAX_OBJECT_SIZE` (256 MiB) caps every untrusted length: wire frames, diff --git a/docs/adr/0046-wal-bucket-remotes.md b/docs/adr/0046-wal-bucket-remotes.md new file mode 100644 index 0000000..5bc60c5 --- /dev/null +++ b/docs/adr/0046-wal-bucket-remotes.md @@ -0,0 +1,149 @@ +# ADR-0046: Bucket remotes via an immutable-pack WAL + one CAS'd manifest + +- **Status:** Accepted +- **Date:** 2026-08-26 +- **Phase:** 36a + +## Context + +Every remote sc has to date (ADR-0013's local-path transport, ADR-0022's +`ssh://`, ADR-0026's `sc+http://`/`sc+https://`) requires a long-lived `sc +serve` process or a reachable host running one: something has to be up, +listening, and single-writer-locked (`.sc/lock`) to accept a push. That rules +out the "no server to run" deployment shape a plain object store (S3, GCS, a +shared directory) offers — durable, highly available, IAM-native storage with +no process to keep alive. Cursor's public *Git at any scale* write-up (the +"Continuity" architecture) and the `walgit` project built on it +(`docs/research/walgit-evaluation.md`) demonstrate the shape: treat the +bucket itself as the repository, linearized by one small object that every +writer contends for with a compare-and-swap. + +We want the same property without the coordinator process: bucket write +access should be equivalent to `sc serve --http`-hosted write access, but the +"server" is whatever bucket API the storage vendor already runs. + +## Decision + +Add a **bucket remote**: a `Transport` implementation (`BucketTransport` in +`crates/repo/src/bucket_transport.rs`) whose object graph and refs live +entirely in a flat key/value bucket, structured as a **write-ahead log of +immutable packs plus one compare-and-swapped manifest**: + +- **Packs are immutable and content-addressed** — every `put_pack`/staged + `put_object` flush uploads a new pack keyed by its own hash + (`scl_core::pack::PackWriter`); nothing already in the bucket is ever + overwritten. +- **The log is parent-linked, not self-numbered.** Each `log/` entry + (`walfmt::LogEntry`) records the ref updates it applied and a + `parent_seq` pointing at the entry it was built on. Readers walk backward + from the manifest's `head_seq` following `parent_seq` — an entry's own + claimed `seq` is trusted only for self-consistency (it must match the slot + it was read from and strictly exceed its parent), never for reachability. + A losing racer's orphaned append is invisible to every reader by + construction, not by convention. +- **One manifest is the sole linearization point.** `walfmt::Manifest` + (`head_seq`, `checkpoint_seq`, `head_branch`) lives at a single well-known + key, rewritten with `Bucket::put_if_tag` (compare-and-swap on the prior + read's tag). A push appends its log entry, then CASes the manifest forward; + a 412-equivalent (tag mismatch) means someone else's push landed first — + the caller refreshes and reports `Error::NonFastForward`, exactly the + contract `LocalTransport`/`sc+http://` already give the CLI. **No leader + election, no lock file, no lease** — the bucket's native compare-and-swap + primitive is the only coordination the design needs. +- **`objio` is a new leaf crate** exposing the `Bucket` trait (`get`, + `put_new`, `put_if_tag`, `list`) plus two backends: `DirBucket` (a local + directory, for tests/demos and the `sc+wal://` scheme) and `S3Bucket` + (an S3-compatible object store, for `sc+s3://`). It depends on no other + workspace crate — object-store SDKs (`aws-sdk-s3`) are quarantined here the + same way `gix` is quarantined in `gitio` and RustCrypto in `crypto`. + `repo → objio` is a new leaf dependency edge alongside `repo → tlsio`. +- **Two URL schemes, one transport.** `sc+wal://` opens a `DirBucket` at + a local path (everything after the scheme is the directory); `sc+s3:// + /` splits the first path segment as the bucket name and + the remainder as a key prefix, opening an `S3Bucket`. Both dispatch through + the same `BucketTransport`; `BucketUrl::parse` rejects a malformed URL + (e.g. bare `sc+s3://` with no bucket name) at `sc remote add` time, before + any network or filesystem call. +- **Partial clone (`--filter`) is refused against bucket remotes.** The WAL + format has no per-prefix negotiation yet; a filtered clone needs a served + remote (ADR-0037) until that lands. +- **Untrusted-length guard on reads.** Manifest, log entries, and idx + metadata are capped at `MAX_OBJECT_SIZE` (256 MiB, ADR-0039) before + decoding — a hostile or corrupted bucket cannot force an unbounded + allocation. Pack bodies are read through + `scl_core::pack::read_object_at_bounded`, which caps both the compressed + record length and the decompressed output at `MAX_OBJECT_SIZE`, mirroring + `parse_pack_reader`'s bounded decode — this is the same zstd-bomb guard + P28 put on every other untrusted transfer path. **This is deliberately a + different code path from `Store`'s own on-disk reads:** the trusted local + object store still calls the unbounded `read_object_at`, per ADR-0039's + explicit split between "objects this process already verified onto local + disk" (unbounded) and "objects arriving from something we don't control" + (bounded). A bucket, even one the user configured, is on the untrusted + side of that line — nothing about owning the credentials that let you + *write* to a bucket implies the bytes already sitting there are trustworthy + reads. + +**Confidentiality property, unchanged from ADR-0013:** transfer moves objects +verbatim. Encrypted-path blobs (P7) and secret objects travel as ciphertext +through the bucket exactly as through any other transport — a bucket reader +without the recipient key gets bytes it cannot decrypt. What *is* new: +**public (unprotected) content sits in the bucket as plaintext at rest**, +same as it would in a served remote's `.sc/objects/`, so bucket ACLs are the +confidentiality perimeter for public content, not the sc protocol. + +## Consequences + +- A bucket remote needs no long-running `sc serve` process and no listener + to secure (ADR-0031/ADR-0040 resource-limit and access-control machinery + is simply not in play) — the trade is that the bucket vendor's IAM/ACL + becomes the access-control surface instead. +- Multiple writers can push concurrently with no shared lock file and no + out-of-band coordination; the manifest CAS is the only serialization + point, proven under a fleet of racing writers (`bucket_transport.rs` test + suite). +- The log grows unboundedly with no compaction yet — every open reader walks + the full parent chain back from `head_seq`. Checkpoint folding (P36b) and + bucket GC/compaction are deferred (see `ROADMAP.md`). +- `sc serve` still cannot host a bucket as its backing store (P36c) — a + bucket remote today is written to directly by every `sc push`/`sc fetch` + client, not brokered through a server process. +- Partial clone, and by extension every partial-clone-only op, is unavailable + against a bucket remote until per-prefix negotiation is designed. + +## Alternatives considered + +- **A dumb ref-file-per-branch bucket** (one object per branch pointing at a + tip, objects written loose). Simpler to implement, but has no atomic + multi-branch update and no append-only log for a fleet of writers to + reconcile against — two concurrent pushes to different branches can leave + the bucket in a state no single writer ever intended, and there is nothing + to replay to detect or recover from it. Rejected. +- **A `Store`-level backend** (make the object store itself pluggable, with + a bucket-backed `Store` impl sitting where `.sc/objects/` sits today). + Reopens the P3 (persistent store) and P8 (packfiles/GC) designs to a third + backend and blurs the transport/storage boundary those phases established. + Deferred — the transport-level design keeps `Store` untouched and ships + faster. +- **Adopt `walgit` itself** as (or in front of) `sc serve`. `walgit` is a + git-smart-HTTP server over a bucket WAL, not an sc-native peer — using it + would mean speaking git's wire protocol, which is exactly the git-bridge + role P18 already covers (`sc remote add --git`). Its design (CAS'd + manifest as linearization point, checkpoint-plus-log-tail cold start, + bundle-uri static clones) is the direct inspiration for this ADR's + structure; see `docs/research/walgit-evaluation.md` for the full + evaluation and which of its ideas remain deferred (checkpoint folding, + static-bundle clone offload). + +## As built (P36a) + +`crates/objio` (leaf crate: `Bucket` trait, `DirBucket`, `S3Bucket`); +`crates/repo/src/walfmt.rs` (versioned, strict, fail-closed WAL encoding: +`Manifest`, `LogEntry`, `RefUpdate`); `crates/repo/src/bucket_transport.rs` +(`BucketTransport`, full `Transport` impl, `BucketUrl` parse/dispatch); wired +into `open_transport` for `sc+wal://` and `sc+s3://`, validated eagerly at +`sc remote add`. 12 tests in `bucket_transport.rs` covering the base +round-trip, racing/fleet pushes, crash-mid-push recovery, and the +`MAX_OBJECT_SIZE` cap. CLI plumbing (remote-add validation, push, clone, +fetch through the real binary) proven in +`crates/cli/tests/bucket_remote.rs`. diff --git a/docs/research/walgit-evaluation.md b/docs/research/walgit-evaluation.md new file mode 100644 index 0000000..98a668d --- /dev/null +++ b/docs/research/walgit-evaluation.md @@ -0,0 +1,244 @@ +# walgit evaluation — could/should src-control use it as a git server? + +Research date: 2026-08-26 + +## Recommendation (summary) + +walgit ([github.com/tobi/walgit](https://github.com/tobi/walgit)) is a +three-day-old, MIT-licensed Rust git server — one binary in front of an +S3/GCS bucket, implementing Cursor's "Continuity" write-ahead-log +architecture, serving git **smart HTTP v0/v2 only** (no SSH), by Tobi Lütke +with two other contributors and 9 commits total. + +- **(a) As a P18 bridge remote endpoint:** **Could: yes, today, zero code + changes.** It is a standard smart-HTTP git server and our bridge delegates + transport and auth to system `git`. **Should: no action needed** — nothing + to build; at most a smoke test if a user reports it. +- **(b) As a recommended/bundled self-hostable server:** **Should: not yet.** + Days old, pre-1.0 with an explicit no-backward-compat policy for everything + except bucket data, effectively a single-org project. Revisit in 6–12 + months; the deployment story (one binary + bucket) is genuinely the best in + class if it matures. +- **(c) As a design source:** **Should: yes — this is the real value.** The + CAS'd-manifest-as-linearization-point, bundle-uri static clones, and + checkpoint-plus-log-tail cold start map directly onto sc's content-addressed + store and Deferred roadmap items. Read `AGENTS.md` and + `docs/BUNDLE_URI_DESIGN.md` there; do not import code. + +## What walgit is (primary sources) + +All claims from the repo itself unless noted. + +**Metadata** ([GitHub API `/repos/tobi/walgit`](https://api.github.com/repos/tobi/walgit), +fetched 2026-08-26): language Rust, license MIT, created 2026-08-23, last push +2026-08-25, 1,686 stars, 92 forks, 3 open issues. **9 commits total**, 3 +contributors — igrigorik (5), tobi (3, Tobi Lütke), dsfaccini (1) — per the +[commits](https://api.github.com/repos/tobi/walgit/commits) and +[contributors](https://api.github.com/repos/tobi/walgit/contributors) API. +Initial public release commit is dated 2026-08-23 ("walgit: initial public +release"), i.e. the code was developed privately and dropped as one squash — +the public history says nothing about real development duration. + +**Thesis** ([README.md](https://github.com/tobi/walgit/blob/main/README.md)): +"a git server that is one binary in front of an object store … no database, no +leader and no local state that matters." Every instance is a disposable cache; +"the bucket is the repository." It is explicitly "a Rust implementation of the +architecture Cursor described in *Git at any scale* (the system they call +Continuity)" — the Cursor post is vendored verbatim at +`docs/reference/cursor-git-at-any-scale.md`. + +**Storage model** (README "How it works" + [AGENTS.md](https://github.com/tobi/walgit/blob/main/AGENTS.md) §2): +the repository lives in the bucket as a WAL under `repos///`: + +- `manifest.pb` — tiny, **compare-and-swap rewritten**; the single + linearization point. "That CAS *is* the consensus — no election, no quorum, + no primary" (README). +- `log/.pb` — immutable entries (PUSH, COMPACT, CHECKPOINT, SETTINGS). +- `wal/.pack|.idx|.rev|.bitmap|.commit-graph` — immutable + content-addressed packs. +- `checkpoints//` — folded ref snapshot so cold start is snapshot + tail. +- `leases/` — CAS-with-TTL, "the only cross-instance mutex." + +A push: index the pack in a scratch dir (`git index-pack --fix-thin`), check +connectivity + policy, upload pack + log entry, CAS the manifest; on a 412 +re-read and retry. A read: one conditional GET of the manifest (304 → serve +local). Reads are strictly consistent — "there is no 'eventually'" (README +invariants). + +**Protocols served** (README "What it does" table): git **smart HTTP v0/v2** +— `ls-refs` with prefixes, fetch with filter/shallow/deepen, receive-pack +(atomic, deletes, tags, push options, report-status-v2), sha1 and sha256 +repos; plus **bundle-uri** static-file clones, **Git LFS** (batch API + basic +transfer, objects in the bucket), a React web UI + JSON API/SDK, per-repo push +policy (`policy.json`), webhooks, and auth modes `none`/`token`(bearer or +Basic)/`oidc`. **No SSH transport anywhere in the tree** — the git surface is +HTTP-only (confirmed against the full recursive +[tree listing](https://api.github.com/repos/tobi/walgit/git/trees/main?recursive=1): +no ssh module exists; `crates/walgit-server/src/tls.rs` + rustls/rcgen handle +in-process HTTPS). + +**Implementation** ([Cargo.toml](https://github.com/tobi/walgit/blob/main/Cargo.toml)): +Rust edition 2024, rust-version 1.90, tokio + axum + hyper, prost/tonic +protobuf, aws-sdk-s3 + google-cloud-storage backends, rustls 0.23 + rcgen for +in-process TLS, and a **heavy `gix` dependency** (gix 0.86 plus ~18 individual +gix-* crates: gix-pack, gix-protocol, gix-transport, gix-odb, gix-negotiate, +…). Division of labor per README: "Upstream `git` does upload-pack/repack/ +bundle; walgit does receive-pack, the WAL and the plumbing" — i.e. **system +git is a runtime requirement on the server**, with gix used where measured +faster (GOAL.md §7: "Upstream `git` where it is right … `gix` where it is +faster and measured"). There is also a gix-based upload-pack driver +(`crates/walgit-git/src/upload_gix.rs`). + +**Maturity signals, both directions.** Young: created 2026-08-23; 9 commits; +version 0.1.0; AGENTS.md opens with "**No backwards compatibility (pre-1.0)** +… Data in the bucket is the one exception." CI was only added 2026-08-25 by +the second contributor ("Add a CI workflow", commit history), fixing a broken +workspace build and non-gating test gates in the same PR — the initial drop +did not build clean. Serious: the test surface is unusually large for a new +repo — a ~104 KB fault-injection simulation suite +(`crates/walgit-server/tests/sim.rs`, "crashes, partitions, stale reads" per +README), a ~109 KB e2e suite driving real git, a cross-backend store contract +suite (`crates/walgit-store/tests/contract.rs`), and a 500k-ref scale test +(`crates/walgit-git/tests/refs500k.rs`). [GOAL.md](https://github.com/tobi/walgit/blob/main/GOAL.md) +cites a reference workload of a 57 GiB / 73 M-object / 466 k-ref monorepo +(clone 2075 s → 8 s with blob:none + bundles). These are the project's own +claims; nobody outside it has had time to validate them. + +**License:** MIT (`LICENSE`, confirmed by the API license field). No +compatibility issue with this workspace. + +## The src-control side (what "a git server" means for us) + +Grounding, from this repo's own docs: + +- Hosted-Git interop (P18) is a **system-git mirror bridge** + (`docs/adr/0028-network-git-remotes.md`): each git-backed network remote + keeps a bare mirror at `.sc/git-remotes//mirror.git`; `sc fetch` runs + `git fetch --prune` into it, `sc push` runs the P10 export then `git push`. + "Auth is fully delegated to the spawned `git` (ssh-agent, credential + helpers, tokens); … `sc` has no credential surface." `SC_GIT` overrides the + binary (`crates/gitio/src/bridge.rs:30`). The chosen ADR rationale: gix + cannot push, so system git is transport-only and the gix quarantine + (ADR-0007) holds for object translation. +- Identity across the boundary is the persisted `git_oid ↔ sc_id` marks map + (`docs/adr/0018-git-as-a-remote.md`), independent of which git server sits + at the other end. +- sc's native transports are its own: framed-stdio ssh:// (ADR-0022), + sc+http:// (ADR-0036), sc+https:// via `tlsio` with TOFU pinning + (ADR-0042), wire `PROTOCOL_VERSION` 4, streaming packs (ADR-0035). These + speak the sc object model (BLAKE3 CAS, sealed objects) — **not** git's + protocol — so a git server can never serve them. + +Consequence: for src-control, "the git side of interop needs a standard git +remote endpoint" is the whole requirement. Anything `git fetch`/`git push` +can talk to works, unmodified. + +## Evaluation by slot + +### (a) Remote endpoint for the P18 bridge — could: yes; should: nothing to do + +walgit serves standard smart HTTP v0/v2 including receive-pack with +report-status-v2 (README feature table), and authenticates via +`Authorization: Bearer` or the token as an HTTP Basic password, with an +install script that configures a git credential helper (README +"Authentication"). Our bridge spawns real `git` and delegates auth to git's +credential machinery (ADR-0028), so `sc remote add wg https://git.example.com/owner/repo.git --git` +should work against walgit exactly as against GitHub — including +`auto_create_on_push` creating the repo on first push (README quickstart). + +Two caveats, neither blocking: + +- **HTTP only.** A user whose walgit is auth'd via bearer token uses git's + credential helper (walgit's installer sets one up); our bridge passes git's + stderr through unmodified, so walgit's sideband-2 progress narration + surfaces naturally. +- **bundle-uri** is a client-side optimization git negotiates itself + (`transfer.bundleURI`); the mirror fetch neither needs nor conflicts + with it. + +Fit: automatic. Maturity/maintenance risk: irrelevant here — the risk is the +user's, whoever operates the endpoint, and no src-control code or docs depend +on it. **Recommendation: no work. Optionally note in a future P18 doc pass +that any smart-HTTP server (GitHub, GitLab, cgit+nginx, walgit) works.** + +### (b) A self-hostable server to recommend or bundle — should: not yet + +What it would buy over alternatives: `git daemon`/gitolite/plain-ssh-bare-repo +need a real filesystem and per-machine state; forges (Gitea/GitLab) need a +database and upkeep. walgit is one binary + one bucket, disposable instances, +built-in TLS, OIDC, LFS, and a web UI — operationally the simplest +self-hosting story described anywhere, *if the claims hold*. + +Why not yet, with evidence: + +- **Age and bus factor.** Public for three days at research time (created + 2026-08-23); 9 commits; effectively one author plus two drive-by + contributors. No releases, no tags, no published binaries or crates.io + packages (API: `has_downloads: false`, no releases). +- **Explicit instability.** AGENTS.md: pre-1.0, "no backwards compatibility … + delete the old shape in the same change" for routes, config keys, clients — + only bucket data formats are stable. Recommending it to users means + recommending a moving target. +- **Unvalidated by anyone else.** The impressive numbers (2075 s → 8 s clone) + are the project's own acceptance table (GOAL.md); the fault-injection sim + is self-authored. Three days is not enough external exercise for a system + whose consistency story rests on object-store CAS semantics across S3 + vendors. +- **We have no bundling need.** src-control's native remotes already cover + self-hosting sc-to-sc (`sc serve --stdio|--http`, ADR-0022/0036/0042); the + git side exists for interop with wherever users already are, which is + overwhelmingly hosted forges. + +**Recommendation: do not recommend or bundle now. Re-evaluate around +2027-Q1: look for tagged releases, external contributors, and independent +deployment reports.** MIT license poses no obstacle whenever that happens. + +### (c) Design source for our own roadmap — should: yes, actively + +This is where walgit (and the Cursor Continuity design it implements — +vendored at walgit's `docs/reference/cursor-git-at-any-scale.md`) earns study +time. Direct resonances with sc: + +- **CAS'd manifest as the only commit point.** walgit's entire consistency + model is "immutable content-addressed objects + one tiny CAS'd pointer" + (README invariants). sc's store is already content-addressed + (BLAKE3, CLAUDE.md invariants); its refs are the mutable pointer. If sc + ever grows an object-storage backend or a multi-writer hosted mode, the + manifest-CAS pattern is the proven shape — it replaces our single-writer + `.sc/` lock (ADR-0011) with object-store primitives and no coordinator. + Notably, ADR-0013 already anticipated "remote/managed-Git backends behind + adapters" on the `Transport`/store seam. +- **Checkpoint + log-tail cold start** (walgit AGENTS.md §2.1) is the same + idea as our oplog (ADR-0024) applied to *state reconstruction* rather than + undo — relevant if sc servers ever need to be disposable caches. +- **bundle-uri static clones** (walgit `docs/BUNDLE_URI_DESIGN.md`): moving + clone bytes out of the server into CDN-able immutable artifacts. sc's + streaming pack transfer (ADR-0035) still ships every byte through + `sc serve`; a "static bundle cut as a pure function of the log" would bound + server cost for large sc repos. Candidate ROADMAP → Deferred entry. +- **Tasks/narration** ("nothing waits silently" — walgit README): sideband + progress for long server-side work is a UX idea our transports could adopt. +- **What not to copy:** walgit is plaintext-at-rest by design (the bucket + holds cleartext packs; auth is perimeter-only). sc's sealed objects, + protected paths, and private branches (ADR-0014/0043/0044) are exactly the + properties walgit's model lacks — an sc-over-object-store design would keep + our sealing and borrow only their coordination layer. + +**Recommendation: add a ROADMAP → Deferred note referencing this file for +(i) object-store-backed sc remotes via manifest-CAS and (ii) static bundle +offload for `sc serve` — both explicitly "recorded, not built."** (Not done +in this research pass; ROADMAP edits are out of scope for a research note.) + +## Sources + +- https://api.github.com/repos/tobi/walgit (metadata, license, dates, counts) +- https://api.github.com/repos/tobi/walgit/commits, …/contributors +- https://github.com/tobi/walgit/blob/main/README.md +- https://github.com/tobi/walgit/blob/main/GOAL.md +- https://github.com/tobi/walgit/blob/main/AGENTS.md +- https://github.com/tobi/walgit/blob/main/Cargo.toml +- Repo tree: https://api.github.com/repos/tobi/walgit/git/trees/main?recursive=1 +- Local: `CLAUDE.md`, `docs/adr/0013-remote-sync-model.md`, + `docs/adr/0018-git-as-a-remote.md`, `docs/adr/0028-network-git-remotes.md`, + `docs/adr/0022-ssh-native-transport.md`, `docs/adr/0036-http-transport.md`, + `docs/adr/0042-in-binary-tls-sc-https.md`, `crates/gitio/src/bridge.rs` diff --git a/docs/superpowers/plans/2026-08-26-wal-bucket-backend-p36a.md b/docs/superpowers/plans/2026-08-26-wal-bucket-backend-p36a.md new file mode 100644 index 0000000..771a89b --- /dev/null +++ b/docs/superpowers/plans/2026-08-26-wal-bucket-backend-p36a.md @@ -0,0 +1,1712 @@ +# P36a: WAL Bucket Backend — objio + BucketTransport Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** An S3-compatible bucket (or a local directory standing in for one) becomes an sc remote — `sc remote add origin sc+s3://bucket/prefix` (or `sc+wal://`) with clone/fetch/push, multi-writer safety via one compare-and-swapped manifest, and zero coordinator. + +**Architecture:** New leaf crate `objio` (trait `Bucket` + `DirBucket` + `S3Bucket`) quarantines the S3 SDK exactly as `tlsio` quarantines rustls. In `scl-repo`, a `walfmt` module defines the versioned manifest/log-entry codec and a `bucket_transport` module implements the existing `Transport` trait over a `Bucket`, so `open_transport` dispatch gives clone/fetch/push for free — `sync.rs` is untouched. Spec: `docs/superpowers/specs/2026-08-26-wal-bucket-backend-design.md`. Checkpoints (P36b) and bucket-backed `sc serve` (P36c) are separate follow-on plans; the manifest carries a `checkpoint_seq` field from day one so P36b is format-compatible. + +**Tech Stack:** Rust 2021 (workspace-inherited), `thiserror`, `blake3`, `hex`; `objio` additionally `aws-sdk-s3` + `aws-config` + `tokio` (current-thread runtime, confined inside the crate behind a sync API). + +## Global Constraints + +- Dependency direction: `repo → objio`; `objio` depends on **no workspace crate** (leaf, like `tlsio`). `core` never learns about buckets. +- New crate manifest inherits exactly `version`/`edition`/`license`/`publish` via `.workspace = true`, package name `scl-objio`, ends with `[lints] workspace = true` (house pattern, per `crates/tlsio/Cargo.toml`). +- Add deps with `cargo add`, never hand-guessed pins. +- Everything read from a bucket is untrusted: every decoded length is bounds-checked, `scl_core::MAX_OBJECT_SIZE` caps any single fetched value, unknown format versions fail closed. +- Errors: `thiserror` enums, lowercase messages, no trailing period; CLI converts with `?` into `anyhow`. +- Never silently drop data; CAS retry exhaustion and non-fast-forward fail loudly. +- Every test that touches disk cleans up and asserts the path is gone (`let _ = remove_dir_all` at setup, bare `.unwrap()` at teardown — house style per `crates/repo/src/sync.rs:615`). +- Public types/fns get doc comments explaining intent, not mechanics. +- Bucket keys are fixed internal strings (`manifest`, `log/`, `packs/.pack|.idx`); `DirBucket` still validates keys against path traversal. +- Partial-clone `filter` over bucket remotes is refused loudly (recorded deferred), never silently ignored. + +--- + +### Task 1: `objio` crate — `Bucket` trait, `DirBucket`, contract tests + +**Files:** +- Create: `crates/objio/Cargo.toml` +- Create: `crates/objio/src/lib.rs` +- Create: `crates/objio/src/dir.rs` +- Modify: `Cargo.toml` (workspace members) +- Test: in `#[cfg(test)] mod tests` inside `crates/objio/src/lib.rs` (shared contract fn) — house style is tests next to code + +**Interfaces:** +- Consumes: nothing in-workspace (leaf). +- Produces (used by Tasks 2, 4, 5): + - `scl_objio::Error` (thiserror enum), `scl_objio::Result` + - `pub enum Fetched { Unchanged, Absent, New { bytes: Vec, tag: String } }` + - `pub trait Bucket: Send { fn get(&self, key: &str, cached_tag: Option<&str>) -> Result; fn put_new(&self, key: &str, bytes: &[u8]) -> Result; fn put_if_tag(&self, key: &str, bytes: &[u8], expected_tag: Option<&str>) -> Result>; fn list(&self, prefix: &str) -> Result>; }` + - `pub struct DirBucket; impl DirBucket { pub fn open(root: impl Into) -> Result }` + - `pub fn contract_suite(b: &dyn Bucket)` — reusable conformance checks (pub so the S3 test reuses it) + +- [ ] **Step 1: Scaffold the crate and register it** + +```bash +mkdir -p crates/objio/src +``` + +`crates/objio/Cargo.toml`: +```toml +[package] +name = "scl-objio" +version.workspace = true +edition.workspace = true +license.workspace = true +publish.workspace = true + +[dependencies] +thiserror = "2.0.18" + +[lints] +workspace = true +``` + +Then `cargo add --package scl-objio blake3 hex` (tag computation), and edit root `Cargo.toml` members to: +```toml +members = ["crates/core", "crates/vfs", "crates/gitio", "crates/crypto", "crates/repo", "crates/cli", "crates/tlsio", "crates/objio", "apps/desktop/src-tauri"] +``` + +- [ ] **Step 2: Write the failing contract test** + +In `crates/objio/src/lib.rs` (the trait/enum/error will not exist yet — that is the point): + +```rust +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn dir_bucket_passes_contract() { + let root = std::env::temp_dir().join(format!("scl-objio-dir-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + let b = DirBucket::open(&root).unwrap(); + contract_suite(&b); + std::fs::remove_dir_all(&root).unwrap(); + assert!(!root.exists()); + } +} +``` + +And the contract body (in `lib.rs`, `pub` — Task 2's S3 test reuses it): + +```rust +/// Conformance checks every `Bucket` implementation must pass. +/// Panics on violation (test helper). +pub fn contract_suite(b: &dyn Bucket) { + // absent key + assert!(matches!(b.get("manifest", None).unwrap(), Fetched::Absent)); + // put_if_tag with expected None = create; returns the new tag + let t1 = b.put_if_tag("manifest", b"v1", None).unwrap().expect("create succeeds"); + // create again must fail (precondition) + assert!(b.put_if_tag("manifest", b"v1x", None).unwrap().is_none()); + // conditional get: matching tag => Unchanged; stale/no tag => New with same tag + assert!(matches!(b.get("manifest", Some(&t1)).unwrap(), Fetched::Unchanged)); + let Fetched::New { bytes, tag } = b.get("manifest", None).unwrap() else { panic!("expected New") }; + assert_eq!(bytes, b"v1"); + assert_eq!(tag, t1); + // CAS: wrong tag refused, right tag succeeds and returns a new tag + assert!(b.put_if_tag("manifest", b"v2", Some("bogus")).unwrap().is_none()); + let t2 = b.put_if_tag("manifest", b"v2", Some(&t1)).unwrap().expect("cas succeeds"); + assert_ne!(t1, t2); + // put_new: first write true, second false, content untouched + assert!(b.put_new("log/1", b"entry-one").unwrap()); + assert!(!b.put_new("log/1", b"entry-two").unwrap()); + let Fetched::New { bytes, .. } = b.get("log/1", None).unwrap() else { panic!() }; + assert_eq!(bytes, b"entry-one"); + // list is prefix-scoped and sorted + assert!(b.put_new("log/2", b"x").unwrap()); + assert_eq!(b.list("log/").unwrap(), vec!["log/1".to_string(), "log/2".to_string()]); + assert_eq!(b.list("packs/").unwrap(), Vec::::new()); +} +``` + +- [ ] **Step 3: Run to verify it fails** + +Run: `cargo test -p scl-objio` +Expected: compile FAIL — `Bucket`, `Fetched`, `DirBucket`, `contract_suite` not found. + +- [ ] **Step 4: Implement `lib.rs` (error, trait, enum) and `dir.rs`** + +`crates/objio/src/lib.rs`: +```rust +//! Object-storage access for sc bucket remotes (P36). +//! +//! Quarantine rule: object-store SDKs live here and only here — the rest of +//! the workspace sees the [`Bucket`] trait. Leaf crate: depends on no other +//! workspace crate (like `tlsio`). + +mod dir; +pub use dir::DirBucket; + +#[derive(Debug, thiserror::Error)] +pub enum Error { + #[error("bucket key rejected: {0}")] + BadKey(String), + #[error("bucket io: {0}")] + Io(#[from] std::io::Error), + #[error("bucket backend: {0}")] + Backend(String), +} + +pub type Result = std::result::Result; + +/// Outcome of a conditional read. +pub enum Fetched { + /// The caller's cached tag still matches — no bytes transferred. + Unchanged, + /// The key does not exist. + Absent, + /// Current value plus its tag (ETag / content hash). + New { bytes: Vec, tag: String }, +} + +/// A flat key/value object store with the three primitives the WAL needs: +/// conditional read, create-if-absent, and compare-and-swap overwrite. +pub trait Bucket: Send { + /// Conditional read. `cached_tag` matching the current value returns + /// [`Fetched::Unchanged`] without transferring bytes. + fn get(&self, key: &str, cached_tag: Option<&str>) -> Result; + /// Create-only write (if-none-match). `Ok(false)` = key already exists; + /// the existing value is never touched. + fn put_new(&self, key: &str, bytes: &[u8]) -> Result; + /// Compare-and-swap overwrite. `expected_tag: None` = create-new. + /// `Ok(None)` = precondition failed (someone else won); `Ok(Some(tag))` + /// = committed, with the new value's tag. + fn put_if_tag(&self, key: &str, bytes: &[u8], expected_tag: Option<&str>) -> Result>; + /// Keys under `prefix`, sorted. + fn list(&self, prefix: &str) -> Result>; +} + +/// Reject traversal and absolute keys before any backend touches them. +pub(crate) fn validate_key(key: &str) -> Result<()> { + if key.is_empty() + || key.starts_with('/') + || key.contains('\\') + || key.split('/').any(|c| c.is_empty() || c == "." || c == "..") + || key.chars().any(|c| c.is_whitespace() || c.is_control()) + { + return Err(Error::BadKey(format!("{key:?}"))); + } + Ok(()) +} +``` +(plus `contract_suite` from Step 2 and the test module). + +`crates/objio/src/dir.rs` — tag is `hex(blake3(bytes))`; CAS is serialized by a spin-held lock file (local-machine simulation of the S3 precondition; real S3 needs no lock): +```rust +//! Local-directory `Bucket` — the test/demo backend behind `sc+wal://`. + +use crate::{validate_key, Bucket, Error, Fetched, Result}; +use std::path::{Path, PathBuf}; + +pub struct DirBucket { + root: PathBuf, +} + +fn tag_of(bytes: &[u8]) -> String { + hex::encode(blake3::hash(bytes).as_bytes()) +} + +impl DirBucket { + /// Open (creating if needed) a directory as a bucket. + pub fn open(root: impl Into) -> Result { + let root = root.into(); + std::fs::create_dir_all(&root)?; + Ok(DirBucket { root }) + } + + fn key_path(&self, key: &str) -> Result { + validate_key(key)?; + Ok(self.root.join(key)) + } + + /// Spin-acquire `/.cas-lock`; bounded so a crashed holder surfaces + /// as a loud error, not a hang. + fn lock(&self) -> Result { + let path = self.root.join(".cas-lock"); + for _ in 0..2000 { + match std::fs::OpenOptions::new().write(true).create_new(true).open(&path) { + Ok(_) => return Ok(CasLock { path }), + Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => { + std::thread::sleep(std::time::Duration::from_millis(1)); + } + Err(e) => return Err(e.into()), + } + } + Err(Error::Backend(format!("cas lock stuck (stale {} ?)", path.display()))) + } + + fn write_via_tmp(&self, path: &Path, bytes: &[u8]) -> Result<()> { + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent)?; + } + let tmp = path.with_extension(format!("tmp-{}", std::process::id())); + std::fs::write(&tmp, bytes)?; + std::fs::rename(&tmp, path)?; + Ok(()) + } +} + +struct CasLock { + path: PathBuf, +} +impl Drop for CasLock { + fn drop(&mut self) { + let _ = std::fs::remove_file(&self.path); + } +} + +impl Bucket for DirBucket { + fn get(&self, key: &str, cached_tag: Option<&str>) -> Result { + let path = self.key_path(key)?; + let bytes = match std::fs::read(&path) { + Ok(b) => b, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(Fetched::Absent), + Err(e) => return Err(e.into()), + }; + let tag = tag_of(&bytes); + if cached_tag == Some(tag.as_str()) { + return Ok(Fetched::Unchanged); + } + Ok(Fetched::New { bytes, tag }) + } + + fn put_new(&self, key: &str, bytes: &[u8]) -> Result { + let path = self.key_path(key)?; + let _lock = self.lock()?; + if path.exists() { + return Ok(false); + } + self.write_via_tmp(&path, bytes)?; + Ok(true) + } + + fn put_if_tag(&self, key: &str, bytes: &[u8], expected_tag: Option<&str>) -> Result> { + let path = self.key_path(key)?; + let _lock = self.lock()?; + let current = match std::fs::read(&path) { + Ok(b) => Some(tag_of(&b)), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => None, + Err(e) => return Err(e.into()), + }; + if current.as_deref() != expected_tag { + return Ok(None); + } + self.write_via_tmp(&path, bytes)?; + Ok(Some(tag_of(bytes))) + } + + fn list(&self, prefix: &str) -> Result> { + fn walk(dir: &Path, root: &Path, out: &mut Vec) -> std::io::Result<()> { + let rd = match std::fs::read_dir(dir) { + Ok(rd) => rd, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(()), + Err(e) => return Err(e), + }; + for entry in rd { + let entry = entry?; + let p = entry.path(); + if p.file_name().is_some_and(|n| { + let n = n.to_string_lossy(); + n == ".cas-lock" || n.contains(".tmp-") + }) { + continue; + } + if p.is_dir() { + walk(&p, root, out)?; + } else { + out.push(p.strip_prefix(root).unwrap().to_string_lossy().replace('\\', "/")); + } + } + Ok(()) + } + validate_key(prefix.trim_end_matches('/'))?; + let mut out = Vec::new(); + walk(&self.root, &self.root, &mut out)?; + out.retain(|k| k.starts_with(prefix)); + out.sort(); + Ok(out) + } +} +``` + +- [ ] **Step 5: Run to verify it passes** + +Run: `cargo test -p scl-objio` +Expected: PASS (`dir_bucket_passes_contract`). + +- [ ] **Step 6: Commit** + +```bash +git add Cargo.toml Cargo.lock crates/objio +git commit -m "feat(objio): leaf crate with Bucket trait, DirBucket, contract suite (P36a)" +``` +(Cargo.lock is staged with the dep change — house rule.) + +--- + +### Task 2: `S3Bucket` — S3-compatible backend behind the same trait + +**Files:** +- Create: `crates/objio/src/s3.rs` +- Modify: `crates/objio/src/lib.rs` (add `mod s3; pub use s3::S3Bucket;`) +- Modify: `crates/objio/Cargo.toml` (via `cargo add`) +- Test: env-gated test in `crates/objio/src/s3.rs` + +**Interfaces:** +- Consumes: `Bucket`, `Fetched`, `Error`, `Result`, `validate_key`, `contract_suite` from Task 1. +- Produces (used by Task 6): `pub struct S3Bucket; impl S3Bucket { pub fn open(bucket: &str, prefix: &str) -> Result }` — implements `Bucket`. Credentials/region/endpoint come from the SDK's standard chain (`AWS_*` env, config files, `AWS_ENDPOINT_URL_S3` for MinIO/R2); sc adds no credential surface. + +- [ ] **Step 1: Add the SDK deps** + +```bash +cargo add --package scl-objio aws-config aws-sdk-s3 tokio --features tokio/rt +``` +(Exact feature flags: `tokio` needs only `rt`; if `cargo add` output shows `aws-config` requires the `behavior-version-latest` feature for `aws_config::load_from_env`, enable it — follow the SDK's own compile errors, do not guess pins.) + +- [ ] **Step 2: Write the env-gated failing test** + +In `crates/objio/src/s3.rs`: +```rust +#[cfg(test)] +mod tests { + /// Live-backend parity: set SC_OBJIO_S3_BUCKET (and standard AWS_* env, + /// e.g. AWS_ENDPOINT_URL_S3 for MinIO) to run; skipped otherwise so CI + /// stays hermetic on DirBucket. + #[test] + fn s3_bucket_passes_contract_when_configured() { + let Ok(bucket) = std::env::var("SC_OBJIO_S3_BUCKET") else { + eprintln!("skipped: SC_OBJIO_S3_BUCKET not set"); + return; + }; + let prefix = format!("scl-objio-contract-{}", std::process::id()); + let b = super::S3Bucket::open(&bucket, &prefix).unwrap(); + crate::contract_suite(&b); + } +} +``` + +- [ ] **Step 3: Run to verify it fails to compile** + +Run: `cargo test -p scl-objio` +Expected: compile FAIL — `S3Bucket` not found. + +- [ ] **Step 4: Implement `S3Bucket`** + +Shape (the SDK calls are the part to adapt to the compiler — semantics are fixed): +```rust +//! S3-compatible `Bucket` backend (AWS, MinIO, R2 via AWS_ENDPOINT_URL_S3). + +use crate::{validate_key, Bucket, Error, Fetched, Result}; + +pub struct S3Bucket { + rt: tokio::runtime::Runtime, + client: aws_sdk_s3::Client, + bucket: String, + prefix: String, +} + +impl S3Bucket { + /// Connect using the SDK's standard credential/region/endpoint chain. + pub fn open(bucket: &str, prefix: &str) -> Result { + let rt = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .map_err(|e| Error::Backend(format!("tokio runtime: {e}")))?; + let conf = rt.block_on(aws_config::load_from_env()); + Ok(S3Bucket { + client: aws_sdk_s3::Client::new(&conf), + rt, + bucket: bucket.to_string(), + prefix: prefix.trim_matches('/').to_string(), + }) + } + + fn full_key(&self, key: &str) -> Result { + validate_key(key)?; + Ok(if self.prefix.is_empty() { key.to_string() } else { format!("{}/{key}", self.prefix) }) + } +} + +impl Bucket for S3Bucket { + fn get(&self, key: &str, cached_tag: Option<&str>) -> Result { + let k = self.full_key(key)?; + let mut req = self.client.get_object().bucket(&self.bucket).key(&k); + if let Some(tag) = cached_tag { + req = req.if_none_match(tag); + } + match self.rt.block_on(async { + match req.send().await { + Ok(out) => { + let tag = out.e_tag().unwrap_or_default().to_string(); + let bytes = out.body.collect().await.map(|b| b.into_bytes().to_vec()); + Ok(Some((bytes, tag))) + } + Err(e) => Err(e), + } + }) { + Ok(Some((Ok(bytes), tag))) => Ok(Fetched::New { bytes, tag }), + Ok(Some((Err(e), _))) => Err(Error::Backend(format!("s3 get body: {e}"))), + Ok(None) => unreachable!(), + Err(e) => { + // 304 => Unchanged; NoSuchKey/404 => Absent; else Backend + let raw = aws_sdk_s3::error::ProvideErrorMetadata::code(&e).unwrap_or_default().to_string(); + let http = e.raw_response().map(|r| r.status().as_u16()); + match (raw.as_str(), http) { + (_, Some(304)) => Ok(Fetched::Unchanged), + ("NoSuchKey", _) | (_, Some(404)) => Ok(Fetched::Absent), + _ => Err(Error::Backend(format!("s3 get {k}: {e}"))), + } + } + } + } + + fn put_new(&self, key: &str, bytes: &[u8]) -> Result { + let k = self.full_key(key)?; + let req = self.client.put_object().bucket(&self.bucket).key(&k) + .if_none_match("*") + .body(bytes.to_vec().into()); + match self.rt.block_on(req.send()) { + Ok(_) => Ok(true), + Err(e) if e.raw_response().map(|r| r.status().as_u16()) == Some(412) => Ok(false), + Err(e) => Err(Error::Backend(format!("s3 put_new {k}: {e}"))), + } + } + + fn put_if_tag(&self, key: &str, bytes: &[u8], expected_tag: Option<&str>) -> Result> { + let k = self.full_key(key)?; + let mut req = self.client.put_object().bucket(&self.bucket).key(&k).body(bytes.to_vec().into()); + req = match expected_tag { + Some(tag) => req.if_match(tag), + None => req.if_none_match("*"), + }; + match self.rt.block_on(req.send()) { + Ok(out) => Ok(Some(out.e_tag().unwrap_or_default().to_string())), + Err(e) if e.raw_response().map(|r| r.status().as_u16()) == Some(412) => Ok(None), + Err(e) => Err(Error::Backend(format!("s3 put_if_tag {k}: {e}"))), + } + } + + fn list(&self, prefix: &str) -> Result> { + let full = self.full_key(prefix.trim_end_matches('/'))?; + let mut out = Vec::new(); + let mut cont: Option = None; + loop { + let mut req = self.client.list_objects_v2().bucket(&self.bucket).prefix(format!("{full}/")); + if let Some(c) = &cont { + req = req.continuation_token(c); + } + let resp = self.rt.block_on(req.send()).map_err(|e| Error::Backend(format!("s3 list: {e}")))?; + for obj in resp.contents() { + if let Some(k) = obj.key() { + let rel = k.strip_prefix(&self.prefix).unwrap_or(k).trim_start_matches('/'); + out.push(rel.to_string()); + } + } + match resp.next_continuation_token() { + Some(c) => cont = Some(c.to_string()), + None => break, + } + } + out.sort(); + Ok(out) + } +} +``` +Adapt method/type names to the SDK the compiler presents (e.g. the exact error-inspection API); the **semantics table is the contract**: 304→`Unchanged`, 404/NoSuchKey→`Absent`, 412→`put_new false` / `put_if_tag None`. Important: S3 ETags for plain PUTs are quoted MD5 strings — treat them as opaque tags (never parse), which the trait already enforces. + +- [ ] **Step 5: Verify compile + contract** + +Run: `cargo test -p scl-objio` (compiles, s3 test self-skips). +If MinIO is available locally: `docker run -d --rm -p 9000:9000 -e MINIO_ROOT_USER=sc -e MINIO_ROOT_PASSWORD=scsecret1 --name scminio minio/minio server /data`, create a bucket `scl-test` with `mc`, then +`AWS_ACCESS_KEY_ID=sc AWS_SECRET_ACCESS_KEY=scsecret1 AWS_REGION=us-east-1 AWS_ENDPOINT_URL_S3=http://127.0.0.1:9000 SC_OBJIO_S3_BUCKET=scl-test cargo test -p scl-objio s3_bucket_passes_contract -- --nocapture` — Expected: PASS. Not required for the task to land (CI stays DirBucket-only). + +- [ ] **Step 6: Commit** + +```bash +git add Cargo.lock crates/objio +git commit -m "feat(objio): S3-compatible Bucket backend, env-gated live contract test (P36a)" +``` + +--- + +### Task 3: `walfmt` — manifest + log-entry codec in `repo` + +**Files:** +- Modify: `crates/repo/Cargo.toml` (add `scl-objio = { version = "0.1.0", path = "../objio" }` — same shape as the `scl-tlsio` line at `crates/repo/Cargo.toml:17`) +- Modify: `crates/repo/src/lib.rs` (add `pub mod walfmt;` and `pub mod bucket_transport;` placeholder comes in Task 4 — here only `walfmt`) +- Modify: `crates/repo/src/error.rs` (two new variants) +- Create: `crates/repo/src/walfmt.rs` +- Test: `#[cfg(test)] mod tests` in `walfmt.rs` + +**Interfaces:** +- Consumes: `scl_core::{ObjectId, MAX_OBJECT_SIZE}`. +- Produces (used by Tasks 4, 5): + - `pub struct Manifest { pub head_seq: u64, pub checkpoint_seq: u64, pub head_branch: String }` with `pub fn encode(&self) -> Vec` / `pub fn decode(bytes: &[u8]) -> Result` + - `pub struct RefUpdate { pub branch: String, pub old: Option, pub new: ObjectId }` + - `pub struct LogEntry { pub seq: u64, pub parent_seq: u64, pub packs: Vec, pub updates: Vec }` with `encode`/`decode` same shape + - `pub fn log_key(seq: u64) -> String` → `log/`; `pub fn pack_key(hash: &str) -> String` → `packs/.pack`; `pub fn idx_key(hash: &str) -> String` → `packs/.idx` + - New error variants: `Error::Wal(String)` (`#[error("bucket wal: {0}")]`) and `Error::ObjIo(#[from] scl_objio::Error)` (`#[error("bucket: {0}")]`) + +- [ ] **Step 1: Write the failing tests** + +```rust +#[cfg(test)] +mod tests { + use super::*; + use scl_core::ObjectId; + + fn some_id(byte: u8) -> ObjectId { + // Any real id: hash a one-byte payload. Use whatever constructor + // refs::read_branch_tip uses to parse hex tips (check refs.rs:35 and + // reuse the identical call) — or simplest: ObjectId::of(&[byte]). + ObjectId::of(&[byte]) + } + + #[test] + fn manifest_round_trips_and_rejects_garbage() { + let m = Manifest { head_seq: 7, checkpoint_seq: 0, head_branch: "main".into() }; + let bytes = m.encode(); + let back = Manifest::decode(&bytes).unwrap(); + assert_eq!(back.head_seq, 7); + assert_eq!(back.checkpoint_seq, 0); + assert_eq!(back.head_branch, "main"); + // wrong magic, truncated, future version, trailing junk: all refused + assert!(Manifest::decode(b"XXXX").is_err()); + assert!(Manifest::decode(&bytes[..bytes.len() - 1]).is_err()); + let mut future = bytes.clone(); + future[4] = 0xFF; // bump version byte + assert!(Manifest::decode(&future).is_err()); + let mut junk = bytes.clone(); + junk.push(0); + assert!(Manifest::decode(&junk).is_err()); + } + + #[test] + fn log_entry_round_trips_with_and_without_old_tips() { + let e = LogEntry { + seq: 3, + parent_seq: 2, + packs: vec!["ab12".into()], + updates: vec![ + RefUpdate { branch: "main".into(), old: Some(some_id(1)), new: some_id(2) }, + RefUpdate { branch: "feat".into(), old: None, new: some_id(3) }, + ], + }; + let back = LogEntry::decode(&e.encode()).unwrap(); + assert_eq!(back.seq, 3); + assert_eq!(back.parent_seq, 2); + assert_eq!(back.packs, vec!["ab12".to_string()]); + assert_eq!(back.updates.len(), 2); + assert_eq!(back.updates[0].old, Some(some_id(1))); + assert_eq!(back.updates[1].old, None); + assert_eq!(back.updates[1].new, some_id(3)); + } + + #[test] + fn decode_caps_hostile_lengths() { + // a length prefix claiming 1 GiB must fail fast, not allocate + let mut evil = Manifest { head_seq: 1, checkpoint_seq: 0, head_branch: "m".into() }.encode(); + let n = evil.len(); + evil[n - 2..].copy_from_slice(&[0xFF, 0xFF]); // corrupt branch length tail + assert!(Manifest::decode(&evil).is_err()); + } + + #[test] + fn keys_are_stable() { + assert_eq!(log_key(7), "log/00000000000000000007"); + assert_eq!(pack_key("abcd"), "packs/abcd.pack"); + assert_eq!(idx_key("abcd"), "packs/abcd.idx"); + } +} +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test -p scl-repo walfmt` +Expected: compile FAIL — module/types not found. + +- [ ] **Step 3: Implement the codec** + +`crates/repo/src/walfmt.rs`. Encoding rules (all little-endian, strict decode = every length bounds-checked against remaining input, entire buffer must be consumed): + +- Manifest: `b"SCWM"` + `u32 version=1` + `u64 head_seq` + `u64 checkpoint_seq` + `u32 branch_len` + branch UTF-8. `branch_len` cap 4096. +- LogEntry: `b"SCWE"` + `u32 version=1` + `u64 seq` + `u64 parent_seq` + `u32 npacks` (cap 65536) + per pack (`u32 len` cap 128 + ASCII-hex string) + `u32 nupdates` (cap 65536) + per update (`u32 branch_len` cap 4096 + branch + `u8 has_old` (0/1 only) + optional 32 raw old bytes + 32 raw new bytes). + +```rust +//! On-bucket WAL encoding (P36a). Versioned, strict, fail-closed: readers +//! refuse unknown versions and any length that overruns the buffer. + +use crate::error::{Error, Result}; +use scl_core::ObjectId; + +const MANIFEST_MAGIC: &[u8; 4] = b"SCWM"; +const ENTRY_MAGIC: &[u8; 4] = b"SCWE"; +const VERSION: u32 = 1; +const MAX_NAME: usize = 4096; +const MAX_LIST: usize = 65536; + +struct Cursor<'a> { + buf: &'a [u8], + at: usize, +} +impl<'a> Cursor<'a> { + fn take(&mut self, n: usize) -> Result<&'a [u8]> { + if self.buf.len() - self.at < n { + return Err(Error::Wal(format!("truncated at byte {}", self.at))); + } + let s = &self.buf[self.at..self.at + n]; + self.at += n; + Ok(s) + } + fn u32(&mut self) -> Result { + Ok(u32::from_le_bytes(self.take(4)?.try_into().unwrap())) + } + fn u64(&mut self) -> Result { + Ok(u64::from_le_bytes(self.take(8)?.try_into().unwrap())) + } + fn string(&mut self, cap: usize) -> Result { + let n = self.u32()? as usize; + if n > cap { + return Err(Error::Wal(format!("length {n} exceeds cap {cap}"))); + } + String::from_utf8(self.take(n)?.to_vec()).map_err(|_| Error::Wal("non-utf8 name".into())) + } + fn id(&mut self) -> Result { + let raw: [u8; 32] = self.take(32)?.try_into().unwrap(); + Ok(ObjectId::from_bytes(raw)) // ← if no such constructor exists, check + // how scl-core builds an ObjectId from raw digest bytes (grep + // `impl ObjectId` in crates/core) and use that; ids are 32 raw bytes + // on the wire here, never hex. + } + fn done(&self) -> Result<()> { + if self.at != self.buf.len() { + return Err(Error::Wal(format!("{} trailing bytes", self.buf.len() - self.at))); + } + Ok(()) + } +} + +fn header<'a>(bytes: &'a [u8], magic: &[u8; 4], what: &str) -> Result> { + let mut c = Cursor { buf: bytes, at: 0 }; + if c.take(4)? != magic { + return Err(Error::Wal(format!("not a {what} (bad magic)"))); + } + let v = c.u32()?; + if v != VERSION { + return Err(Error::Wal(format!("{what} version {v} not supported (this build speaks {VERSION})"))); + } + Ok(c) +} +``` +then `Manifest`/`RefUpdate`/`LogEntry` structs with `encode` (mirror writes: `extend_from_slice(magic)`, `to_le_bytes`, …) and `decode` using the cursor, ending with `c.done()?`. Key helpers: +```rust +/// `log/` zero-padded so lexical order == numeric order. +pub fn log_key(seq: u64) -> String { + format!("log/{seq:020}") +} +pub fn pack_key(hash: &str) -> String { + format!("packs/{hash}.pack") +} +pub fn idx_key(hash: &str) -> String { + format!("packs/{hash}.idx") +} +``` +Error variants appended at the **end** of the enum in `crates/repo/src/error.rs` (house pattern — recent variants last, with rationale docs): +```rust + /// P36a: the bucket WAL is untrusted input; decode/consistency failures + /// are their own variant so callers can distinguish "bucket corrupt or + /// newer-format" from transport errors. + #[error("bucket wal: {0}")] + Wal(String), + #[error("bucket: {0}")] + ObjIo(#[from] scl_objio::Error), +``` + +- [ ] **Step 4: Run tests** + +Run: `cargo test -p scl-repo walfmt` +Expected: PASS (4 tests). If `ObjectId::from_bytes`/`ObjectId::of` names differ, fix to the real constructors found in `crates/core` — the test and impl must use the same ones. + +- [ ] **Step 5: Commit** + +```bash +git add Cargo.lock crates/repo +git commit -m "feat(repo): walfmt manifest/log-entry codec, strict versioned decode (P36a)" +``` + +--- + +### Task 4: `BucketTransport` read half — WAL view, refs, objects, `get_pack` + +**Files:** +- Create: `crates/repo/src/bucket_transport.rs` +- Modify: `crates/repo/src/lib.rs` (`pub mod bucket_transport;`) +- Test: `#[cfg(test)] mod tests` in `bucket_transport.rs` + +**Interfaces:** +- Consumes: Task 1 trait (`scl_objio::{Bucket, DirBucket, Fetched}`), Task 3 codec, `scl_core::pack::{parse_index, read_object_at, PackWriter, IndexEntry}`, `crate::reachable::{ObjectSource, reachable_objects}`, `scl_core::{Object, ObjectId, MAX_OBJECT_SIZE}`. +- Produces (used by Tasks 5, 6): + - `pub struct BucketTransport` with `pub fn from_bucket(bucket: Box) -> Result` (Task 6 adds `open(url)`) + - internal `fn refresh(&self) -> Result<()>`, `fn view(&self) -> Ref<'_, Option>`, `fn object_bytes(&self, id: &ObjectId) -> Result>` + - read-half `Transport` methods compile (write half `todo!()`-free: Task 5 fills them — until then they return `Err(Error::Wal("write half lands in Task 5".into()))` so the crate stays warning-clean and honest) + +- [ ] **Step 1: Write the failing test — seed a bucket by hand, read it back** + +The test builds a tiny WAL directly with `walfmt` + `DirBucket` (no write half yet), packing objects from a scratch persistent `Store`: + +```rust +#[cfg(test)] +mod tests { + use super::*; + use crate::transport::Transport; + use crate::walfmt::{idx_key, log_key, pack_key, LogEntry, Manifest, RefUpdate}; + use scl_core::{Object, ObjectId}; + use scl_objio::{Bucket, DirBucket}; + + /// Build pack+idx bytes for the given objects (reuses core's builder). + fn pack_of(objects: &[(ObjectId, Vec)]) -> (String, Vec, Vec) { + let (pack, idx) = scl_core::pack::build_pack(objects).unwrap(); + let hash = hex::encode(blake3::hash(&pack).as_bytes()); + (hash, pack, idx) + } + + /// A minimal one-commit object set: blob → tree → snapshot, exactly as a + /// real repo would store them. Reuse the object constructors the repo + /// crate already uses in its own tests (see sync.rs tests for the + /// canonical way to mint a commit); returns (tip, objects). + fn tiny_history() -> (ObjectId, Vec<(ObjectId, Vec)>) { + let root = std::env::temp_dir().join(format!("scl-bt-hist-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&root); + std::fs::create_dir_all(&root).unwrap(); + let repo = crate::repo::Repo::init(&root).unwrap(); + std::fs::write(root.join("f.txt"), b"hello wal").unwrap(); + let tip = repo.commit("t", "c1").unwrap(); + let store_arc = repo.vfs().store(); + let mut store = store_arc.lock().unwrap(); + let ids = crate::reachable::reachable_objects(&mut *store, &[tip]).unwrap(); + let objects = ids.iter().map(|id| (*id, store.get(id).unwrap().encode())).collect(); + drop(store); + drop(repo); + std::fs::remove_dir_all(&root).unwrap(); + (tip, objects) + } + + #[test] + fn reads_refs_objects_and_packs_from_a_hand_built_wal() { + let broot = std::env::temp_dir().join(format!("scl-bt-read-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&broot); + let bucket = DirBucket::open(&broot).unwrap(); + + let (tip, objects) = tiny_history(); + let (hash, pack, idx) = pack_of(&objects); + assert!(bucket.put_new(&pack_key(&hash), &pack).unwrap()); + assert!(bucket.put_new(&idx_key(&hash), &idx).unwrap()); + let entry = LogEntry { + seq: 1, + parent_seq: 0, + packs: vec![hash.clone()], + updates: vec![RefUpdate { branch: "main".into(), old: None, new: tip }], + }; + assert!(bucket.put_new(&log_key(1), &entry.encode()).unwrap()); + let m = Manifest { head_seq: 1, checkpoint_seq: 0, head_branch: "main".into() }; + bucket.put_if_tag("manifest", &m.encode(), None).unwrap().unwrap(); + + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + assert_eq!(t.list_refs().unwrap(), vec![("main".to_string(), tip)]); + assert_eq!(t.head_branch().unwrap(), "main"); + assert!(t.has_object(&tip).unwrap()); + let bytes = t.get_object(&tip).unwrap(); + assert_eq!(ObjectId::of(&bytes), tip); + // get_pack: full closure with no haves reproduces every object + let mut out = Vec::new(); + t.get_pack(&[tip], &[], None, &mut out).unwrap(); + let got = scl_core::pack::parse_pack(&out).unwrap(); + assert_eq!(got.len(), objects.len()); + // filter is refused loudly, not ignored + let filt = vec!["src/".to_string()]; + assert!(t.get_pack(&[tip], &[], Some(&filt), &mut Vec::new()).is_err()); + drop(t); + std::fs::remove_dir_all(&broot).unwrap(); + } + + #[test] + fn empty_bucket_lists_no_refs_and_head_branch_errors() { + let broot = std::env::temp_dir().join(format!("scl-bt-empty-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&broot); + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + assert_eq!(t.list_refs().unwrap(), Vec::<(String, ObjectId)>::new()); + assert!(t.head_branch().is_err()); + drop(t); + std::fs::remove_dir_all(&broot).unwrap(); + } + + #[test] + fn off_chain_log_entries_are_ignored() { + // manifest head=1; a stray log/2 (orphan from a crashed/losing pusher) + // must not affect refs. + let broot = std::env::temp_dir().join(format!("scl-bt-orphan-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&broot); + let bucket = DirBucket::open(&broot).unwrap(); + let (tip, objects) = tiny_history(); + let (hash, pack, idx) = pack_of(&objects); + bucket.put_new(&pack_key(&hash), &pack).unwrap(); + bucket.put_new(&idx_key(&hash), &idx).unwrap(); + let e1 = LogEntry { seq: 1, parent_seq: 0, packs: vec![hash], + updates: vec![RefUpdate { branch: "main".into(), old: None, new: tip }] }; + bucket.put_new(&log_key(1), &e1.encode()).unwrap(); + let orphan = LogEntry { seq: 2, parent_seq: 1, packs: vec![], + updates: vec![RefUpdate { branch: "evil".into(), old: None, new: tip }] }; + bucket.put_new(&log_key(2), &orphan.encode()).unwrap(); + let m = Manifest { head_seq: 1, checkpoint_seq: 0, head_branch: "main".into() }; + bucket.put_if_tag("manifest", &m.encode(), None).unwrap().unwrap(); + + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + assert_eq!(t.list_refs().unwrap(), vec![("main".to_string(), tip)]); + drop(t); + std::fs::remove_dir_all(&broot).unwrap(); + } +} +``` +(`hex` and `blake3` are already deps of `scl-repo` — see `crates/repo/Cargo.toml:18-19`.) + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test -p scl-repo bucket_transport` +Expected: compile FAIL — `BucketTransport` not found. + +- [ ] **Step 3: Implement the read half** + +`crates/repo/src/bucket_transport.rs`: + +```rust +//! A [`Transport`] over an object-store bucket (P36a): immutable packs + +//! parent-linked log entries, one CAS'd manifest as the sole commit point. +//! See ADR-0046 and docs/superpowers/specs/2026-08-26-wal-bucket-backend-design.md. + +use crate::error::{Error, Result}; +use crate::transport::Transport; +use crate::walfmt::{idx_key, log_key, pack_key, LogEntry, Manifest, RefUpdate}; +use scl_core::pack::{parse_index, read_object_at, IndexEntry, PackWriter}; +use scl_core::{Object, ObjectId}; +use scl_objio::{Bucket, Fetched}; +use std::cell::RefCell; +use std::collections::BTreeMap; +use std::io::Write; + +/// Reconstructed state of the WAL at one manifest tag. +struct WalView { + tag: String, + manifest: Manifest, + /// branch -> tip, after replaying the parent chain oldest-first. + refs: BTreeMap, + /// object id -> (pack hash, offset, length), from every on-chain pack's idx. + index: BTreeMap, +} + +pub struct BucketTransport { + bucket: Box, + view: RefCell>, + /// Body bytes of the most recently used pack (walk locality). + pack_cache: RefCell)>>, + // Task 5 adds: staged objects + pending pack hashes. +} + +/// Untrusted-length guard (P28 parity): refuse any WAL metadata value — +/// manifest, log entry, idx — larger than MAX_OBJECT_SIZE before decoding. +/// (Pack bodies may legitimately exceed it; their per-record lengths are +/// capped inside `parse_pack_reader`/`read_object_at` in core.) +fn capped(what: &str, bytes: Vec) -> Result> { + if bytes.len() > scl_core::MAX_OBJECT_SIZE { + return Err(Error::Wal(format!("{what} exceeds MAX_OBJECT_SIZE (256 MiB)"))); + } + Ok(bytes) +} + +impl BucketTransport { + pub fn from_bucket(bucket: Box) -> Result { + let t = BucketTransport { + bucket, + view: RefCell::new(None), + pack_cache: RefCell::new(None), + }; + t.refresh()?; + Ok(t) + } + + /// One conditional GET of the manifest; on change, rebuild refs + index + /// by walking parent links head -> 0 (seq numbers are claims; the chain + /// is the truth — off-chain entries are garbage). + fn refresh(&self) -> Result<()> { + let cached_tag = self.view.borrow().as_ref().map(|v| v.tag.clone()); + match self.bucket.get("manifest", cached_tag.as_deref())? { + Fetched::Unchanged => Ok(()), + Fetched::Absent => { + *self.view.borrow_mut() = None; + Ok(()) + } + Fetched::New { bytes, tag } => { + let bytes = capped("manifest", bytes)?; + let manifest = Manifest::decode(&bytes)?; + let mut entries = Vec::new(); + let mut seq = manifest.head_seq; + while seq != 0 { + let Fetched::New { bytes, .. } = self.bucket.get(&log_key(seq), None)? else { + return Err(Error::Wal(format!("log entry {seq} referenced by chain but absent"))); + }; + let e = LogEntry::decode(&capped("log entry", bytes)?)?; + if e.seq != seq { + return Err(Error::Wal(format!("log entry at {} claims seq {}", seq, e.seq))); + } + seq = e.parent_seq; + if e.parent_seq >= e.seq { + return Err(Error::Wal(format!("log entry {} has non-decreasing parent {}", e.seq, e.parent_seq))); + } + entries.push(e); + } + entries.reverse(); // oldest first + let mut refs = BTreeMap::new(); + let mut index = BTreeMap::new(); + for e in &entries { + for u in &e.updates { + crate::refs::validate_incoming_branch(&u.branch)?; // see note below + refs.insert(u.branch.clone(), u.new); + } + for hash in &e.packs { + let Fetched::New { bytes, .. } = self.bucket.get(&idx_key(hash), None)? else { + return Err(Error::Wal(format!("pack {hash} on chain but idx absent"))); + }; + let bytes = capped("pack idx", bytes)?; + for IndexEntry { id, offset, length } in parse_index(&bytes)? { + index.insert(id, (hash.clone(), offset, length)); + } + } + } + *self.view.borrow_mut() = Some(WalView { tag, manifest, refs, index }); + Ok(()) + } + } + } + + /// Canonical bytes of one object, via its pack (downloaded + cached). + fn object_bytes(&self, id: &ObjectId) -> Result> { + let (hash, offset, _len) = { + let view = self.view.borrow(); + let view = view.as_ref().ok_or_else(|| Error::Wal("bucket remote is empty".into()))?; + view.index.get(id).cloned().ok_or(Error::CorruptObject(*id))? + }; + let mut cache = self.pack_cache.borrow_mut(); + if cache.as_ref().map(|(h, _)| h.as_str()) != Some(hash.as_str()) { + let Fetched::New { bytes, .. } = self.bucket.get(&pack_key(&hash), None)? else { + return Err(Error::Wal(format!("pack {hash} on chain but body absent"))); + }; + *cache = Some((hash.clone(), bytes)); + } + let (_, pack) = cache.as_ref().unwrap(); + Ok(read_object_at(pack, offset, id)?.encode()) + } +} + +/// `ObjectSource` over the bucket for reachability walks. +struct BucketSource<'a>(&'a BucketTransport); +impl crate::reachable::ObjectSource for BucketSource<'_> { + fn get(&mut self, id: &ObjectId) -> Result { + let bytes = self.0.object_bytes(id)?; + Object::decode(&bytes).map_err(Into::into) + } +} +``` + +Two adaptation notes for the implementer: +- `validate_incoming_branch`: `validate_branch_name` is `pub(crate)` at `crates/repo/src/repo.rs:1913` and re-exported crate-internally at `refs.rs:13` — call it as `crate::refs::validate_branch_name(&u.branch)?` (same-crate, so visibility is fine); the name in the sketch is a placeholder for exactly that call. +- If `read_object_at`'s `length` field or `Object::decode`'s error type differ in detail, follow the real signatures in `crates/core/src/pack.rs:202` and the object codec — the test pins behavior. + +`Transport` impl, read methods (write half stubs return `Err(Error::Wal("bucket write half lands in Task 5".into()))` for now): +```rust +impl Transport for BucketTransport { + fn list_refs(&self) -> Result> { + self.refresh()?; + Ok(self.view.borrow().as_ref() + .map(|v| v.refs.iter().map(|(b, id)| (b.clone(), *id)).collect()) + .unwrap_or_default()) + } + + fn head_branch(&self) -> Result { + self.refresh()?; + self.view.borrow().as_ref() + .map(|v| v.manifest.head_branch.clone()) + .ok_or_else(|| Error::Remote("bucket remote is empty (no manifest)".into())) + } + + fn has_object(&self, id: &ObjectId) -> Result { + self.refresh()?; + Ok(self.view.borrow().as_ref().is_some_and(|v| v.index.contains_key(id))) + } + + fn get_object(&self, id: &ObjectId) -> Result> { + self.refresh()?; + self.object_bytes(id) + } + + fn get_pack(&self, wants: &[ObjectId], haves: &[ObjectId], filter: Option<&[String]>, out: &mut dyn Write) -> Result<()> { + if filter.is_some() { + return Err(Error::InvalidArgument( + "partial clone from bucket remotes is not supported yet; clone via a served remote".into(), + )); + } + self.refresh()?; + let mut src = BucketSource(self); + // haves the bucket doesn't know can't shrink the pack — skip them. + let known_haves: Vec = { + let view = self.view.borrow(); + haves.iter().copied() + .filter(|h| view.as_ref().is_some_and(|v| v.index.contains_key(h))) + .collect() + }; + let have_set = crate::reachable::reachable_objects(&mut src, &known_haves)?; + let want_set = crate::reachable::reachable_objects(&mut src, wants)?; + let ids: Vec = want_set.difference(&have_set).copied().collect(); + let mut writer = PackWriter::new(out, ids.len() as u32)?; + for id in &ids { + let bytes = self.object_bytes(id)?; + writer.write_object(id, &bytes)?; + } + writer.finish()?; // idx discarded — transfer needs the body only + Ok(()) + } + + fn put_object(&self, _id: &ObjectId, _bytes: &[u8]) -> Result<()> { + Err(Error::Wal("bucket write half lands in Task 5".into())) + } + fn update_ref(&self, _branch: &str, _id: &ObjectId, _expected_old: Option<&ObjectId>) -> Result<()> { + Err(Error::Wal("bucket write half lands in Task 5".into())) + } + fn put_pack(&self, _src: &mut dyn std::io::Read) -> Result> { + Err(Error::Wal("bucket write half lands in Task 5".into())) + } +} +``` +Add `pub mod bucket_transport;` to `crates/repo/src/lib.rs` alongside the other `pub mod` transport lines. + +- [ ] **Step 4: Run tests** + +Run: `cargo test -p scl-repo bucket_transport` +Expected: PASS (3 tests). + +- [ ] **Step 5: Commit** + +```bash +git add crates/repo +git commit -m "feat(repo): BucketTransport read half — WAL view, refs, objects, get_pack (P36a)" +``` + +--- + +### Task 5: `BucketTransport` write half — staging, `put_pack`, CAS `update_ref` + +**Files:** +- Modify: `crates/repo/src/bucket_transport.rs` +- Test: extend `#[cfg(test)] mod tests` there + +**Interfaces:** +- Consumes: Task 4's struct + view; `scl_core::pack::{build_pack, parse_pack_reader}`; `scl_objio::Bucket::{put_new, put_if_tag}`; `crate::walfmt`. +- Produces (used by Tasks 6, 7): the complete `Transport` impl. Commit protocol (fixed contract): flush staged objects → upload `packs/.pack`+`.idx` (`put_new`, exists = dedup success) → write `log/` (`put_new`, taken seq → next) → CAS `manifest` (`put_if_tag`). On CAS loss: if this branch's tip moved off `expected_old` → `Error::NonFastForward`; otherwise re-read and retry (fresh log entry, correct parent; the old entry becomes off-chain garbage). `MAX_CAS_RETRIES: u32 = 16`, exhaustion → `Error::Remote("manifest cas contention: gave up after 16 attempts")`. + +- [ ] **Step 1: Write the failing tests** + +```rust + #[test] + fn push_via_trait_round_trips_into_a_fresh_bucket() { + let broot = std::env::temp_dir().join(format!("scl-bt-write-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&broot); + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + + let (tip, objects) = tiny_history(); + // exactly what sync::push does: pack, then CAS'd ref update + let (pack, _idx) = scl_core::pack::build_pack(&objects).unwrap(); + let ids = t.put_pack(&mut std::io::Cursor::new(pack)).unwrap(); + assert_eq!(ids.len(), objects.len()); + t.update_ref("main", &tip, None).unwrap(); + + // a second transport sees it + let t2 = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + assert_eq!(t2.list_refs().unwrap(), vec![("main".to_string(), tip)]); + assert_eq!(t2.head_branch().unwrap(), "main"); + assert!(t2.has_object(&tip).unwrap()); + drop((t, t2)); + std::fs::remove_dir_all(&broot).unwrap(); + } + + #[test] + fn update_ref_honors_expected_old_semantics() { + let broot = std::env::temp_dir().join(format!("scl-bt-cas-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&broot); + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + let (tip, objects) = tiny_history(); + let (pack, _) = scl_core::pack::build_pack(&objects).unwrap(); + t.put_pack(&mut std::io::Cursor::new(pack)).unwrap(); + t.update_ref("main", &tip, None).unwrap(); + // stale expected_old (None while the branch exists) => NonFastForward + let other = ObjectId::of(b"not the tip"); + assert!(matches!(t.update_ref("main", &other, None), Err(Error::NonFastForward))); + // setting to the value it already has succeeds regardless of expected_old (trait doc) + t.update_ref("main", &tip, None).unwrap(); + t.update_ref("main", &tip, Some(&other)).unwrap(); + drop(t); + std::fs::remove_dir_all(&broot).unwrap(); + } + + #[test] + fn put_object_stages_and_update_ref_commits_them() { + let broot = std::env::temp_dir().join(format!("scl-bt-stage-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&broot); + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + let (tip, objects) = tiny_history(); + for (id, bytes) in &objects { + t.put_object(id, bytes).unwrap(); + } + // corrupt bytes are rejected at staging time + assert!(t.put_object(&tip, b"garbage").is_err()); + t.update_ref("main", &tip, None).unwrap(); + let t2 = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + assert!(t2.has_object(&tip).unwrap()); + drop((t, t2)); + std::fs::remove_dir_all(&broot).unwrap(); + } +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test -p scl-repo bucket_transport` +Expected: the three new tests FAIL with `Error::Wal("bucket write half lands in Task 5")`. + +- [ ] **Step 3: Implement** + +Add fields to the struct: +```rust + /// Objects staged by `put_object`, flushed into a pack at `update_ref`. + staged: RefCell)>>, + /// Hashes of packs uploaded (put_pack / flushed staging) awaiting a + /// manifest commit that references them. + pending_packs: RefCell>, +``` +Upload helper + trait methods: +```rust +impl BucketTransport { + /// Upload one pack (+idx) content-addressed by BLAKE3 of the pack bytes. + /// Already-present keys are success (identical content — dedup). + fn upload_pack(&self, objects: &[(ObjectId, Vec)]) -> Result<(String, Vec)> { + let (pack, idx) = scl_core::pack::build_pack(objects)?; + let hash = hex::encode(blake3::hash(&pack).as_bytes()); + self.bucket.put_new(&pack_key(&hash), &pack)?; + self.bucket.put_new(&idx_key(&hash), &idx)?; + Ok((hash, objects.iter().map(|(id, _)| *id).collect())) + } + + fn flush_staged(&self) -> Result<()> { + let staged = std::mem::take(&mut *self.staged.borrow_mut()); + if staged.is_empty() { + return Ok(()); + } + let (hash, _) = self.upload_pack(&staged)?; + self.pending_packs.borrow_mut().push(hash); + Ok(()) + } +} +``` +```rust + fn put_object(&self, id: &ObjectId, bytes: &[u8]) -> Result<()> { + if ObjectId::of(bytes) != *id { + return Err(Error::CorruptObject(*id)); + } + self.staged.borrow_mut().push((*id, bytes.to_vec())); + Ok(()) + } + + fn put_pack(&self, src: &mut dyn std::io::Read) -> Result> { + // Spill + verify first (P25 invariant: never trust a live stream), + // then rebuild deterministically — build_pack output is byte-stable + // for the same objects in the same order (pinned in core), so the + // rebuilt pack's hash names identical content identically. + let mut objects: Vec<(ObjectId, Vec)> = Vec::new(); + scl_core::pack::parse_pack_reader(src, |id, obj| { + objects.push((id, obj.encode())); + Ok(()) + })?; + let (hash, ids) = self.upload_pack(&objects)?; + self.pending_packs.borrow_mut().push(hash); + Ok(ids) + } + + fn update_ref(&self, branch: &str, id: &ObjectId, expected_old: Option<&ObjectId>) -> Result<()> { + self.flush_staged()?; + const MAX_CAS_RETRIES: u32 = 16; + for _ in 0..MAX_CAS_RETRIES { + self.refresh()?; + let (current, prev_tag, head_seq, checkpoint_seq, head_branch) = { + let view = self.view.borrow(); + match view.as_ref() { + Some(v) => ( + v.refs.get(branch).copied(), + Some(v.tag.clone()), + v.manifest.head_seq, + v.manifest.checkpoint_seq, + v.manifest.head_branch.clone(), + ), + None => (None, None, 0, 0, branch.to_string()), + } + }; + if current.as_ref() == Some(id) { + self.pending_packs.borrow_mut().clear(); // already there — idempotent + return Ok(()); + } + if current.as_ref() != expected_old { + return Err(Error::NonFastForward); + } + // claim a seq (collisions with orphans just advance) + let entry = LogEntry { + seq: 0, // set in the claim loop + parent_seq: head_seq, + packs: self.pending_packs.borrow().clone(), + updates: vec![RefUpdate { branch: branch.to_string(), old: current, new: *id }], + }; + let mut seq = head_seq + 1; + let seq = loop { + let mut e = LogEntry { seq, ..entry.clone() }; + e.seq = seq; + if self.bucket.put_new(&log_key(seq), &e.encode())? { + break seq; + } + seq += 1; + }; + let manifest = Manifest { head_seq: seq, checkpoint_seq, head_branch }; + if let Some(tag) = self.bucket.put_if_tag("manifest", &manifest.encode(), prev_tag.as_deref())? { + // committed: refresh local view cheaply and clear pendings + self.pending_packs.borrow_mut().clear(); + let _ = tag; // next refresh() re-reads; keeping it simple + self.refresh()?; + return Ok(()); + } + // lost the CAS: our log entry is now off-chain garbage; loop + // re-reads and either detects a moved tip (NonFastForward above) + // or retries with a fresh entry under the new parent. + } + Err(Error::Remote("manifest cas contention: gave up after 16 attempts".into())) + } +``` +(`LogEntry` needs `Clone` — add `#[derive(Clone)]` to it and `RefUpdate` in `walfmt.rs`.) + +- [ ] **Step 4: Run tests** + +Run: `cargo test -p scl-repo bucket_transport` +Expected: PASS (all 6). + +- [ ] **Step 5: Commit** + +```bash +git add crates/repo +git commit -m "feat(repo): BucketTransport write half — staged packs + CAS'd manifest commit (P36a)" +``` + +--- + +### Task 6: URL schemes, `remote add` validation, end-to-end sync tests + +**Files:** +- Modify: `crates/repo/src/bucket_transport.rs` (add `BucketUrl` + `BucketTransport::open`) +- Modify: `crates/repo/src/stdio_transport.rs:281-293` (`open_transport` dispatch) +- Modify: `crates/repo/src/lib.rs` (re-export `BucketTransport`, `BucketUrl`) +- Modify: `crates/cli/src/main.rs:3646-3652` (`run_remote` add-time validation) +- Test: extend `bucket_transport.rs` tests + +**Interfaces:** +- Consumes: Tasks 1–5; `Repo::{clone_url, fetch, push, remote_add}` (unchanged). +- Produces: `pub struct BucketUrl { pub scheme: BucketScheme, pub bucket: String, pub prefix: String }`, `pub enum BucketScheme { Wal, S3 }`, `BucketUrl::parse(url: &str) -> Result`, `BucketTransport::open(url: &str) -> Result`. URL grammar: `sc+wal:///abs/path` or `sc+wal://rel/path` (everything after the scheme is the directory path), `sc+s3:///` (first component bucket name, rest prefix; empty prefix allowed). + +- [ ] **Step 1: Write the failing tests** + +```rust + #[test] + fn clone_push_fetch_round_trip_over_sc_wal_url() { + let pid = std::process::id(); + let broot = std::env::temp_dir().join(format!("scl-bt-e2e-bucket-{pid}")); + let a_root = std::env::temp_dir().join(format!("scl-bt-e2e-a-{pid}")); + let b_root = std::env::temp_dir().join(format!("scl-bt-e2e-b-{pid}")); + for d in [&broot, &a_root, &b_root] { + let _ = std::fs::remove_dir_all(d); + } + std::fs::create_dir_all(&a_root).unwrap(); + let url = format!("sc+wal://{}", broot.display()); + + // A: init, commit, add bucket remote, push (creates the bucket repo) + let a = crate::repo::Repo::init(&a_root).unwrap(); + std::fs::write(a_root.join("f.txt"), b"one").unwrap(); + let tip1 = a.commit("t", "c1").unwrap(); + a.remote_add("origin", &url).unwrap(); + assert_eq!(a.push("origin").unwrap(), tip1); + + // B: clone from the bucket + let b = crate::repo::Repo::clone_url(&url, &b_root).unwrap(); + assert_eq!(b.head_tip().unwrap(), Some(tip1)); + assert_eq!(std::fs::read(b_root.join("f.txt")).unwrap(), b"one"); + + // B commits and pushes; A fetches and sees it + std::fs::write(b_root.join("g.txt"), b"two").unwrap(); + let tip2 = b.commit("t", "c2").unwrap(); + b.push("origin").unwrap(); + drop(b); + let fetched = a.fetch("origin").unwrap(); + assert!(fetched.iter().any(|(br, id)| br == "main" && *id == tip2)); + + // stale push from A (still at tip1 + its own commit) => NonFastForward + std::fs::write(a_root.join("h.txt"), b"three").unwrap(); + a.commit("t", "c3").unwrap(); + assert!(matches!(a.push("origin"), Err(Error::NonFastForward))); + drop(a); + for d in [&broot, &a_root, &b_root] { + std::fs::remove_dir_all(d).unwrap(); + } + } + + #[test] + fn bucket_url_parses_and_rejects() { + let u = BucketUrl::parse("sc+s3://mybucket/team/repo").unwrap(); + assert!(matches!(u.scheme, BucketScheme::S3)); + assert_eq!(u.bucket, "mybucket"); + assert_eq!(u.prefix, "team/repo"); + let u = BucketUrl::parse("sc+s3://mybucket").unwrap(); + assert_eq!(u.prefix, ""); + let u = BucketUrl::parse("sc+wal:///tmp/x").unwrap(); + assert!(matches!(u.scheme, BucketScheme::Wal)); + assert!(BucketUrl::parse("sc+s3://").is_err()); + assert!(BucketUrl::parse("sc+s3://b\nad/x").is_err()); + assert!(BucketUrl::parse("http://nope").is_err()); + } +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `cargo test -p scl-repo bucket_transport` +Expected: compile FAIL — `BucketUrl` not found. + +- [ ] **Step 3: Implement URL parsing + dispatch** + +In `bucket_transport.rs`: +```rust +pub enum BucketScheme { + Wal, + S3, +} + +/// `sc+wal://` (local test/demo backend) or `sc+s3:///`. +pub struct BucketUrl { + pub scheme: BucketScheme, + pub bucket: String, + pub prefix: String, +} + +impl BucketUrl { + pub fn parse(url: &str) -> Result { + let (scheme, rest) = if let Some(r) = url.strip_prefix("sc+wal://") { + (BucketScheme::Wal, r) + } else if let Some(r) = url.strip_prefix("sc+s3://") { + (BucketScheme::S3, r) + } else { + return Err(Error::InvalidArgument(format!("not an sc+wal:// or sc+s3:// url: {url}"))); + }; + if rest.is_empty() || rest.chars().any(|c| c == '\r' || c == '\n') { + return Err(Error::InvalidArgument(format!("bad bucket url: {url}"))); + } + Ok(match scheme { + BucketScheme::Wal => BucketUrl { scheme, bucket: rest.to_string(), prefix: String::new() }, + BucketScheme::S3 => { + let (bucket, prefix) = rest.split_once('/').unwrap_or((rest, "")); + if bucket.is_empty() { + return Err(Error::InvalidArgument(format!("bad bucket url: {url}"))); + } + BucketUrl { scheme, bucket: bucket.to_string(), prefix: prefix.trim_matches('/').to_string() } + } + }) + } +} + +impl BucketTransport { + /// Open the right bucket backend for a bucket URL. + pub fn open(url: &str) -> Result { + let parsed = BucketUrl::parse(url)?; + let bucket: Box = match parsed.scheme { + BucketScheme::Wal => Box::new(scl_objio::DirBucket::open(&parsed.bucket)?), + BucketScheme::S3 => Box::new(scl_objio::S3Bucket::open(&parsed.bucket, &parsed.prefix)?), + }; + BucketTransport::from_bucket(bucket) + } +} +``` +`open_transport` (`crates/repo/src/stdio_transport.rs`) gets one new arm **before** the local-path catch-all: +```rust + } else if url.starts_with("sc+wal://") || url.starts_with("sc+s3://") { + Ok(Box::new(crate::bucket_transport::BucketTransport::open(url)?)) +``` +`crates/repo/src/lib.rs` re-exports: `pub use bucket_transport::{BucketTransport, BucketUrl};` next to the existing transport re-exports. `crates/cli/src/main.rs` `run_remote` add-arm grows fail-fast validation exactly parallel to the `ssh://` line at `main.rs:3648-3650`: +```rust + if url.starts_with("ssh://") { + scl_repo::SshUrl::parse(&url)?; // fail fast on malformed URLs + } + if url.starts_with("sc+wal://") || url.starts_with("sc+s3://") { + scl_repo::BucketUrl::parse(&url)?; // fail fast on malformed URLs + } +``` + +- [ ] **Step 4: Run tests** + +Run: `cargo test -p scl-repo bucket_transport && cargo test -p scl-repo sync` +Expected: PASS, including the untouched sync suite (proves the dispatch change breaks nothing). + +- [ ] **Step 5: Commit** + +```bash +git add crates/repo crates/cli +git commit -m "feat: sc+wal:// and sc+s3:// bucket remotes wired into open_transport (P36a)" +``` + +--- + +### Task 7: Concurrency + crash-safety proof tests + +**Files:** +- Test: extend `#[cfg(test)] mod tests` in `crates/repo/src/bucket_transport.rs` + +**Interfaces:** +- Consumes: everything above; `std::thread`. +- Produces: the P36a acceptance evidence named in the spec (two-writer race, fleet hammer, crash debris). + +- [ ] **Step 1: Write the three tests (they should pass immediately if Tasks 4–6 are correct — treat any failure as a real bug, not a test to weaken)** + +```rust + #[test] + fn racing_pushes_same_branch_one_wins_one_gets_non_fast_forward() { + let pid = std::process::id(); + let broot = std::env::temp_dir().join(format!("scl-bt-race-{pid}")); + let _ = std::fs::remove_dir_all(&broot); + // seed: one commit on main + let t0 = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + let (base, objects) = tiny_history(); + let (pack, _) = scl_core::pack::build_pack(&objects).unwrap(); + t0.put_pack(&mut std::io::Cursor::new(pack)).unwrap(); + t0.update_ref("main", &base, None).unwrap(); + drop(t0); + + // two threads race an update from the same expected_old to different tips + let mk_tip = |tag: &[u8]| { + let obj = Object::blob(tag.to_vec()); // any distinct object works as a fake tip + (obj.id(), obj.encode()) + }; + let results: Vec> = std::thread::scope(|s| { + let handles: Vec<_> = [b"racer-a".as_slice(), b"racer-b".as_slice()] + .into_iter() + .map(|tag| { + let broot = broot.clone(); + s.spawn(move || { + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + let (tip, bytes) = mk_tip(tag); + t.put_object(&tip, &bytes).unwrap(); + t.update_ref("main", &tip, Some(&base)) + }) + }) + .collect(); + handles.into_iter().map(|h| h.join().unwrap()).collect() + }); + let wins = results.iter().filter(|r| r.is_ok()).count(); + let nffs = results.iter().filter(|r| matches!(r, Err(Error::NonFastForward))).count(); + assert_eq!((wins, nffs), (1, 1), "exactly one winner and one clean refusal: {results:?}"); + std::fs::remove_dir_all(&broot).unwrap(); + } + + #[test] + fn fleet_hammer_distinct_branches_all_land_without_coordinator() { + let pid = std::process::id(); + let broot = std::env::temp_dir().join(format!("scl-bt-fleet-{pid}")); + let _ = std::fs::remove_dir_all(&broot); + const N: usize = 8; + std::thread::scope(|s| { + for i in 0..N { + let broot = broot.clone(); + s.spawn(move || { + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + let obj = Object::blob(format!("agent-{i}").into_bytes()); + t.put_object(&obj.id(), &obj.encode()).unwrap(); + // distinct branches: contention is manifest-level only, so + // every one must eventually land via CAS retry. + t.update_ref(&format!("work-{i}"), &obj.id(), None).unwrap(); + }); + } + }); + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + let refs = t.list_refs().unwrap(); + assert_eq!(refs.len(), N, "all {N} agent branches present: {refs:?}"); + drop(t); + std::fs::remove_dir_all(&broot).unwrap(); + } + + #[test] + fn crash_debris_before_the_cas_is_invisible_and_later_pushes_step_over_it() { + let pid = std::process::id(); + let broot = std::env::temp_dir().join(format!("scl-bt-crash-{pid}")); + let _ = std::fs::remove_dir_all(&broot); + let bucket = DirBucket::open(&broot).unwrap(); + let t = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + let (tip, objects) = tiny_history(); + let (pack, _) = scl_core::pack::build_pack(&objects).unwrap(); + t.put_pack(&mut std::io::Cursor::new(pack)).unwrap(); + t.update_ref("main", &tip, None).unwrap(); + + // simulate a pusher that died after pack + log entry, before the CAS: + let orphan_obj = Object::blob(b"never committed".to_vec()); + let (opack, oidx) = scl_core::pack::build_pack(&[(orphan_obj.id(), orphan_obj.encode())]).unwrap(); + let ohash = hex::encode(blake3::hash(&opack).as_bytes()); + bucket.put_new(&pack_key(&ohash), &opack).unwrap(); + bucket.put_new(&idx_key(&ohash), &oidx).unwrap(); + bucket.put_new(&log_key(2), &LogEntry { + seq: 2, parent_seq: 1, packs: vec![ohash], + updates: vec![RefUpdate { branch: "doomed".into(), old: None, new: orphan_obj.id() }], + }.encode()).unwrap(); + + // invisible to readers… + let t2 = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + assert_eq!(t2.list_refs().unwrap(), vec![("main".to_string(), tip)]); + assert!(!t2.has_object(&orphan_obj.id()).unwrap()); + // …and a live push steps over the claimed seq 2 (lands at 3+) and works. + let next = Object::blob(b"after crash".to_vec()); + t2.put_object(&next.id(), &next.encode()).unwrap(); + t2.update_ref("recovered", &next.id(), None).unwrap(); + let t3 = BucketTransport::from_bucket(Box::new(DirBucket::open(&broot).unwrap())).unwrap(); + let refs = t3.list_refs().unwrap(); + assert!(refs.contains(&("recovered".to_string(), next.id()))); + assert!(!refs.iter().any(|(b, _)| b == "doomed")); + drop((t, t2, t3)); + std::fs::remove_dir_all(&broot).unwrap(); + } +``` +Adaptation note: `Object::blob(...)`/`obj.id()` — use whatever constructor core's own tests use to mint a standalone blob object (grep `Object::` in `crates/core/src/store.rs` tests); the intent is "any distinct valid object". + +- [ ] **Step 2: Run** + +Run: `cargo test -p scl-repo bucket_transport -- --nocapture` +Expected: PASS (all 3 new + all prior). If the race test deadlocks or double-commits, the bug is in `update_ref`'s retry loop — fix the code, never the assertion. + +- [ ] **Step 3: Full workspace check** + +Run: `cargo test --workspace && cargo clippy --workspace --all-targets` +Expected: PASS / no new warnings. + +- [ ] **Step 4: Commit** + +```bash +git add crates/repo +git commit -m "test(repo): bucket WAL race, fleet-hammer, and crash-debris proofs (P36a)" +``` + +--- + +### Task 8: CLI integration test + docs (CLAUDE.md, ADR-0046, ROADMAP, THREAT-MODEL) + +**Files:** +- Create: `crates/cli/tests/bucket_remote.rs` +- Create: `docs/adr/0046-wal-bucket-remotes.md` +- Modify: `CLAUDE.md` (dependency rule + quarantine + capability map) +- Modify: `ROADMAP.md` (Deferred entries) +- Modify: `docs/THREAT-MODEL.md` (bucket trust boundary) + +**Interfaces:** +- Consumes: the shipped `sc` binary behavior from Tasks 1–7. +- Produces: the user-facing and agent-facing record; nothing downstream. + +- [ ] **Step 1: Write the CLI integration test** + +`crates/cli/tests/bucket_remote.rs`, following the harness in `crates/cli/tests/ssh_remote.rs:7-25` (module doc states the division of labor: wire correctness is proven in-crate; this file proves flag/URL plumbing end-to-end through the binary): + +```rust +//! `sc+wal://` bucket remotes through the real binary. Wire/WAL correctness +//! is proven in scl-repo's bucket_transport tests; this exercises CLI +//! plumbing: remote add validation, push, clone, fetch. + +use std::path::{Path, PathBuf}; +use std::process::{Command, Output}; + +fn sc(dir: &Path, args: &[&str]) -> Output { + let mut cmd = Command::new(env!("CARGO_BIN_EXE_sc")); + cmd.args(args).current_dir(dir); + cmd.output().expect("sc runs") +} + +fn tmp(tag: &str) -> PathBuf { + let d = std::env::temp_dir().join(format!("scl-cli-bucket-{tag}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + d +} + +#[test] +fn bucket_clone_push_fetch_round_trip_and_url_validation() { + let a = tmp("a"); + let bucket = tmp("bucket"); + let url = format!("sc+wal://{}", bucket.display()); + + let out = sc(&a, &["init"]); + assert!(out.status.success(), "{out:?}"); + std::fs::write(a.join("f.txt"), b"via cli").unwrap(); + assert!(sc(&a, &["commit", "-m", "c1"]).status.success()); + // malformed bucket URL is refused at add time + let bad = sc(&a, &["remote", "add", "borig", "sc+s3://"]); + assert!(!bad.status.success()); + assert!(sc(&a, &["remote", "add", "origin", &url]).status.success()); + assert!(sc(&a, &["push", "origin"]).status.success()); + + let parent = tmp("bparent"); + let b = parent.join("b"); + let out = sc(&parent, &["clone", &url, b.to_str().unwrap()]); + assert!(out.status.success(), "{out:?}"); + assert_eq!(std::fs::read(b.join("f.txt")).unwrap(), b"via cli"); + + std::fs::write(b.join("g.txt"), b"round trip").unwrap(); + assert!(sc(&b, &["commit", "-m", "c2"]).status.success()); + assert!(sc(&b, &["push", "origin"]).status.success()); + let out = sc(&a, &["fetch", "origin"]); + assert!(out.status.success(), "{out:?}"); + + for d in [&a, &bucket, &parent] { + std::fs::remove_dir_all(d).unwrap(); + assert!(!d.exists()); + } +} +``` +(Adapt `commit -m`/`clone`/`fetch` argv to the real clap surface — check `sc --help` output or `crates/cli/tests/ssh_remote.rs`'s calls and mirror them exactly.) + +- [ ] **Step 2: Run** + +Run: `cargo test -p scl-cli --test bucket_remote` +Expected: PASS. + +- [ ] **Step 3: Write ADR-0046 and update the standing docs** + +- `docs/adr/0046-wal-bucket-remotes.md` — follow the house ADR shape (Status/Date/Phase/Context/Decision/Consequences/Alternatives, see ADR-0013). Decision: bucket = WAL of immutable packs + parent-linked log + one CAS'd manifest; `objio` leaf crate; both schemes; filter refused; alternatives: dumb ref-file-per-branch bucket (rejected: no atomic multi-ref, no log for fleets), Store-level backend (deferred: reopens P3/P8), walgit itself (git-protocol server — covered by P18, see `docs/research/walgit-evaluation.md`). +- `CLAUDE.md`: + - Dependency rule sentence: adapters `{cli, desktop} → repo → {vfs, gitio, crypto} → core`, leaf edges `repo → tlsio` **and `repo → objio`** (`objio` depends on no workspace crate). Add the quarantine sentence: "**Object-store SDKs must stay quarantined in `objio`** — if you find yourself reaching for S3 elsewhere, add a function to `objio` instead." + - Capability map: change "All 35 phases are built and tested" to 36 with row: `| P36 | P36a built: bucket WAL remotes (sc+wal://, sc+s3://) — immutable packs + CAS'd manifest, multi-writer safe, no coordinator; checkpoints (P36b) and bucket-backed serve (P36c) pending | [0046](docs/adr/0046-wal-bucket-remotes.md) |` + - Standing boundaries: add "**Bucket remotes hold public content plaintext at rest** — bucket ACL is the perimeter (sealed content stays ciphertext, unchanged); partial-clone `filter` against bucket remotes is refused." +- `ROADMAP.md` → Deferred: bucket compaction/gc; leases; checkpoint fold (P36b, next); bucket-backed `sc serve` (P36c, next); partial clone from bucket remotes; static-bundle clone offload; native GCS backend; `S3Bucket` streaming (bodies currently buffered in RAM, `MAX_OBJECT_SIZE`-bounded per object but pack-sized per transfer). +- `docs/THREAT-MODEL.md`: new trust boundary — the bucket is untrusted storage with cooperative writers: write access = full authority over refs and history (fast-forward is cooperative); readers verify BLAKE3 per object, strict-decode the WAL fail-closed, cap lengths; sealed content remains E2E-sealed; public content confidentiality = bucket ACL. + +- [ ] **Step 4: Full verification** + +Run: `cargo test --workspace && cargo clippy --workspace --all-targets && cargo run --bin sc -- demo --agents 4` +Expected: all green; demo still proves zero residue. + +- [ ] **Step 5: Commit** + +```bash +git add crates/cli CLAUDE.md ROADMAP.md docs +git commit -m "feat(cli)+docs: bucket remote integration test, ADR-0046, CLAUDE.md/ROADMAP/THREAT-MODEL for P36a" +``` diff --git a/docs/superpowers/specs/2026-08-26-wal-bucket-backend-design.md b/docs/superpowers/specs/2026-08-26-wal-bucket-backend-design.md new file mode 100644 index 0000000..4762c44 --- /dev/null +++ b/docs/superpowers/specs/2026-08-26-wal-bucket-backend-design.md @@ -0,0 +1,182 @@ +# Design: WAL + manifest-CAS object-storage backend for sc (P36) + +- **Date:** 2026-08-26 +- **Status:** Approved design, pre-implementation +- **Origin:** walgit/Cursor-Continuity evaluation in + `docs/research/walgit-evaluation.md` (slot (c): borrow the coordination + layer, keep sc's sealing) +- **ADR to write at build time:** ADR-0046 + +## Goal and use cases + +Let an S3-compatible bucket be an sc remote — and let `sc serve` run as a +disposable cache in front of the same bucket — using immutable +content-addressed writes plus one compare-and-swapped manifest as the only +consensus mechanism. No database, no leader, no coordinator process. + +The MVP must prove two use cases: + +1. **Team sharing, zero ops.** A team (or one person on several machines) + collaborates through one bucket: push/fetch/clone with multi-writer safety + via manifest CAS; sealed content stays sealed at rest. Acceptance: two + writers race a push — one lands, the other cleanly retries; nothing + corrupts; an unauthorized bucket reader gets only ciphertext for + protected/private/secret content. +2. **Agent fleets at scale.** Many parallel agents share state through the + bucket with high-frequency small pushes and cheap cold starts + (checkpoint + log tail). Acceptance: N concurrent writers hammer the WAL + with no coordinator and no corruption. + +Hosted-service ops machinery (compaction, leases) is explicitly deferred. + +Both front doors ship over one bucket format: + +- **Direct-bucket transport:** `sc remote add origin sc+s3://bucket/prefix` + — clients read/write the WAL and CAS the manifest themselves using + object-store credentials. No server anywhere. +- **Bucket-backed `sc serve`:** stateless/disposable instances; clients keep + speaking sc's native protocol (wire `PROTOCOL_VERSION` 4) and never see + the bucket. + +## Bucket format (the load-bearing contract) + +One repository per bucket prefix. Four key kinds; only the manifest is +mutable: + +| Key | Mutability | Content | +|---|---|---| +| `manifest` | CAS-rewritten only | Tiny versioned binary: format version, head log seq, latest checkpoint seq | +| `log/` | Immutable | Parent-linked chain entries: `PUSH {pack ids, ref updates old→new}`, `CHECKPOINT {folded-through seq}` | +| `packs/.pack` | Immutable | Existing P8 pack format, unchanged, keyed by pack checksum | +| `checkpoints/` | Immutable | Full ref-map fold at ``, so cold start = checkpoint + log tail | + +Rules: + +- **The manifest CAS is the entire consensus.** S3: conditional PUT with + `If-Match` ETag (`If-None-Match` on create). Local-dir backend: atomic + rename compare. GCS-native (deferred) would use generation preconditions. +- **Readers trust only what the manifest chain references.** Any other key + is unreferenced garbage (crash debris) — harmless, swept by future + compaction (deferred). +- **Format version is strict.** Unknown manifest/log/checkpoint version → + refuse, fail closed (format-break discipline mirrors the canonical- + encoding invariant in CLAUDE.md). + +## Crates and the dependency rule + +- **New leaf crate `objio`** quarantines the S3 SDK, exactly parallel to + `tlsio`: it depends on no workspace crate. Public trait `Bucket`: + `get` (conditional, returns bytes + tag), `put` (immutable, + if-none-match), `put_if_match`, `list`. Two impls: + - S3-compatible (AWS, MinIO, Cloudflare R2 via endpoint override). + - Local-directory (tests, demos, `sc+wal://`). +- **Credentials:** the SDK's standard chain (env, config, IMDS). sc grows no + credential surface — same stance as the git bridge (ADR-0028). +- **WAL format + sync logic live in `repo`**, behind the existing + `Transport` seam from ADR-0013 (which anticipated exactly this backend). + New workspace edge: `repo → objio`, like `repo → tlsio`. +- **CLI schemes:** `sc+s3://bucket/prefix` (S3-compatible) and + `sc+wal://` (local backend, demos/tests). +- CLAUDE.md gains the quarantine sentence: object-store SDKs stay in + `objio`; reach for a function there, never for the SDK elsewhere. + +## Data flow + +**Push.** Compute missing objects by ref-frontier reachability (existing +push logic) → upload pack (idempotent: keyed by checksum) → write +`log/` with if-none-match (seq taken → try next) → CAS the manifest +with the fast-forward gate re-checked against the freshly read state. +CAS conflict → re-read manifest, re-verify fast-forward, retry with bounded +backoff; on exhaustion fail loudly (never silently drop — house invariant). +A crash at any step leaves only unreferenced keys; the commit point is +atomic at the manifest. + +**Fetch.** Conditional GET of the manifest (tag unchanged → up to date in +one round trip) → walk the log **by parent links from the manifest head** +back to the last-seen entry → download referenced packs → ingest through +the existing P25 two-pass atomic-after-verify path → update remote-tracking +refs. Seq numbers are claims, not truth: a CAS loser may abandon an entry +at a claimed seq (it re-reads and writes a fresh entry with the correct +parent before retrying the CAS), so the chain may have holes and orphan +siblings. Readers never scan seq ranges; only the parent chain from the +manifest head is authoritative, and off-chain entries are garbage. + +**Clone / cold start.** Manifest → latest checkpoint → tail the log → +download packs. Never replays the whole log. + +**Checkpoints.** Opportunistic, coordinator-free: any client that observes a +log tail longer than a threshold (default 64 entries, one tunable +constant) after its own push folds the +refs and writes `checkpoints/`, then CASes the manifest's checkpoint +pointer. Concurrent folds are harmless — both are valid; the CAS picks one; +the loser's checkpoint is unreferenced garbage. + +## Bucket-backed `sc serve` + +`sc serve --http|--https` gains a bucket-backed store mode. Local disk is a +pure cache of immutable packs (content-addressed, so trivially correct); +freshness is one conditional manifest GET per request — reads are strictly +consistent, there is no "eventually". Pushes from native-protocol clients go +through the same WAL commit path as direct-bucket clients. Instances are +disposable: kill one mid-push and nothing corrupts (the client retries; the +bucket never saw a manifest CAS). Existing access control and limits are +untouched: bearer tokens (P29), TLS via `tlsio` (P32), connection/timeout/ +pack-size limits (P31), `MAX_OBJECT_SIZE` (P28). + +## Security boundaries + +- **Sealed content needs zero new work.** Protected-path ciphertext + (P7/P33), private-branch sealed objects (P34), and wrapped secrets (P2) + travel and rest in the bucket verbatim as ciphertext — ADR-0013's + "confidential by construction" property, unchanged. +- **Public content is plaintext at rest in the bucket.** Bucket ACL is its + perimeter — the same trust model as any remote clone. Gets an explicit + THREAT-MODEL.md entry. +- **Everything read from the bucket is untrusted input:** BLAKE3 + verification on every object (id == hash), `MAX_OBJECT_SIZE` caps on + every length, strict ref-name validation on log/checkpoint contents (P28 + parity), strict versioned decode failing closed on unknown versions. +- Direct-bucket mode's write authorization IS bucket-write permission; + fast-forward gates are cooperative, not adversarial (a hostile bucket + writer can already destroy data). Server-fronted mode keeps the P29 token + model for adversarial clients. + +## Error handling + +Per-crate `thiserror`: `objio::Error` (network, auth, precondition-failed, +not-found); new `repo` variants (`CasConflict` internal-retried, surfaced +only on retry exhaustion; format-version refusal; garbage/chain-integrity +errors). CLI converts to `anyhow` with `?` as everywhere else. + +## Testing + +- **Cross-backend contract suite** both `Bucket` impls must pass + (conditional semantics, CAS atomicity, list ordering) — borrowed from + walgit's store-contract idea. +- **Two-writer race:** concurrent pushes; exactly one CAS wins; loser + retries and lands; final refs and reachability consistent. +- **Crash injection:** abort between each push step (after pack, after log + entry, before/after CAS) — repository always intact, only unreferenced + garbage remains. +- **Fleet hammer:** N threads pushing small commits concurrently against + the local-dir backend; no coordinator, no corruption, all commits land. +- **Cold start:** checkpoint + tail reconstruction equals full-log replay. +- CI runs the local-dir backend; MinIO parity runs behind an opt-in env + flag. Tests that touch disk clean up and assert the path is gone (house + rule). + +## Phasing + +- **P36a:** `objio` + bucket format + `BucketTransport` clone/fetch/push, + race + crash tests green. +- **P36b:** checkpoints + log-tail cold start. +- **P36c:** bucket-backed `sc serve`. + +**Deferred (record in ROADMAP → Deferred):** bucket compaction/gc, leases, +static-bundle clone offload (CDN-able clones), native GCS backend. + +## What we deliberately do not copy from walgit + +Plaintext-at-rest with perimeter-only auth. sc keeps end-to-end sealing; +we borrow only the coordination layer (WAL + manifest CAS + checkpoint +cold start).