[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine (stacked on #394) · next: PITR surface (#335 rebased), catalog index, backup job, restore job.
Closes #626 (part of #624).

What

New crate crates/backup-store: one trait, BackupStore (put, get, head, list, delete_prefix, validate), with a filesystem implementation and an S3 implementation on aws-sdk-s3. The backup job and the restore path (later PRs) write and read portable backups through it; #396 section 2 (docs/design/15-backup-restore.md) states the rules those callers honour and points at this crate's docs for what the store does. The README and lib.rs now say that the design overview states the rules the store's callers honour and this crate's docs are the record of what the store does; the README's list/delete_prefix bullet and the BackupStore::list doc state that the filesystem store returns the whole listing as one Vec before the stream yields (about 72 bytes plus the key per object) while the S3 listing is paged, and the IAM section says the example drops the s3:prefix condition when the prefix is empty. One commit, squashed with #336's work (Lee is co-author); supersedes #336. Rebased onto main after #384 and #338 merged.

Keys are /-separated component paths bounded at 1024 bytes, each component at most 255 bytes (the filesystem's NAME_MAX); empty components, ., .., backslashes, control characters and U+FFFE/U+FFFF (which S3's XML listings cannot carry) are rejected. That rule is the upper bound both stores accept; the key module doc is the one list of where each narrows it. Prefix matching is component-aligned. put returns ObjectMeta (bytes written, ETag where the store has one) so the manifest writer needs no second head. After a failed put the key holds nothing, the previous object, or the new complete object; never a partial one.

Filesystem store. Every key is resolved one component at a time with symlink_metadata; any symlink, in-root or escaping, is refused on put, get (which also opens with O_NOFOLLOW) and head. list and delete_prefix resolve their prefix with the same walk, so a symlink above the final prefix component is refused too; a symlink at the prefix itself is nothing there; below the prefix, list skips symlinks and delete_prefix removes them as links without following.

Those checks are path-based and can be raced by a process that can write under the root, so the store requires that only the owner can reach the tree, and enforces it on Unix: open() and validate() refuse a root that grants any permission to group or others (read bits included) or is owned by another uid, and a root whose ancestor is owned by a third user or is writable by group or others without the sticky bit. Every operation re-checks the root, refuses it if its canonical path changed since open (an ancestor moved or replaced by a symlink), and applies the owner-only rule to every existing directory it descends into, on the way to a list/delete_prefix prefix as well as below it. Directories are created 0700 and objects 0600 at creation. Every refusal names the path and the fix.

Writes go to a temporary sibling, flush, fsync, rename, fsync of the directory and its ancestors. The flush surfaces the last write's error: tokio's sync_all waits for the last background write but does not return its error (tokio keeps only its kind for a later write or flush), so without a flush before the drop a short file would be renamed into place. A put that does not reach the rename (an error, a future dropped at any point before the rename, the temporary's open on the blocking pool included, a panicking body) removes the temporary from a drop guard, as the S3 store aborts its upload; a failure after it (the directory sync) leaves the new, complete object rather than deleting an object that replaced the previous one. Only a process that ends without running the guard (killed, or exiting without waiting for the runtime's blocking threads) leaves a .put-*.tmp. Directories a failed put created stay, empty and invisible to list, until delete_prefix removes them (removing them raced a concurrent put into the same directory, and missed failures inside the creation walk). Listings report only names that pass the key rule; temporaries, non-UTF-8 names and names with a backslash are skipped and still deleted. A put onto a key that is a directory is refused before the body is read. Every operation resolves its path with one component walk. delete_prefix checks every directory under the prefix against the owner-only rule before it removes anything, so a refusal leaves the prefix whole; it leaves the directories above the prefix and treats an entry another deleter removed first as already gone (not counted; the trait defines the count, and concurrent S3 counts may overlap), so two reconcilers retrying the same prefix do not fail each other. A delete_prefix whose future is dropped stops before its next entry removal (the blocking walk checks a flag the dropped future sets), as the S3 store's stops at its next request. validate() re-applies the owner-only and ancestor rules, checks the root's canonical path, and writes, flushes, syncs and removes a probe (a probe a concurrent delete_prefix("") removed first counts as removed). Paths only a race or a crash reaches (a file removed between the walk and the open, an entry vanishing mid-listing, O_NOFOLLOW after the walk, the canonical check in prepare_parent, re-checks of directories a concurrent process created or changed, a symlink ancestor, the fsyncs) are kept as defence in depth and listed in the module doc as untested.

Consequence for operators, stated in the README and #396: a backup pickup agent (rsync, restic, borg) cannot read the tree as a member of the server's group; it runs as the server's user or as root. A root under a directory owned by a storage administrator or an NFS mount owned by nobody, or under a group-writable parent such as a root:staff 0775 /srv, is refused at startup.

S3 store. open refuses an empty bucket, a part size outside 5 MiB to 5 GiB, an endpoint that is not an absolute http:// or https:// URI or carries user information, a query, a fragment or an IPv6 zone id, or that http::Uri or url::Url refuses (refused with a message that names the source the endpoint came from and the problem class, never the value), and an http:// endpoint whose host is not a recognised loopback form (127.x.y.z, ::1, localhost) or that is used without force_path_style (the SDK picks virtual-hosted addressing by bucket name: a virtual-hostable name on localhost, an access point ARN on any custom endpoint; on an IP host every plain name is path-style, so the rule is keyed on the scheme because that choice is the SDK's and may change), unless allow_insecure_endpoint is set, which logs a warning; the error names the client, the host and the endpoint's source. open also refuses the two names the SDK routes to S3 Express, a directory bucket (--x-s3) and a directory-bucket access point (--xa-s3), and the client has S3 Express session auth disabled, so a client that reaches such a name sends no CreateSession (a test-only open skips the refusal to check that). The rules apply to every endpoint setting the SDK reads for a client that sends requests for the store: the S3 client's (the config file's, or AWS_ENDPOINT_URL, AWS_ENDPOINT_URL_S3 or the shared config's, which open reads and sets explicitly), those of the STS, SSO and SSO OIDC clients the credential providers build (AWS_ENDPOINT_URL_STS, _SSO, _SSO_OIDC, AWS_ENDPOINT_URL, the profile's endpoint_url and services entries), which are checked whether or not the configured credentials use them, and the IMDS endpoint setting (AWS_EC2_METADATA_SERVICE_ENDPOINT, the profile's ec2_metadata_service_endpoint), whose http:// form may name only loopback, 169.254.169.254 or [fd00:ec2::254]; a refusal names the client and the setting. The IMDS and general settings are read from the environment and the selected profile directly, as the SDK's IMDS client and endpoint provider read them, so a variable the SDK gives no meaning to (AWS_EC2_METADATA_SERVICE_ENDPOINT_, AWS_ENDPOINT_URL_, a trailing underscore) cannot shadow the one it reads; the service-key lookup is kept for S3, STS, SSO and SSO OIDC. AWS_CONTAINER_CREDENTIALS_FULL_URI keeps the SDK's own rule and is read only for the proxy rule. The settings are read from a load that sends nothing, so a refused configuration sends no request; the load that builds the clients follows, and resolves the region through IMDS when none is configured (off EC2, set region or AWS_EC2_METADATA_DISABLED=true). Without the override an interceptor also refuses any http:// request the S3 client would send to another host before it is sent; the credential providers' requests are checked only through their endpoints at open. An http:// endpoint is reached through an HTTP client built with aws-smithy-http-client's Builder (rustls with aws-lc, as the SDK's default) and no proxy, so HTTP_PROXY, http_proxy, ALL_PROXY and all_proxy cannot carry a signed clear-text S3 request off the host; https:// keeps the SDK's proxy behaviour (HTTP CONNECT only, so a socks5:// ALL_PROXY fails it). When any endpoint of the credential chain is http:// (an STS, SSO, SSO OIDC or general setting, AWS_CONTAINER_CREDENTIALS_FULL_URI, or the IMDS endpoint setting) the chain's clients get the same proxy-less client, so a credential request to a loopback emulator or sidecar never reaches the proxy (its https:// requests then bypass HTTPS_PROXY too; the S3 client's do not); otherwise the chain uses the SDK's default client and the README says NO_PROXY must list 169.254.169.254 (fd00:ec2::254 in IPv6 mode) and 169.254.170.2. Multipart above the configured part size (default 8 MiB) with bounded concurrency and at most 10,000 parts; a failure anywhere after CreateMultipartUpload, including a failed CompleteMultipartUpload, aborts the upload, and a drop guard, armed until the inline abort returns, aborts it when the put's future is dropped while its runtime runs (a cancelled job). Runtime shutdown drops the guard's spawned abort unpolled, an in-flight CreateMultipartUpload task included, so uploads of puts in flight at shutdown, parts included, stay until a delete_prefix covering them or the lifecycle rule removes them; the trait doc says a caller that must leave nothing awaits its puts first. CreateMultipartUpload runs in a spawned task that arms the guard and hands it back, so a put dropped while that request is in flight still aborts the upload. A create the SDK retried after the server created the first leaves an empty upload that only the lifecycle rule removes; validate's probe create runs in a spawned task that arms the guard, as a put's does. delete_prefix also aborts in-progress uploads under the prefix (component-aligned, as for objects), which sweeps what a killed process left; a denied ListMultipartUploads or AbortMultipartUpload there is logged and the call succeeds once the objects are gone, and any other listing or abort error fails it; the README recommends an AbortIncompleteMultipartUpload lifecycle rule as the backstop, and on a versioned or Object Lock bucket NoncurrentVersionExpiration plus Expiration { ExpiredObjectDeleteMarker: true }. Requests carry a 60 s attempt and 300 s operation timeout; each part must upload within the attempt, and the docs give the bandwidth per part size. Resolving credentials is bounded separately by the SDK's 5 s identity-cache load timeout (identity resolver timed out after 5s); the SDK's retry settings reach the S3 client only. A GetObject body is streamed after those timeouts end and is bounded per chunk (60 s) beside the SDK's stalled-stream protection. Paginated listing that skips keys failing the key rule (console folder markers); DeleteObjects batches of 1000 over the raw listing, per-key errors surfaced (AccessDenied as PermissionDenied, keys without the store prefix). A key that fits the shared rule but exceeds S3's 1024-byte limit once the bucket prefix is added is refused as InvalidKey; a bucket prefix over 512 bytes is refused at open. Errors are reduced to code and service message in Display (no request or credential material; Debug of a transport error carries the SDK's), denials included, so a KMS or explicit-deny message reaches the operator, and a response without an S3 code carries its HTTP status (HTTP 500 (no error code in the response)); NoSuchBucket and NoSuchUpload are not NotFound; InvalidToken, ExpiredToken and TokenRefreshRequired are PermissionDenied; transport errors display their source chain, so "dispatch failure" carries its cause. validate() lists one key under the prefix rather than calling HeadBucket, because a HEAD error has no body: a 403 keeps S3's code (AccessDenied, InvalidAccessKeyId, SignatureDoesNotMatch), a 404 names the missing bucket, a 301 the bucket's region. It then probes each other grant on a random key under the prefix (DeleteObject and GetObject of that absent key, CreateMultipartUpload then AbortMultipartUpload, ListMultipartUploads), leaving no object or upload (on a versioned bucket, one delete marker), and a denial names the missing grant (kms:GenerateDataKey for the create; kms:Decrypt is not exercised). A probe abort answered NoSuchUpload (a concurrent delete_prefix("") swept the probe upload) passes. The README's IAM section gives the two-resource shape (s3:ListBucket and s3:ListBucketMultipartUploads on the bucket ARN, the four object actions on bucket/<prefix>*; a statement on the object ARN alone grants neither list action and validate fails with "lack s3:ListBucket"), and notes for S3-compatible servers (only moto has been run; the SDK's default CRC32 request checksums and the WHEN_REQUIRED knobs; TLS trust from the system store, or from SSL_CERT_FILE/SSL_CERT_DIR alone when either is set (an empty one fails every https:// request and aborts open in a debug build); general-purpose buckets only, since directory buckets return unordered listings). open_with_interceptor and open_accepting_s3_express_names are test-util-gated seams; test-util also exposes the client config, settable part and page limits, and a removal hook on the filesystem store's delete_prefix walk.

On both stores, a put that overlaps a delete_prefix covering its key may fail or may leave its complete object after the deletion returns; the store gives no ordering.

Configuration. BackupStoreConfig models the [backup.store] table (kind = "filesystem" | "s3", default filesystem) with [backup.store.filesystem] and [backup.store.s3] (which gains allow_insecure_endpoint, default false). The server will hold it under its own key rather than flatten it, because deny_unknown_fields does not survive serde(flatten); only the section matching kind is read. The struct has no Default; the server supplies the default root.

Dependencies and CI. aws-config and aws-sdk-s3 are pinned to the newest releases that build on the workspace rust-version (1.88); their transitive aws-*/aws-smithy-* crates are held in Cargo.lock at 1.88-compatible versions, and .cargo/config.toml sets the resolver's incompatible-rust-versions = "fallback" so a plain cargo update keeps them there. aws-config enables sso (adding aws-sdk-sso 1.93.0 and aws-sdk-ssooidc 1.95.0, both 1.88-compatible), so a server whose credentials come from an SSO profile can reach the bucket; the feature also links ring, hex and zeroize, a second cryptography library beside aws-lc-rs. http 1 (already in the graph) is a direct dependency for the endpoint check and adds no package. aws-types 1.3.11 (already in the graph) is a direct dependency, to read the S3-specific endpoint setting; it adds no package. aws-runtime 1.6.0 (already in the graph) is a direct dependency, to name the SDK's shared config file set when reading the selected profile; it adds no package. url (=2.5.8) and aws-smithy-http-client (=1.1.9, rustls-aws-lc), both already in the graph through the SDK, are direct dependencies, to apply the SDK's URL parser at open and to build the proxy-less client for an http:// endpoint; they add no package. test.yml installs moto from requirements-moto.txt (moto 5.2.4 and its direct dependencies pinned at the versions the last green run resolved, keying the pip cache), starts it in server mode, runs the workspace with this crate excluded, then runs this crate with the S3 variables scoped to that step and EXTENDDB_TEST_S3_REQUIRED set, so the S3 suite fails rather than skips when the endpoint is missing, and prints moto.log when the job fails. One shipped crate moves with them: crc 3.4.0 to 3.3.0, because crc-fast 1.9.0 (1.10.0 needs Rust 1.89) requires crc ~3.3 and sqlx-core shares the one crc. The four licence notice files are regenerated; the crc line is the only content change. Both CI test steps use --locked. On the stable leg a further step runs the already-built conformance binary's filesystem:: tests as root (sudo env EXTENDDB_TEST_REQUIRE_ROOT=1), so the uid cells run in CI and fail rather than skip if the step is not root; the step sets pipefail; nothing is compiled as root.

Why

#336 was approved with the filesystem store creating world-readable files under a permissive umask and with its S3 tests never executing in CI (design review of #348). Three adversarial review rounds on this PR then found: a traversable root still exposed permissive subdirectories; list and delete_prefix skipped the ownership check and followed a symlink planted above the prefix (an in-root alias made delete_prefix destroy another backup; an escaping link made it delete outside the root); a second deleter racing on the same prefix failed with ENOENT, which the design's reconciler contract forbids; the post-rename cleanup deleted an object that had replaced an older one; the S3 abort test never started a multipart upload; the 301 and 403 branches were untested; NoSuchBucket read as a missing object; and the lockfile did not build on the 1.88 toolchain CI tests. A fourth round found: a dropped or crashed put left its multipart upload open, and nothing could abort it; validate reported every misconfiguration as a bare "dispatch failure"; the 403 code the docs promised is not in a HEAD response; the failed-put skeleton cleanup raced a sibling put and missed failures inside the creation walk; listings could yield keys get refuses; the uid rule and the post-rename failure path had no test; pagination and per-key delete errors were untested; a sub-table deny_unknown_fields test tested the top level. The maintainer's review of the fifth revision found: a full disk let a short file replace an object (tokio drops the last write's error at sync_all); the upload sweep matched by string prefix and aborted a sibling's upload; a 256-byte component passed the key rule and failed on the filesystem, and U+FFFE broke S3 listings; validate passed on credentials that could not delete or abort; a denied upload listing failed delete_prefix forever; delete_prefix("") deleted entries before refusing a permissive directory; a put dropped during CreateMultipartUpload left an upload; an empty bucket, a part over 5 GiB and a plain-HTTP remote endpoint were accepted; SSO profiles failed for want of a feature; token errors were not permission errors; and several claims had no test. The fifth review round found: an http:// endpoint from AWS_ENDPOINT_URL, AWS_ENDPOINT_URL_S3 or the shared config bypassed the loopback rule, and localhost without path style sent requests to <bucket>.localhost, a name the resolver may look up in DNS; validate did not probe s3:GetObject; a put dropped during its inline abort left an upload; denials dropped S3's message; on a versioned bucket validate and delete_prefix leave delete markers the docs did not mention; an ancestor moved and replaced by a symlink was refused by put only; the list of untested paths was incomplete; the root CI step could skip its uid cells silently. A sixth round found: a filesystem put dropped mid-body or whose body panicked left its temporary behind (the S3 half of the same PR had the drop guard, the filesystem half did not); the three S3 tests that send a request without an endpoint failed on a machine without AWS credentials and passed with real ones by using them; a scheme-less endpoint passed open and failed at validate with the SDK quoting the configured value, password included; a response without an S3 code read as unknown with its status dropped; the GetObject body had no bound of the store's own; the per-key DeleteObjects test shared one fixed prefix between processes; moto's dependency tree floated in CI; and the README's IAM paragraph hid the bucket/object resource split that decides whether validate passes and said nothing about the SDK's default request checksums. The maintainer's review of 2026-10-28 found: an endpoint with user information, a query or an IPv6 zone id passed open and the SDK then quoted it, password included, in validate's error; a signed clear-text request to a loopback http:// endpoint went through HTTP_PROXY; the configured endpoint's precedence over the SDK's sources had no test; a put dropped while its temporary's open was in flight left the temporary; the loopback refusal named a key the operator may not have set; the child-process tests inherited proxy variables; the README described the request checksum as aws-chunked trailers; four proxy-based tests failed rather than skipped against an https:// endpoint; and a denied AbortMultipartUpload failed every delete_prefix retry. A seventh round found: the SDK's STS/SSO clients followed a clear-text AWS_ENDPOINT_URL or profile endpoint_url that the S3 check never looked at, carrying a web identity token off the host; CreateSession for a directory bucket bypassed the request check; a dotted bucket on an IP http:// endpoint passed open; the docs did not say the credential chain uses the proxy or that trace SDK logs carry the session token; runtime shutdown sends no abort, which the docs denied; a put racing delete_prefix can succeed, which the docs denied; a dropped filesystem delete_prefix kept deleting; validate failed beside delete_prefix(""); and two proxy tests failed under load. An eighth round found: a loopback http:// credential endpoint passed open while HTTP_PROXY or ALL_PROXY carried the credential request, token included, to the proxy (the round-7 fix covered the S3 client only); --xa-s3 names, which the SDK also routes to S3 Express, opened and were operated as S3 Express targets, so the session-auth flag's "no observable effect" claim was false; the IMDS endpoint setting was unchecked and the README contradicted the config doc about it; SSL_CERT_FILE/SSL_CERT_DIR replace the trust store, which the README did not say; the dropped-delete_prefix test did not pin the per-entry stop (two mutants survived); and the docs misdescribed the credential chain, open's IMDS lookup, and the SDK's addressing of a dotted bucket on an IP host. A ninth round found: the IMDS and general endpoint checks read their settings through a service key with an empty service id, which tries AWS_EC2_METADATA_SERVICE_ENDPOINT_ and AWS_ENDPOINT_URL_ (a trailing underscore, which the SDK never reads) before the real variables, so an https:// value there hid a non-loopback http:// IMDS endpoint from the check while the SDK's client sent to it; and the docs did not state the SDK's 5 s credential-resolution bound, that the retry settings do not reach the provider clients, AWS_DEFAULT_REGION, fd00:ec2::23, the IMDS mode profile key, legacy SSO or login_session. The format PR (#394) needs the ETag and size from put.

Testing done

cargo test -p extenddb-backup-store: 33 unit tests and 116 conformance tests (62 filesystem, 54 S3) pass against a tempdir and, with EXTENDDB_TEST_S3_REQUIRED=1 and the endpoint pointed at moto 5.2.4, against moto; the same on cargo +1.88.0 test --locked. Thirteen S3 tests need no endpoint (the two open length refusals, client configuration, transport errors against a closed port, SSO, endpoints from the SDK environment, the request-time cleartext refusal, an http:// endpoint reached without the environment's proxy, the credential providers' endpoints, the IMDS endpoint setting, the credential providers' requests bypassing the proxy, both S3 Express names refused at open, no CreateSession to an S3 Express name). None needs anything from the environment: the tests that run in a child, every one that sends a request among them, get test credentials (testing keys, a web identity token file, an SSO token cache or a container token), AWS_REGION and AWS_EC2_METADATA_DISABLED=true (except the cells whose IMDS endpoint is a loopback listener), and none of the parent's proxy variables; the crate passes with no endpoint, no credentials and an empty HOME. Three filesystem tests need root and print a skip reason otherwise, or fail when EXTENDDB_TEST_REQUIRE_ROOT is set; as root all 55 of the previous revision passed in a container, and the CI step runs them as root; the 62 of this revision have not been run as root here (no usable sudo). 89 hand mutants of the rules fixed in review (34 in round 4, 14 in round 5, 4 in round 6, 13 for the maintainer's review of 2026-10-28, 15 in round 7, 7 in round 8, 2 in round 9); 88 are each killed by the suite. One is not: the one that lets a child inherit the parent's proxy variables fails two child tests only together with the mutant that leaves the proxy enabled. The suite also passes with HTTP_PROXY set in the parent, and it passed three times concurrently against one moto with 32 busy loops on 16 cores.

Unit tests: the key rule (255-byte components counted in bytes, U+FFFE/U+FFFF and their neighbours, a 1024-byte key made of valid components), configuration and open refusals (empty bucket, 5 GiB part ceiling, seven non-loopback http:// endpoints refused (including 127.0.0.1.evil.example), seven loopback or https:// ones accepted, an http:// endpoint refused without path style (localhost, a dotted bucket on an IP host, an access point ARN), every class the syntax rule refuses (empty, scheme-less, ftp://, a stray space, user information, a query, a fragment (?@ and #@ among them), an IPv6 zone id, a host url::Url refuses) refused without echoing the value (the scheme included) and naming the client and the source, and the configuration's Debug without the endpoint's user information), the request-time refusal of an http:// request to another host, and a table pinning map_sdk_error for every code and status cell, with and without an S3 code.

Filesystem tests cover: shared conformance (round trip, overwrite, empty body, listing and prefix rules, list order by key bytes, delete counts, invalid keys including a 256-byte component and U+FFFE/U+FFFF, multipart-sized bodies, failed-put cleanup, metadata agreement between put, head and list); 0600/0700 at creation; ten root modes with any group or other bit refused and 0700/0500 accepted; a world-writable parent and grandparent refused, a sticky one accepted; a permissive subdirectory refused by put, get, head, list and delete_prefix whether it is the prefix or above it, and delete_prefix refusing before it deletes anything; as root, a root, subdirectory or ancestor owned by uid 65534 refused as the module doc says; a full disk (a child process with a lowered RLIMIT_FSIZE: the put fails with EFBIG and the previous object survives; validate fails); a root widened after open refused by every operation; symlinks (dangling at the destination, dangling in the parent chain, relative, chained, symlinked root, in-root aliases, aliases and escaping links above a prefix, links below the prefix removed without following); two concurrent delete_prefix calls both succeed and count every object once; temporaries hidden and still deleted; delete_prefix("") keeps the root usable; a put onto a directory refused before the body is polled; failed puts leave only empty directories; a directory-sync failure after the rename keeps the new object; names failing the key rule (non-UTF-8 file and directory names among them) hidden and deleted; the root replaced by a symlink refused; an ancestor moved and replaced by a symlink refused by every operation; delete_prefix keeps the directories above the prefix; a put dropped mid-body once its temporary exists, a put whose body panics, and 200 puts and validates aborted after 0 to 400 µs leave no temporary; a dropped delete_prefix removes at most one entry after the drop, the walk held at a chosen removal (a file with a sibling after it, the last file of a directory, a directory) through the test hook; validate succeeds beside delete_prefix("").

S3 tests against moto cover: the shared conformance; the multipart abort paths (a body failing inside the third part, a failed complete, NoSuchUpload at complete, a dropped put, a put or a validate dropped while CreateMultipartUpload is in flight (dropped once moto lists the upload, while a delaying TCP proxy holds the response; a sleep in an interceptor blocked the put's own poll), and an orphaned upload swept); the part-count limit and the in-flight part bound; pagination of objects, and of uploads through an interceptor that cuts the pages (moto never truncates ListMultipartUploads); the sweep's component-aligned filter; a per-key DeleteObjects denial (injected; moto has no IAM); foreign keys hidden and deleted; every validate probe (GetObject included), the grant each denial names, and no object or upload left behind; a denied upload listing or upload abort in delete_prefix succeeding, and a 500 or ExpiredToken listing or abort failing it; a put dropped during its inline abort and a validate dropped during its probe abort (a proxy swallows the abort, and the drop waits until it has); a 301 and a 403 via a response-rewriting interceptor; a GetObject body cut after its headers by a TCP proxy failing within the body idle timeout (moto cannot produce any of these); a transport failure's cause; a missing bucket on validate, get, list and put; the length caps; the client's timeouts, region and addressing style; an SSO profile reaching the SSO provider; endpoints from AWS_ENDPOINT_URL, AWS_ENDPOINT_URL_S3 and the shared config refused at open, naming their source, and with the override used by the client; endpoint winning over each SDK source; eight refused endpoint shapes through endpoint and each SDK source (51 child processes, two of them under AWS_IGNORE_CONFIGURED_ENDPOINT_URLS, which neither lifts a refusal nor drops a loopback setting); an http:// endpoint reached without the environment's proxy and an https:// one through HTTPS_PROXY; a validate whose probe abort answers NoSuchUpload passing; runtime shutdown with puts in flight leaving uploads that delete_prefix sweeps; the credential providers' endpoints refused at open (nine child cells, one with AWS_ENDPOINT_URL shadowed by an https AWS_ENDPOINT_URL_; a listener on a non-loopback address of the host receives nothing); the IMDS endpoint setting (seven child cells: a non-loopback http:// one from the variable, from the variable beside an https AWS_EC2_METADATA_SERVICE_ENDPOINT_, or from the profile refused with nothing sent, user information refused without the secret, loopback and the two IMDS addresses accepted); the credential providers' requests reaching a loopback http:// endpoint and never the proxy (seven child cells: STS through its own setting and through AWS_ENDPOINT_URL on localhost, the container URI, SSO, an SSO OIDC refresh, the IMDS setting, and an https:// S3 endpoint still through HTTPS_PROXY beside an http:// STS setting); both S3 Express names refused at open; the client sending no CreateSession to an S3 Express name; an http:// request to another host refused before it is sent. The five proxy-based tests skip against an https:// endpoint.

cargo fmt --all -- --check
cargo clippy -p extenddb-backup-store --all-targets -- -D warnings -D clippy::pedantic   # 0 findings
cargo clippy -p extenddb-backup-store --lib -- -D warnings -D clippy::pedantic             # 0 findings, without test-util
cargo clippy --locked --workspace --all-targets -- -D warnings                            # 0 findings
cargo test --locked -p extenddb-backup-store                                              # 33 + 116 (S3 vs moto)
HTTP_PROXY=http://127.0.0.1:1 cargo test --locked -p extenddb-backup-store                # 33 + 116
cargo +1.88.0 test --locked -p extenddb-backup-store                                      # 33 + 116
cargo +1.88.0 check --locked --workspace --all-targets
cargo check --locked --workspace --all-targets                                            # stable
./devtools/generate-software-license-notices --check [--backend mongodb|cassandra]; ./devtools/generate-dev-license-notices --check

The test harness's ensure_bucket now does HeadBucket first, creates once per process and retries OperationAborted; moto never answers 409, so that retry is untested.

Not changed here

• list() collects the filesystem walk into a Vec before streaming. The trait promises lexicographic order, which a directory walk cannot provide incrementally, and listings under the backup layout are small.
• The symlink TOCTOU is bounded by the ownership requirement, not closed; closing it needs openat2(RESOLVE_BENEATH), which std does not expose. The ancestor rule is checked at open and validate only; every operation checks that the root's canonical path has not changed.
• head on a missing bucket reads as the key being absent and on a denied one as HTTP 403 (no error code in the response): HEAD responses have no body; validate is the check that names the problem.
• Race-only and crash-only paths have no test; the filesystem module doc lists them.
• Nothing here runs against Amazon S3, a versioned or Object Lock bucket, or SSE-KMS (moto creates no delete marker for an absent key); the delete-marker and KMS statements rest on the AWS documentation. No S3-compatible server other than moto has been run; the README names the SDK's default request checksums and the knob that turns them off for a server that lacks them.
• A .put-*.tmp left by a process that ended without running the guard (killed, or exiting without waiting for the runtime's blocking threads), or by a guard whose unlink failed, is removed only by a delete_prefix covering it; the reconciler's prefix deletion is that sweep.
• The three bare cargo build --release jobs in integration.yml build the SDK into the workspace build and feature-unify hyper/client, hyper-util/{client,client-legacy,client-proxy}, md-5/std and num-integer/default into the extenddb binary (Build release 5:03 -> 7:51, 4:14 -> 8:07, 4:54 -> 5:56 against abf8082); building -p extenddb there as the Dockerfile does is a workspace-wide CI change outside this crate.
• Cargo.lock gains duplicate versions none of which is avoidable at this SDK version: foldhash 0.1/0.2, hashbrown 0.16 (a fourth version, via lru), http 0.2 + http-body 0.4 beside their 1.x, sha1 0.10 beside 0.11, spin 0.9/0.10; today they compile only for this crate's tests and the workspace-wide builds.
• No consumer of the crate yet; the configuration PR wires [backup.store] into the server and the backup job PR calls the trait. The licence notices change materially then.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: #396, section 2. The BackupStore trait is internal to this series and has no implementors outside the crate.

Breaking changes

None for users; the crate is not yet consumed. Operators who adopt the filesystem store later need a 0700 root under root- or server-owned ancestors; the README and #396 state the pickup-agent consequence. .cargo/config.toml is new to the repository and changes how cargo update picks versions for every crate (it prefers versions that build on 1.88).

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.

[LeeroyHannigan]

I built the crate on stable and 1.88.0 and ran the suite in a container against a tempdir and moto 5.2.4: 18 unit and 72 conformance tests pass, as the description says. I ran the S3 tests and direct probes against Amazon S3 in us-east-1, drove the filesystem store under SIGKILL, injected I/O errors, a full tmpfs and a same-uid symlink racer, and ran mutants of fs.rs and s3.rs against the suite. The gates, the licence notice checks and both CI legs pass, with all 25 S3 tests run.

The owner-only rule, the modes at creation, crash safety at every kill point, concurrent deletes and byte-order listing matched the description. One Blocking item and five more are inline. Smaller items:

• The uid cells can run in a root CI job that chowns a root, a subdirectory and an ancestor to another uid. Mutants that accept a foreign owner pass today.
• delete_prefix("") removes some objects before it refuses a foreign-owned subdirectory.
• A put dropped while CreateMultipartUpload is in flight leaves an empty upload open (8 or 9 of 16 on S3).
• open accepts an empty bucket, a part size over 5 GiB and an http:// endpoint to a host that is not loopback.
• aws-config is built without sso, so an SSO profile fails at the first request.
• 400 InvalidToken and 400 ExpiredToken map to Other, not PermissionDenied.
• On a versioned bucket delete_prefix leaves noncurrent versions. The README should name the lifecycle rule.
• ensure_bucket calls CreateBucket in every test. Amazon S3 answers parallel calls on an existing bucket with 409 OperationAborted, and 15 to 20 of the 25 S3 tests fail there.

Once the six inline items are fixed, I will approve.

[LeeroyHannigan]

I re-ran the items from my review against this head in the shapes they were measured in (a full 4 MiB tmpfs for the filesystem store, Amazon S3 for the upload sweep, the key rule and the token codes, moto with IAM enforcement for the grants), with 37 mutants of the code that answers them, and found no item open. On the full disk the put fails with ENOSPC and the previous object is byte-identical; delete_prefix("sib/x") on Amazon S3 aborts the uploads under sib/x/ and leaves sib/xy/, sib/x-other and sibling/; a 256-byte component and U+FFFE are refused before either store is touched; validate names each of the six grants it needs and a delete_prefix retry with the upload listing denied succeeds; the S3 suite on Amazon S3 has no 409 left. Of the eight filesystem claims that had no test, three now have one and five are listed as untested defence in depth, which the item allowed. I also ran validate's request sequence under a role carrying the README's policy as printed: the read of a missing key under the prefix answers 404 NoSuchKey, so that policy passes validate.

The code added since that review got the same reading: 31 unit and 103 conformance tests pass on stable and 1.88.0 against a tempdir and moto, clippy 1.97.0 is clean, CI runs the three uid tests as root, and the extenddb binary's dependency graph has neither this crate nor the SDK. Of 17 mutants of the fixes added since, 16 fail a test. Three inline items need code: an endpoint with a query component or an IPv6 zone id passes open, and validate then fails with the SDK's error carrying the value, password included; the clear-text rule does not see an HTTP proxy, so a signed request to a loopback http:// endpoint went in clear text to the listener HTTP_PROXY named; and the configured endpoint's precedence over the SDK's sources has no test, and the mutant that drops it passes all 134. Six smaller items are inline: a put dropped during its open leaves a hidden temporary; the loopback refusal names a config key the operator may not have set; the child-process tests inherit proxy variables; the README's checksum sentence; four of the new S3 tests fail rather than skip against an https:// endpoint; and a denied AbortMultipartUpload still fails every delete_prefix retry after the objects are gone, which is the last clause of the grants item.

Once the inline items are fixed, I will approve.

[LeeroyHannigan]

I re-ran the items from my last review against b3e702f in the shapes I measured them in. The endpoint shapes that carried a password are refused at open from the config file and the four SDK sources, with the source named and the value left out of the message. With HTTP_PROXY, http_proxy, ALL_PROXY or all_proxy set, the signed request to a loopback http:// endpoint went to the endpoint and the proxy listener got nothing, and the child tests pass with a proxy in the parent's environment. With effective_endpoint skipping its is_client_config branch the new endpoint cells fail; 3000 puts dropped after 0 to 400 µs left no temporary in two runs; the five proxy tests skip with their reason against an https:// endpoint; the captured PutObject, UploadPart and DeleteObjects requests carry x-amz-checksum-crc32 as the README says; and with AbortMultipartUpload denied, delete_prefix returned Ok on the first attempt and on the two retries.

On Amazon S3 at this head the 54 s3:: tests pass, with the five proxy tests skipping and naming the http:// endpoint they need.

[LeeroyHannigan]

LGTM 🚀