[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · next: PITR surface (#335 rebased), catalog index, backup job, restore job.
Closes #628 (part of #624).

Stacked on #394 (base branch feat/backup-format-streaming): SnapshotInfo is #394's SnapshotDescriptor and the orderability test is its KeyOrder::resolve. The diff shows this PR's single commit on top of #394; GitHub retargets it to main when #394 merges.

What

The one new obligation the engine-owned backup path places on a backend: open a consistent view of one table and stream its items. The trait returns the table definition, a snapshot marker, an ordering flag, and the item stream together, from one operation, so a backup's manifest and data files always describe the same instant. The in-catalog backup code in #384 keeps its own copy of this logic; the engine-side backup job (next PR) is the first caller.

Trait (crates/storage/src/snapshot.rs):

pub struct TableSnapshot {
    pub description: TableDescription,   // the storage-level description at the snapshot instant (not the API response: restore_summary may be set, table_throughput_mode_summary is not, and the counts are the stats pass's)
    pub info: SnapshotInfo,              // = extenddb_core::backup_format::SnapshotDescriptor { kind, marker }
    pub key_ordered: bool,               // items arrive in base-table key order
    pub items: BoxStream<'static, Result<Item, StorageError>>,  // owns the transaction/session
}
pub trait SnapshotEngine: Send + Sync {
    fn open_table_snapshot(&self, account_id: &str, table_name: &str)
        -> BoxFuture<'_, Result<TableSnapshot, StorageError>>;
}

Dropping the stream releases the backend transaction or session. A stream that has not reached its end (including one never polled) must be dropped inside a Tokio runtime: the sqlx backends return its connection through the runtime and panic outside one, MongoDB does not, and the trait doc says so. The first Err ends the stream on every backend. StorageEngine now requires SnapshotEngine; all four in-tree backends implement it. key_schema_is_orderable is KeyOrder::resolve(..).is_ok(), so a stream a backend marks ordered is one the manifest parser accepts, and a table whose range key has no attribute definition streams unordered and complete instead of paging by the hash key alone. The trait doc lists each backend's errors (Cassandra answers Unsupported before it resolves the table). StorageError::SnapshotExpired is new, for a snapshot the server discarded before the stream drained; the engine maps it to a retryable 503 like Transient.

Indexes. The description keeps every index in whatever build state it had and drops only one being deleted (drop_deleting_indexes, for GSIs and vector indexes; no in-tree backend reports a DELETING index today, and unit tests pin the rule); on PostgreSQL a GSI row whose data table is missing is also left out, with the attribute definitions only it used, as the in-catalog backup does (since #401 UpdateTable creates the data table first, so this is a guard). A restore recreates the indexes in the description and builds them from the items, so a CREATING index comes back complete; an index left out would not come back at all.

PostgreSQL. On one shared database the open uses one connection and one REPEATABLE READ READ ONLY transaction whose first statement reads the tables row, so the definition and the items come from one MVCC snapshot and no row lock is taken; a DeleteTable that drops the data table between that snapshot and the open's LOCK TABLE makes the open TableNotFound. With a separate data database the tables row is held FOR SHARE on a catalog connection from the definition read until the data transaction (REPEATABLE READ READ ONLY, ACCESS SHARE on the relation) has pinned its snapshot with SELECT pg_current_snapshot() and the GSI data-table probe has run; the definition is read on that barrier connection (build_table_description_on), so an open holds one catalog connection and the stream one data connection. UpdateTable and DeleteTable take the row FOR UPDATE, so neither commits in the gap; index-status workers do not, so an index may be reported in a slightly older status than the items. The marker is pg_current_snapshot() text on both layouts. An engine-wide permit is held per snapshot from the open until the stream's transaction ends, half the pool on either layout, and an open over the bound is LimitExceeded at once, so open snapshots leave half of the pool their items come from to requests (with a separate data database and the whole pool allowed, ten open snapshots made GetItem wait out the pool timeout); restores' own half-pool slots are separate, so snapshots and restores together can hold the whole pool, and the field doc says so. Items stream through a server-side cursor, FETCH 1000 per round trip, ordered by the primary key; a fetch is bounded by rows, not bytes (up to 27.7 MB of text per row at the default item limit), and the byte budget is the backup job PR's. A table created before #361 declared the key columns COLLATE "C" keeps the database collation; ordering it bytewise would sort the whole table before the first row while the lock, connection and permit are held, so such a table is streamed unordered with a warning. Only a collation named "C" or "POSIX" counts as bytewise; "default" on a C database streams unordered, which is complete. SQLSTATE 40001 and 72000 map to SnapshotExpired. The stream holds ACCESS SHARE until its last fetch, so a DROP TABLE waits behind it; DeleteTable's synchronous path, which commits the catalog removal before the drop, now takes the asynchronous path (row DELETING, worker drop under a lock timeout) when another session holds ACCESS SHARE on any relation it would drop (the base, GSI, LSI and vector-index data tables: data::table_storage_relations, the one list drop_table_storage also drops), probing only when its answer is used (delay < 1 s) and, on the shared layout, on its own catalog transaction rather than a second connection from the same pool (twenty concurrent DeleteTables on a ten-connection shared pool timed out without that), so a caller cancelled during the wait cannot leave the row gone and a data table orphaned while a snapshot, a backup or a plain read holds the table (a write transaction, a GSI backfill batch or a queue claim hold ROW SHARE/ROW EXCLUSIVE/SHARE, which the probe does not match; the drop still waits behind them with the row gone, for one batch or request). On the shared layout a DeleteTable probing between an open's snapshot and its LOCK still takes the synchronous path and its DROP waits behind that stream; the module doc states the two-statement window. The data transaction lives in a guard from BEGIN on, so every path that ends it after it may have taken ACCESS SHARE wakes the control-plane poller: the last page (before its commit is awaited), a fetch error, a stream dropped before its end, and an open that fails or is cancelled; the in-catalog backup's data transaction is held in the same guard and wakes it too. The poller stays active while a DELETING row's drop is deferred by a reader, so a DeleteTable left DELETING finishes within about a second of the reader's release for every reader (Lee measured 51.8 s before for a snapshot stream and round 6 measured 57–60 s for the in-catalog backup and a plain reader; about 160 ms now on every snapshot release path, which also wakes it, and 60–160 ms after a plain reader's release in the test). The admin guide's control_plane_delay_seconds row and the troubleshooting page's DELETING entry state the bound, the same-name CreateTable refusal until then, and a pg_locks query to find the reader.

SQLite. One deferred read transaction covers catalog and data; its first read pins the WAL snapshot and no lock is taken, so writers never wait on a snapshot being opened. A TRUNCATE checkpoint meanwhile waits out the busy timeout and reports busy = 1; the module doc gives the measured reply. build_table_description_on reads the definition, including #384's restore_summary, on that transaction. The stream holds a slot from #384's backup_server_slots for its life (the pool arithmetic that keeps half the pool free counts snapshots); with none free the open returns LimitExceeded at once, and a CreateBackup that waited out its 30 s for a slot says that backups or table snapshots hold them; the pool_size doc gives the arithmetic. Items page through row-value keyset pagination on the primary key index; a page is bounded by rows, not bytes, as on PostgreSQL. An in-memory database has one connection, so the open reads every page in its transaction and returns a stream over the copy, holding no connection once it returns; the copy's stream keeps its backup slot until it ends or is dropped, so the slots bound how many decoded copies exist. This is the shape #384's create_backup_single_connection uses, so CreateBackup keeps working on sqlite-memory.

MongoDB. A session started with snapshot: true reads the tables document, the indexes documents and the base collection at one cluster time, which is the marker. Every read sets readConcern: snapshot explicitly, because the driver only fills it in when neither the operation nor its database carries one, and the production catalog handle carries readConcern: local (as any handle does under a URI with readConcernLevel=); on that handle a read without the level fails: the first reports no snapshot time, which the open refuses naming the read concern, and later ones are refused by the server with code 72 (InvalidOptions). A server that is not a replica set member refuses the first read with NotAReplicaSet (123), mapped to Unsupported. The index read in the session maps 239/246 like the others. Error messages carry the server's code name and message, not the driver's dump of the reply. A document the stream cannot decode ends it with that error. SnapshotTooOld (239) and SnapshotUnavailable (246) map to SnapshotExpired; the server keeps the snapshot for at least minSnapshotHistoryWindowInSeconds and may discard it after, and an idle cursor times out after cursorTimeoutMillis (10 minutes; CursorNotFound maps to Internal). key_ordered is false: a composite key's _id is a netstring of the key text, not the format's order.

Cassandra. open_table_snapshot returns Unsupported. Cassandra has no read that fixes an instant across a table (a token-range scan observes each range at a different time; LWT serializes single partitions), and its in-catalog logical backup says so (ADR-0019). Returning that copy under this trait's contract would make the manifest's snapshot member false, so the backend refuses and the in-catalog path stays its backup path until the design settles it (#396 section 3).

Test hooks. A fetch-size knob, a round-trip counter and a pause at two points of the open (AfterDefinition, BeforeItems) live in extenddb_storage::snapshot::paging; every backend reads the fetch size there and calls the pause. PostgreSQL and SQLite record each FETCH or page query; for MongoDB, whose driver hides getMore, the test fixture records, through command monitoring, each find the server receives on the data database at readConcern snapshot (the stream's; the GSI backfill's find reads at majority) and each getMore on a cursor those finds returned, so the count is the snapshot stream's paging as the server saw it. They and the conformance check compile only with extenddb-storage's test-hooks feature, which each backend crate's tests enable through a dev-dependency on itself (on MongoDB through a new snapshot-test-hooks, so the crate's existing coordination test-hooks stays off in its unit tests, as on main); production builds carry none of them. PostgreSQL also re-exports, under test-hooks, the data relation names and apply_all_migrations (the three extenddb migrate runners), so tests/table_snapshot.rs binds the crate's lists instead of copying them. The trait doc says why SnapshotEngine has no default body and StorageError is not #[non_exhaustive]: both breaks are meant. devtools/run-mongodb-tests publishes its container on 127.0.0.1 only.

Why

#396 section 3. An earlier draft of the trait returned items only and left the definition to a separate DescribeTable; a GSI added or the table deleted between the two calls produces a manifest that does not match its data files. Returning both from one operation is the only way the backup job can promise the manifest describes the data, and it has to be a backend obligation because the mechanism differs per backend.

Four adversarial review rounds on this PR found, and this revision fixes: the conformance check was inert at the production page size, then depended on the backend test setting the page size, then could not observe whether the knob had taken effect (it now sets the shared knob and asserts from the counter that a second fetch happened); the MongoDB engine inherited the client's read concern and the constant for SnapshotTooOld was SnapshotUnavailable's; an ACTIVE-only index filter dropped a GSI mid-backfill; the PostgreSQL ORDER BY lacked COLLATE "C", so a pre-#361 table streamed in locale order under a key_ordered flag and the design's restore would have refused a valid backup; the PostgreSQL open took two catalog connections and could deadlock a full pool; the MongoDB tests dropped fixed database names, so a URI in the environment could destroy another instance's data; the SQLite path held the writer lock across a pool acquire and bypassed the slot accounting; a missing sort-key attribute definition made the stream page by hash key alone and lose rows; SnapshotInfo and the orderability rule were second copies of #394's definitions. The fourth round: the MongoDB counter echoed the batch size the stream asked for (a cursor without batch_size passed); a synchronous DeleteTable during a stream could orphan the data table; a half-created GSI left an unused attribute definition CreateTable refuses; a shared catalog/data pool could deadlock concurrent opens; the Cassandra test created no table; the harness's key-order oracle was a third statement of the order. Lee's review then found: nothing checked that the definition and the items come from one instant or that the stream read after the writer committed (READ COMMITTED, a missing FOR SHARE, a MongoDB read outside the session, and a SQLite stream that read everything during the open all passed); only one key shape was tested; with a separate data database the bound was the whole pool, and ten snapshots made GetItem fail after 30 s; DeleteTable probed only the base data table, so a reader on a GSI table left the catalog row gone and the drop blocked; the collation clause turned a legacy table's read into a full sort; a MongoDB decode error did not end the stream and a standalone server reached the client as a 500; the SQLite refusal blamed backups when snapshots held the slots; and sqlite-memory lost a working CreateBackup. The fifth round found: on the shared layout DeleteTable took a second connection from the pool it already held one from, and twenty concurrent deletes on a ten-connection pool timed out; the open took two connections from one pool on the shared layout; four release paths (a drained stream, a fetch error, an open that failed or was cancelled) did not wake the poller; the one-instant scenario passed with the pauses outside the read window; the SQLite rowid page was untested and the MongoDB counter counted the GSI backfill's reads; a foreign test's open could take the armed pause; N range keys were not among the key shapes; in-memory SQLite copies were not bounded by the slots. Round 6 found that the one-second DeleteTable bound held only for readers that woke the poller (a snapshot stream), while a plain read, pg_dump or the in-catalog backup left the row DELETING until the 60 s idle pass; the poller now stays active while a drop is deferred, the backup wakes it too, and the conformance check asserts the branch it depends on, compares the description with DescribeTable, and bounds every drain.

Testing done

extenddb_storage::snapshot_conformance::run is one check every backend runs against a live database. It sets the shared fetch size to 50, opens a snapshot on a 700-item composite-key table with a GSI, reads 51 items and asserts the round-trip counter advanced by at least two since just before the open, then a writer deletes 100 items, inserts 100, and overwrites all 600 remaining items twice before the reader drains. It asserts 700 items, no duplicates, every deleted item present, no inserted item, every version at its pre-open value, that the counter advanced after the writer committed, key_ordered as the backend declares it and strictly ascending keys by KeyOrder::compare when ordered, and that a second snapshot after the writer stops shows the final round. Then, once per pause point and on a fresh table, it holds the open while it commits a write W1, an UpdateTable adding a GSI and a write W2 (in that order; on PostgreSQL with a separate data database the UpdateTable waits for the row lock and the open resumes after 2 s), and asserts what one instant implies: a description with the GSI has W1, items with W2 have the GSI in the description, and W2 is not in the items at all, since the pause is inside the open's read window (a backend whose pauses sat before both reads passed the two implications), and which of W2's commit and the 2 s timeout resumed the open, against the backend's declared Expectations::update_table_waits_at (PostgreSQL with a separate data database at AfterDefinition: the timeout; everywhere else: W2), so a slow UpdateTable cannot make the W2 assertion vacuous. Before the writer starts, the snapshot's description must equal the storage layer's describe_table with the running counts and the response-only summaries cleared. Every drain is under a 120 s deadline naming the stage and the items read, and the hooks lock gives up after 600 s naming the wait. Six more key shapes (hash-only S, N and B; S hash with S, N or B range) run at a page of 3, with keys whose byte, collation and base64 orders differ and range numbers whose text and numeric orders differ (negative, fractional, 1E-130, 9.9E125, and 38- and 34-digit pairs a last digit apart; the check's Expectations say how many digits a backend stores, and MongoDB's refusal of the 38-digit pair at the put, Decimal128's 34 digits, is asserted). Every test that opens a snapshot or touches the hooks holds snapshot_conformance::hooks_lock(), whose guard run and run_refusals take as an argument, so no other test in the binary can take the pause, reset the knob or feed the counter during a check. These mutants fail it: a fresh transaction per page or every page read during the open on SQLite, and >= in its hash-only or binary range keyset; OFFSET/LIMIT in READ COMMITTED, READ COMMITTED with the cursor, and no FOR SHARE on PostgreSQL; the data cursor or the index read outside the session on MongoDB; the pauses moved outside the read window on SQLite or MongoDB; the pauses moved outside the read window with a 2.5 s UpdateTable (the branch assertion); a description built from a second code path (the DescribeTable comparison); a stream that never ends (the drain deadline, within 120 s instead of hanging the binary). run_refusals covers a missing and a non-ACTIVE table on the three backends that implement snapshots. 27 hand mutants were built for round 4; 25 fail on the assertion they target, and the two that remove COLLATE "C" from the PostgreSQL clause pass because, with legacy tables streamed unordered, the clause is equivalent on every table that is ordered (kept as defence in depth and marked so). Round 5: 18 more mutants (DeleteTable's probe on a second pooled connection, the barrier shape on the shared layout, each wake site removed, the 42P01 mapping, the fetch-error path keeping its slot, no LOCK TABLE, the last page keeping its transaction, the barrier released before the pause, the pauses moved outside the read window on SQLite and MongoDB, rowid >=, the in-memory copy dropping its slot, the MongoDB counter counting every find); 17 fail on the assertion they target, and the barrier released between the pause and the pin passes, because no statement sits between them and UpdateTable takes over a second after the barrier goes. Round 6: the poller breaking at its deadline regardless fails the plain-reader test on both layouts (two earlier shapes of that test, an 8 s and a 12 s hold with the poller started before the DeleteTable, did not catch it: the window's last drop attempt was still waiting out its lock timeout, and the DeleteTable's own wake chained a second window); the SQLite pauses moved before the transaction with a 2.5 s UpdateTable fail the branch assertion; a SQLite open blanking the ARN, id and summaries fails the description comparison; a SQLite keyset stream without its end branch fails the drain deadline.

Passes locally on PostgreSQL 16 (both pool layouts), SQLite (the shared check on a file-backed database; the in-memory copy has its own tests, since a stream that reads every page during the open fails the check's fetch-after-commit assertion by design), a MongoDB 7 replica set (devtools/run-mongodb-tests --mongo-only) and a standalone mongod, and Cassandra 4.1:

• PostgreSQL: crates/storage-postgres/tests/table_snapshot.rs, 42 tests: 19 run on both layouts (on_both_layouts!): consistency; marker and key order; legacy-collation tables streaming unordered and complete on a scratch database checked not to be C-collated, beside an ordered one; DeleteTable with only a GSI or a vector-index data table locked taking the asynchronous path; an unparseable row ending the stream and freeing its permit; a GSI without its data table left out with its attribute; DeleteTable during a snapshot returning DELETING; twenty concurrent DeleteTables on a ten-connection pool; an open held while nine DeleteTables queue; an open completing with nine of ten connections held; the production poller woken, and the table gone within 5 s, by a dropped stream, a drained stream, a fetch error and an open cancelled at BeforeItems; a plain reader holding ACCESS SHARE for 10 s past the poller's window, sending no wake, with the table gone within 3 s of its release; a fetch error freeing the permit; DeleteTable at BeforeItems taking the asynchronous path; a drained stream out of its transaction before its end is polled. Plus five held snapshots on a ten-connection pool on each layout, with GetItem and DescribeTable unblocked, the sixth LimitExceeded and an open succeeding after one is dropped; the SQLSTATE mapping raised on the live server; and, on the shared layout, a table dropped before the open's LOCK being TableNotFound. Every DeleteTable runs under a 5 s timeout. Added to the direct-storage step in integration.yml. Scratch databases are dropped by a Drop guard on success or panic. The full storage-postgres suite (220 tests), fix(postgres): build a GSI added by UpdateTable correctly while writes continue #401's GSI build tests included, passes.
• SQLite: 7 in-module tests (consistency; key order; the in-memory copy releasing its connection; in-memory copies holding their backup slot; a damaged attribute definition streaming unordered and complete at a page of seven with no duplicates and at least six fetches; slots shared with CreateBackup and its refusal text; the WAL checkpoint reply while a snapshot is held), run by cargo test --workspace; scratch files under the test binary's target directory, removed by a Drop guard.
• MongoDB: 10 in-module tests: live tests on per-test scratch databases (snap<uuid>_cat/_data, built the way fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384's live tests build theirs with the production catalog options, with the GSI backfill worker running and the command monitor feeding the counter, dropped by a guard on success or panic), gated on EXTENDDB_TEST_MONGODB_URI and failing rather than skipping when EXTENDDB_TEST_MONGODB_REQUIRED is set, holding the snapshot hooks lock (consistency; readConcernLevel=majority in the URI; an undecodable document ending the stream; reads without the snapshot level failing with code 72, with the error text checked; the round-trip counter counting only the snapshot stream, so a GSI backfill leaves it unchanged; a standalone server answering Unsupported, gated on EXTENDDB_TEST_MONGODB_STANDALONE_URI, which the CI job sets against a standalone mongod it starts, and failing rather than skipping when EXTENDDB_TEST_MONGODB_REQUIRED is set without it); and unit tests pinning the error-code mapping from reply documents. A new integration-mongodb.yml job (in the fan-in) starts a replica set through devtools/run-mongodb-tests --mongo-only and a standalone mongod, both on 127.0.0.1 only, and runs them.
• Cassandra: a direct test asserting Unsupported naming the backend on an existing table and a missing one.
• Storage: unit tests for drop_deleting_indexes (GSIs and vector indexes).
• Engine: SnapshotExpired maps to ServiceUnavailable (storage_error_mapping_tests).

cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo clippy ... -- -W clippy::pedantic                                             # 0 findings on the lines this PR adds and in the files it adds (storage crates and extenddb-engine)
cargo check --workspace --all-targets                                                # with test-hooks via dev-deps
cargo check -p extenddb-storage -p extenddb-storage-postgres -p extenddb-storage-sqlite -p extenddb-storage-mongodb -p extenddb-storage-cassandra --lib   # without
CARGO_TARGET_DIR=target/t188 cargo +1.88.0 check --locked --workspace --all-targets
cargo test -p extenddb-storage -p extenddb-engine -p extenddb-server                 # 132, 179, 202 passed
cargo test -p extenddb-storage-sqlite --lib snapshot_engine                          # 7 passed
EXTENDDB_TEST_PG_CONNECTION_STRING=... cargo test -p extenddb-storage-postgres --test table_snapshot   # 42 passed
EXTENDDB_TEST_PG_CONNECTION_STRING=... cargo test -p extenddb-storage-postgres                         # 220 passed
EXTENDDB_TEST_MONGODB_URI=... EXTENDDB_TEST_MONGODB_REQUIRED=1 EXTENDDB_TEST_MONGODB_STANDALONE_URI=... cargo test -p extenddb-storage-mongodb --lib snapshot_engine   # 10 passed
EXTENDDB_TEST_CASSANDRA=required cargo test -p extenddb-storage-cassandra --test direct_integration snapshot_engine   # 1 passed

Not changed here

• The design's max_concurrent_jobs is the caller-side bound, in the catalog PR; this PR's per-server bounds (SQLite's slot, PostgreSQL's permit) only keep snapshots from exhausting a pool.
• The in-catalog backup path in fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384 does not call this trait.
• The PostgreSQL and SQLite fetches are bounded by rows, not bytes; the byte budget is the backup job PR's (design section 7).
• On the shared PostgreSQL layout, a DeleteTable whose in-use probe lands after an open's snapshot and before its LOCK TABLE is granted takes the synchronous path, and its DROP TABLE waits behind the stream with the catalog row already gone. The window is two statements wide; closing it needs a row lock held for the whole stream (which would block UpdateTable and DeleteTable for the backup's length) or a synchronous drop under lock_timeout before the catalog commit. The module doc and DeleteTable's comment state it. A short GetItem, Query or Scan starting between the probe and the drop is not matched either.
• Attribute pruning for a left-out GSI is not shared between the snapshot and the in-catalog backup (they prune different types); backup_engine::abort_restore builds its own drop list in fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384's code.
• cargo clippy on the local 1.98 toolchain did not fire the chunks_exact lint in storage-sqlite/src/vector_search.rs seen in an earlier round; CI runs stable and 1.88.0 (test.yml).
• The *_wakes_the_poller tests keep a 5 s bound; with the poller now active while a drop is deferred, that bound no longer tells a wake (about 160 ms) from no wake (about 1 s), and a sub-second bound would be flaky on a loaded runner. The plain-reader test covers the poller's own behaviour.
• docs/design/04-component-storage.md (DeleteTable described as always asynchronous) and docs/differences-from-dynamodb.md (a 5 s default delay) are pre-existing inaccuracies not edited here.
• The CodeQL alert on this PR (cleartext-storage-database, storage-sqlite/src/backup.rs) is in a fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384 test on main that this PR does not touch.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic) for the files this PR adds
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: #396, section 3. Storage trait: SnapshotEngine is added to the StorageEngine supertrait bound and implemented by all four in-tree backends; StorageError gains SnapshotExpired.

Breaking changes

• Out-of-tree storage backends must implement SnapshotEngine and handle the new StorageError::SnapshotExpired variant in any exhaustive match.
• PostgreSQL DeleteTable returns DELETING and finishes in the background, instead of synchronously, when another session holds ACCESS SHARE on the table's data or one of its index tables at that moment (a table snapshot, the in-catalog backup, a Scan or Query, pg_dump, a plain read); the table is gone within about a second of that reader's release, and until then DescribeTable reports DELETING, ListTables lists it, and a CreateTable of the same name is refused with ResourceInUseException. The response shape is unchanged (it already reports DELETING). No wire, CLI or on-disk change.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.

[LeeroyHannigan]

I ran every test the description lists on PostgreSQL 16, SQLite, a MongoDB 7 replica set and Cassandra 4.1 (8, 5, 5 and 1 pass), built mutants of each backend's snapshot code and of the shared check, and compared head and main PostgreSQL servers for DeleteTable under concurrent readers. The SQLite snapshot excluded 900 writes and a DeleteTable committed after the open. Descriptions matched DescribeTable member by member on SQLite and MongoDB, N and B pages came back in KeyOrder::compare order, and MongoDB's expiry mapped to SnapshotExpired on a live server. The base-table DeleteTable change closes an orphan that main has. Nothing in the tree opens a snapshot yet, so no request reaches the items below today.

Eight items are inline. Smaller items:

• storage-mongodb/src/snapshot_engine.rs:18 has the read-concern sentence I commented on in #396 line 157: without the explicit level the reads fail.
• table_snapshot.rs:271 passes without COLLATE "C" on a server whose default collation is C. Assert the scratch database's datcollate first.
• devtools/run-mongodb-tests --mongo-only exits with an unauthenticated replica set published on every interface. Publish it on 127.0.0.1, as the printed URI and the CI job already assume.
• The storage-mongodb self dev-dependency turns on the crate's existing test-hooks feature, so its tests no longer build the production commit and backfill path. Give the snapshot knob its own feature.
• After a 20 s reader released, the table stayed DELETING for 51.8 s, because the retry waits for the control-plane poller's idle pass.
• The SQLite module doc should say that PRAGMA wal_checkpoint(TRUNCATE) reports success while a snapshot is held. The WAL grew to 8.67 MB for 900 items of 400 bytes.
• MongoDB snapshot errors carry a hex dump of the BSON reply. Build the message from code_name and errmsg.
• Cassandra answers Unsupported before it resolves the table, where the others answer TableNotFound. Say so in the trait's error list.

Once the inline items are fixed, I will approve.

[LeeroyHannigan]

No test checks that the definition and the items come from one instant, or that the stream reads after the writer commits. The writer starts after open_table_snapshot returns and never changes the catalog: at READ COMMITTED, or without FOR SHARE on the tables read, all 8 PostgreSQL tests pass, and with the data cursor or the index read outside the snapshot session, all 5 MongoDB tests pass. A SQLite stream that reads every page during the open, each in its own transaction, also passes, because the counter only shows fetches before the pause. Add a test-hooks pause between the definition read and the data read in which the check commits a write and starts an UpdateTable, and assert after the drain that paging::fetches() advanced past its value when committed_rx resolved; that assertion passes on all three backends as they are.

[LeeroyHannigan]

Every test table has this key shape, so the others are untested. Removing sk_s COLLATE "C" or the hash-only pk COLLATE "C" passes all 8 PostgreSQL tests, and changing SQLite's hash-only or binary range-key keyset comparison from > to >=, which repeats the last item of every page, passes all 5 SQLite tests. The head handles those shapes correctly; run the same checks on hash-only S, N and B keys and on S and B range keys at a small page size. The vector-index filter in drop_deleting_indexes is untested too: removing it passes snapshot.rs:225.

[LeeroyHannigan]

With a separate data database this bound is the whole data pool. On a ten-connection pool with ten snapshots open, get_item and delete_table each failed after 30.0 s with "pool timed out while waiting for an open connection", which reaches the client as a 500. Keep the bound below the pool in both branches. The test for this permit runs its ten opens in one join_all, and dropping the permit when open_table_snapshot returns passes all 8 tests; hold five snapshots, assert the sixth is LimitExceeded, then drop one and assert an open succeeds.

[LeeroyHannigan]

This probe binds only the base data table, while the synchronous path also drops the GSI and vector-index data tables. With a session holding ACCESS SHARE on a GSI data table only, DeleteTable committed the catalog removal and then blocked in DROP TABLE with no lock timeout; after 900 s tables had no row and both data tables remained. A Query or Scan with IndexName and the GSI backfill worker hold that lock. Probe every relation the drop touches.

[LeeroyHannigan]

On a table whose key columns keep the database collation, this clause turns the ordered read into a Seq Scan and an external merge sort (7,224 kB on disk for 100k rows at work_mem 4MB), so the first FETCH waits for the whole sort while the lock, the connection and the permit are held. The same table without the clause is an Index Scan. Recollate or index such a table before streaming, or report it unordered, and say so at line 50.

[LeeroyHannigan]

Dropping the stream outside the runtime panics on SQLite and PostgreSQL. With rows left, sqlx's PoolConnection drop spawns the connection's return and fails with "this functionality requires a Tokio context". A MongoDB stream dropped the same way returns in under a millisecond, so a caller tested on MongoDB panics on the sqlx backends. State this here for out-of-tree backends, or give the type a release path that does not depend on where it is dropped.

[LeeroyHannigan]

A document this decoder cannot read yields an error and the stream continues: with one such document the stream returned its 10 items, one Document missing item_data field error, then ended normally, where SQLite ends the stream at the error. Set done = true in this arm and say in the trait that the first error ends the stream. Separately, a standalone server refuses the first read with NotAReplicaSet (code 123), which reaches the engine as a 500 instead of the Unsupported sentence at line 121; map that code.

[LeeroyHannigan]

This is the permit create_backup waits 30 s for. With the default pool_size = 10, four held snapshots made CreateBackup wait the full 30 s and answer "Too many backups are in progress on this server (at most 4 at once)" with no backup running, and with a pool_size of 4 or less one held stream blocks every CreateBackup. Name snapshots in that message and update the pool_size doc in crates/storage-sqlite/src/config.rs.

[LeeroyHannigan]

I re-ran the items from my review against this head on live databases, on both PostgreSQL layouts, and found no item open. The one-instant check now fails for each reader shape I named, each on its own assertion; the SQLite >= keysets fail the hash-only and binary shapes; the bound is half the pool on both layouts, and ten opens in one join_all leave get_item and delete_table answering in milliseconds; a reader on a GSI, LSI or vector-index table takes DeleteTable down the asynchronous path with the row kept and every data table gone within a second of the release; a legacy-collation table of 100k rows streams unordered with a warning and no sort; a MongoDB document that does not decode ends the stream and a standalone server answers Unsupported; the plain-reader DELETING wait is 3 to 5 ms against 51.8 s before. The two COLLATE "C" mutants still pass, and the reason given holds: a C and a POSIX column sort alike on the server, so no ordered table reaches a clause that changes anything.

On the code added since: twenty concurrent DeleteTables on a ten-connection shared pool with readers on ten tables all returned within 0.52 s and left no orphan; an open holds one connection per pool on both layouts; twenty streaming snapshots did not move PutItem's p50 or p95 at 200 per second; the branch assertion fails for a slow UpdateTable and a flipped expectation; the release binary carries no test-hook symbol; both toolchains and the CI wiring are as the description says. Fourteen of fifteen mutants fail a test. Two inline items need code. On one shared database the single-transaction open holds no row lock before its LOCK TABLE, and a DeleteTable that probes in that gap commits the catalog removal and waits in DROP TABLE with no timeout: in 6 of 80 unaided open-and-delete races the data tables were present with no catalog row (0 of 40 on a separate data database). And the release wake is inert while a drop is deferred: the active loop sleeps on the timer alone, removal times with and without the wake are the same, both notify_one calls can be removed with all eight *_wakes_the_poller cases passing, and three comments and the description's "about 160 ms" describe a mechanism the code does not have. Four smaller items are inline: the one-second bound does not hold across a server restart; the poller never leaves its one-second loop after a drop failure that persists; the troubleshooting query misses a reader of an index table and states the wrong cadence; and the conformance check clears restore_summary on a false premise. The control_plane_delay_seconds row keeps a default of 5 where the schema inserts 0.25, and the new sentence applies at the shipped default; that text predates this revision and is not a condition.

Once the inline items are fixed, I will approve.

[LeeroyHannigan]

On one shared database the open no longer holds the tables row, so a DeleteTable that probes between the open's first read and its LOCK TABLE takes this synchronous path, commits the catalog removal, then waits in DROP TABLE behind the stream with no lock timeout. I started an open and a DeleteTable of the same table together (the delete 0 to 4 ms later), held the stream 3 s and gave the caller a 2 s timeout; in 6 of 80 races the base and GSI data tables were present after the stream ended and a control-plane pass ran, with no catalog row, and on a separate data database 0 of 40 were. The catalog and the data are one database here, so run the drops in this transaction under a short lock_timeout before the removal commits, and take the asynchronous path when it times out.

[LeeroyHannigan]

While a drop is deferred, the poller stays in its active loop, which sleeps in sleep_or_shutdown on the shutdown token and the one-second timer only, so it does not see this notify until it goes idle. With a snapshot held 12 to 14 s past DeleteTable and then dropped, the table was gone after 956, 40 and 13 ms on the separate layout and 911, 12 and 11 ms on the shared layout, the same spread as a plain reader that sends no wake, and removing the notify from release or from finish passes the eight *_wakes_the_poller cases. Either have the active loop wait on the notify as well as the timer, or correct this comment, the module doc at line 78, the in-catalog backup's comment and the description's "about 160 ms", which say the drop is retried at once.

[LeeroyHannigan]

After a restart this bound does not hold, because startup recovery runs one process_control_plane_transitions pass, which discards the deferred count, and the poller then starts in its idle wait. A 40 s reader that spanned a restart left the table DELETING for 37.2 s after its release, until the idle pass 60.0 s after the startup pass, on both layouts. Use control_plane_pass at startup and wake the poller when it reports a deferred drop.

[LeeroyHannigan]

Every failed drop is counted here, so a drop that fails for a reason that persists keeps the poller in its one-second loop indefinitely. With a view on the data table, 120 s on the shared layout gave 0.97 failed drops per second, each logging this warning and each pass opening a restore-sweep connection, where before this change the poller went idle after its window and retried every 60 s. Count only lock timeouts (SQLSTATE 55P03) as deferred, or back off after repeated failures of another kind.

[LeeroyHannigan]

All eight cases of these tests pass with both notify_one calls removed from HeldSnapshot, and the removal times at the head match the plain reader's, which sends no wake (35 to 142 ms against 37 to 139 ms). The poller stays active while the drop is deferred, so the drop waiting in the lock queue is granted at the release whether or not the wake fires, and the in-catalog backup's wake has no test at all. Name the tests for what they check and drop the wake from the description's claims, or make the wake observable (the comment at snapshot_engine.rs line 155) and assert it.

[LeeroyHannigan]

This query matches only the base data table, while DeleteTable defers on a reader of any of the table's relations; with a session holding only the GSI data table, DeleteTable returned DELETING and this query returned no rows on both layouts. Match every relation of the table (_ddb_<index_id> for each indexes row and _ddb_vec_<index_id> for each vector_indexes row) and add AND granted. The text says the poller retries every second; with a reader holding the table it tries about every 4 s (a 3 s lock wait, then a 1 s pause).

[LeeroyHannigan]

Storage sets restore_summary (build_table_description_on fills it on PostgreSQL and SQLite, and MongoDB reads it from the tables document), which is why the trait doc says it may be present on a restored table. Clearing it here lets a snapshot description that drops or misreports it compare equal. Compare it, and keep the clearing for the counts and table_throughput_mode_summary.