[robinnsc]

What

Backup and restore now behave the same way on all four storage backends (PostgreSQL, SQLite, MongoDB, Cassandra), and closer to DynamoDB, on the points the round-2 and round-4 reviews measured against the service.

• The engine resolves a backup into one validated CreateTableInput before any backend sees it. It runs the multi-part-key and vector-index refusals, applies the overrides, and runs validate_create_table on the result, for every backend. Backends now only create the target, register the restore, and copy. Cassandra (Feat/storage cassandra in tree #338) previously skipped the refusals and dropped the secondary indexes.
• RestoreTableFromBackup returns the target CREATING and copies in a background task on all four backends, so a client that times out or disconnects no longer loses the restore or locks the name and the backup. Once the request has been authenticated and authorized, the engine runs the rest of it (reading the backup, validation, registering the restore) in a task of its own, so a client that disconnects before the response no longer cancels it part-way. A disconnect while the request is still being authenticated or authorized cancels it with nothing created (see Not changed here).
• While a target is CREATING, DeleteTable on it is ResourceInUseException and DeleteBackup on its source is BackupInUseException. A target whose server stopped mid-copy is removed without operator action, and each removal is logged with the table name:
  • PostgreSQL: on a control-plane pass after 60 s.
  • SQLite: at the next start.
  • MongoDB and Cassandra: by a heartbeat sweep that runs at startup and every 15 s, about 45 to 80 s after the owner stopped (the last heartbeat can be up to one 15 s interval before the stop).
• RestoreSummary matches the service. RestoreDateTime is the backup's creation time and SourceTableArn is set. No response that describes a table (CreateTable, UpdateTable, DeleteTable, DescribeTable, the restore response) carries RestoreSummary unless the table is CREATING; one rule in TableDescription::prepare_for_response applies it to all of them.
• DescribeBackup reports SourceTableDetails.ProvisionedThroughput, which is required in the shape and {0, 0} for on-demand. It also reports SourceTableFeatureDetails: GSIs, LSIs, TTL, stream, and SSE. A definition version this build does not know is described without feature details; any other failure to read the definition (a row that does not decode, a lost connection) is an internal error, as it is for RestoreTableFromBackup, instead of a description that hides it.
• DeleteBackup returns the full DescribeBackup description with BackupStatus DELETED, including SourceTableDetails.ProvisionedThroughput and SourceTableFeatureDetails. It alone tolerates a definition that cannot be read (it answers {0, 0} and no feature details, and logs a warning), so a damaged backup stays deletable; this is ExtendDB's choice, stated in the docs.
• ListBackups honours Limit (1 to 100, default 100) and ExclusiveStartBackupArn, and returns LastEvaluatedBackupArn. Summaries carry TableId. It filters on BackupType (USER when absent, every type for ALL; every ExtendDB backup is USER) and on TimeRangeLowerBound (inclusive) and TimeRangeUpperBound (exclusive), before paging. The cursor is checked as the service checks it (measured on DynamoDB): one from another account is ValidationException "Invalid Request: The supplied ExclusiveStartBackupArn does not match the account", one from another region or another table than the TableName filter is refused with the same shape of text, a time bound before the epoch is refused, and a lower bound above the upper bound is "Time range lower bound must be less than or equal to upper bound".
• BackupName and TableName on CreateBackup, and TableName on ListBackups, are validated as the service model says: 3 to 255 characters and [a-zA-Z0-9_.-]+, with the service's "1 validation error detected: Value '' at 'backupName' failed to satisfy constraint: ..." texts, joined under "2 validation errors detected:" when both fail (the order of the two clauses was not captured; the length clause comes first). TableName takes the deployment's configured table-name limits. Before this a 200 KB BackupName was stored and returned in every ListBackups page (a 500 on Cassandra) and a NUL byte was a 500 on PostgreSQL. CreateBackup accepts TableName as the table's ARN (measured on DynamoDB; botocore refuses the ARN form client-side, so only a hand-built request sends it); CreateBackup on a table that does not exist or is still CREATING is TableNotFoundException "Table not found: " (measured) on all four backends.
• Restoring into an existing name is TableAlreadyExistsException; into a name that is itself a CREATING restore target, TableInUseException (measured on DynamoDB); the text is the service's "Table: is already being restored from backup: " whichever backup is restored, where the service answers "Attempt to change a resource which is still in use: Table: is being used." for a different backup. A second RestoreTableFromBackup of a backup while a restore from it is still CREATING is BackupInUseException (measured). Both are decided inside the registration transaction (PostgreSQL, with the backup row locked FOR UPDATE; SQLite), the claim (MongoDB), or the lightweight transaction (Cassandra), so two simultaneous requests cannot both pass; the name is checked before the backup and before a restore slot is taken, so the same backup restored again into its own CREATING target's name is TableInUseException at once on all four, also while the account's restore slots are all in use. A restore of a backup whose DeleteBackup is in progress answers BackupInUseException on MongoDB and BackupNotFoundException on the other three (documented; the service's answer was not measured). A plain CreateTable into a CREATING restore target stays ResourceInUseException "Table already exists: ".
• BackupArn and ExclusiveStartBackupArn are checked as the service checks them (measured on DynamoDB): a string that is not arn:aws:dynamodb:<region>:<account>:table/<name>/backup/<id> with a region in [a-z0-9-], a digit-only account, a name in [a-zA-Z0-9_.-], over 1024 bytes, or carrying a control or non-ASCII byte is ValidationException "Invalid Request: BackupArn is not valid"; a well-shaped ARN whose id is not a backup id is "Invalid Backup ARN" (which text goes with which input was captured for one input only, a bare 13-digit id answered "not valid"; the rest of the pairing is ExtendDB's). Surrounding ASCII whitespace is trimmed (the service was measured trimming a trailing newline). An ARN of the caller's account in another region resolves: the engine looks the backup up under the ARN rebuilt in the server's region first and, when the ARN as written differs, under that spelling, so backups created under an earlier server.region stay reachable by the ARN ListBackups reports (not by that ARN rewritten with the new region), and ListBackups accepts such a backup's stored ARN as ExclusiveStartBackupArn, so paging passes it. A server.region outside [a-z0-9-] logs a startup warning naming the consequence. The ARN is checked before the target name; another account's is AccessDeniedException; another partition's is ValidationException. Before this, every malformed shape resolved to BackupNotFoundException and a NUL byte was a 500 on PostgreSQL.
• Overrides:
  • SSESpecificationOverride is applied.
  • GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, and OnDemandThroughputOverride are applied as the service applies them (measured on DynamoDB): an empty list restores no indexes of that kind, a subset restores those indexes and prunes the attribute definitions only they used, a GSI entry's ProvisionedThroughput is the restored index's throughput, a one-member or changed OnDemandThroughputOverride is the restored table's on-demand throughput (including a PROVISIONED backup restored as on-demand), and the on-demand-to-PROVISIONED route works with per-index throughput in the list. Refused as on the service: an unknown or changed index definition (key schema or projection), a throughput member contradicting the billing mode, {0, 0}, and a PROVISIONED restore of a GSI without throughput, with the service's text "Invalid Request: Must override ProvisionedThroughput if BillingMode is overridden to PROVISIONED for index ": that refusal fires whenever BillingModeOverride=PROVISIONED is sent with a GSI and no list entry supplies its throughput, whatever the backup's billing mode or recorded throughput (Lee's measurement was on an on-demand backup; the PROVISIONED-backup case follows the service's own message and was not measured), and when a list entry omits throughput on a PROVISIONED restore without the override (text not measured). A GSI entry's OnDemandThroughput is accepted and not recorded (ExtendDB keeps no per-index on-demand maximum). NonKeyAttributes are compared as a set.
  • TableClass is no longer carried by a restore: the target is STANDARD, as on the service (measured). The backup definition keeps the class.
  • The RestoreTableFromBackup response no longer carries DeletionProtectionEnabled or the table's ProvisionedThroughput.NumberOfDecreasesToday, which the service omits there; a GSI's NumberOfDecreasesToday is kept (not measured).
  • The refusal of an on-demand backup with GSIs restored as PROVISIONED now says what works: restore with PAY_PER_REQUEST, then switch with UpdateTable.
• A PROVISIONED backup whose GSI has no provisioned throughput restores, and that GSI takes the table's throughput (or ProvisionedThroughputOverride's). ExtendDB's CreateTable accepts such a table and UpdateTable to PROVISIONED produces one; the service cannot produce such a backup, so the choice is ExtendDB's. The earlier restore-only stopgap that refused these backups is removed.
• CreateTable and UpdateTable now apply the two checks the stopgap applied only at restore: a GSI ProvisionedThroughput below 1 on a PROVISIONED table (and on UpdateTable's GSI Create) is ValidationException, and so are more than 100 projected NonKeyAttributes across a table's GSIs and LSIs. Restore runs the same checks through validate_create_table.
• CreateBackup of a table with vector indexes is refused up front. It no longer returns an AVAILABLE backup that can never be restored.
• Backup ids use one shared shape on every backend: 14-digit zero-padded epoch milliseconds, then 8 hex digits. BackupCreationDateTime is that millisecond. The ids earlier builds issued (13 digits plus the hex suffix from v0.1.3 to v0.1.14; bare 13 digits from PostgreSQL v0.1.0 to v0.1.2) still resolve; they carry no time, so one that has been deleted or filtered out cannot serve as a ListBackups cursor.
• The first read on a restore target after DescribeTable reports ACTIVE succeeds (measured on DynamoDB). The server's key-info cache no longer stores a negative entry for a name the catalog has in a non-ACTIVE state: on a not-found answer it asks the catalog once more with DescribeTable and caches nothing. Authorization (main's fix(auth): enforce dynamodb:LeadingKeys on Query, batch and transaction operations #418) records such a name as missing in the key-info snapshot it hands the engine, and fails closed on any other lookup error, so a request against a CREATING name costs one table_key_info and one describe_table load (measured on SQLite with this PR's earlier hand-off, which fix(auth): enforce dynamodb:LeadingKeys on Query, batch and transaction operations #418's snapshot replaces: 40 and 41 catalog loads for 40 refused GetItems). Before this a read during the copy poisoned the cache and reads failed for up to 5 s after ACTIVE; CreateTable targets had the same lag and get the same fix.
• PostgreSQL:
  • The copy commits per batch, with one multi-row INSERT per index.
  • The restore lock session is heartbeated.
  • One server process runs at most 4 restores per account and at most half of pool_size in total (capped at 16; 10 with the default pool_size). A restore that cannot get both within 30 s is LimitExceededException. The whole restore, admission included, runs in a task the request only waits on, so a client that disconnects during the wait does not cancel it; the account slots belong to the engine, not the process.
  • extenddb migrate finds the data database through the name init recorded in the catalog's settings, so a deployment initialised with --data-db <other-name> migrates without further flags, and accepts --data-db to override it. Before, it derived the name from the catalog database name and failed with "database does not exist".
  • Failing to open the restore lock connection is LimitExceededException only when PostgreSQL refuses it with too_many_connections (SQLSTATE 53300); any other failure is an internal error.
  • serve refuses a catalog whose schema_history lacks one of the binary's catalog migration files; the message names the files and extenddb migrate --yes. The stored version alone does not show this case, because another migration of the same number can have stamped the same version.
  • A catalog created by a pre-release 0.0.4 build (no table_restores.source_table_arn) gets the column at serve startup, with a warning. That step waits at most 5 s for the table lock (lock_timeout) and otherwise refuses to start with the migrate advice. serve no longer drops the pre-release restore_date_time default, which a still-running earlier-build server relies on; only migrate --yes does.
  • The unreachable restore_table_to_point_in_time is removed from the trait and from every backend.
• SQLite:
  • serve refuses to start a file database whose tables are in a pre-release 0.0.4 shape, and names the step and extenddb migrate --yes. The upgrade's backup_items rebuild is named in the preview with its row count and logged at start and finish. The pool_size clamp is logged.
  • CreateBackup is admitted against the pool: at most pool_size / 2 - 1 backups at once (4 with the default of 10; each holds its reader and its writer), LimitExceededException after a 30 s wait; a refused backup writes nothing. Before this, a flood of CreateBackups took every pooled connection and every other request waited behind them.
  • A GSI added by UpdateTable with ProvisionedThroughput on a PROVISIONED table made every DescribeTable of that table fail (Internal("index pt: missing field NumberOfDecreasesToday")). The index's throughput is now written in the shape DescribeTable reads, and rows earlier builds wrote are read as the same units with NumberOfDecreasesToday 0, so such a table is describable again after the upgrade without touching the database file.
• MongoDB:
  • DeleteBackup, and the sweep finishing an interrupted delete, no longer release their claim once the backup's data may have been dropped. An error or cancellation after that point leaves the backup AVAILABLE and in use until the sweep finishes the delete; before, it could be left AVAILABLE with no data, and a restore of it produced an empty table.
  • A DeleteBackup that finds the backup claimed by another delete is BackupInUseException ("Backup is being deleted"), where it was BackupNotFoundException.
  • The engine opens the catalog database with read preference primary, read concern local, and write concern majority whatever the connection string says, and the $out copies of CreateBackup and restore are written with w: majority, so the claim protocol's reads see every write the primary acknowledged and an acknowledged claim survives a failover. A connection string that sets a weaker write concern (w=0, a numeric or tagged w, journal=false) is refused at startup, as a non-primary read preference already was.
  • A restore holds its claim on the backup only until the target's catalog document is inserted. A concurrent restore of the same backup that finds the claim held retries for up to 2 s and then fails with BackupInUseException. Before, it failed at once with BackupNotFoundException.
  • Heartbeats refresh at once rather than after the first interval, and a failed refresh no longer skips the others.
  • The sweep leaves a stale delete claim alone while a restore from that backup is CREATING, and finishes the delete afterwards.
  • Twenty concurrent RestoreTableFromBackup requests no longer wedge the server for every later request (reproduced on the previous build with 20 restores of 20 distinct backups: 6 answered, 14 timed out after 60 s, and every later request timed out for the rest of the run; 20 restores of one backup did not wedge, because one backup's claim serializes them). The registration's hand-over left createCollection's pooled connection checked out while it waited for another connection, and the Rust driver never times a checkout out (waitQueueTimeoutMS is not supported in mongodb 3.8.0), so with ten such restores the pool was exhausted for good. Both futures are now polled together. Restores are admitted per account (4) and per server (min(16, pool/2), where the pool is the engine client's maxPoolSize: max_connections, or the driver default of 10, so 5), with LimitExceededException after 30 s; a request whose registration takes longer than 60 s answers an internal error while the restore continues and DescribeTable reports its target once it exists. The whole restore, admission included, runs in a task the request only waits on. The one-restore-at-a-time exclusion on a backup is bounded by the heartbeat: a restore that stalls between its claim and its insert for longer than the stale window loses the claim to the sweep and can be joined by a second restore of the backup (documented; PostgreSQL and Cassandra have no such window).
  • UpdateTable on a CREATING restore target is refused (ResourceInUseException, as on the other three backends; the service answers ResourceNotFoundException). Before, it changed the target mid-restore: billing mode, a third GSI, a stream, and deletion protection were accepted and the finished table was not what the backup described.
  • BackupInUse texts match the other backends ("Backup is being used to restore table : ", or "... to restore another table: " while the claim is held and the target is not yet known).
  • The test-only protocol model covers the new steps (two restores, the delete and its stop points, the sweep's delete takeover) and checks liveness as well as safety. The four tests that explore the full model take about 25 s each in a debug build and are #[ignore]d; cargo test -p extenddb-storage-mongodb --lib protocol_model -- --include-ignored runs them, and the module doc says when to. Its doc comment now says that RRelease is one atomic step for the release, the lost-claim recheck, and the abort, and lists what is not modelled (release failures, the sweep's pre-takeover target check).
• Cassandra:
  • An on-demand GSI now reports ProvisionedThroughput {0, 0} in DescribeTable, and a GSI added by UpdateTable with throughput reports it.
  • An ordinary CreateTable no longer names the V005 restore columns, so an unmigrated catalog breaks only CreateBackup, RestoreTableFromBackup, and DeleteBackup.
  • The restore heartbeat starts when the target row is inserted, before the target's tables are created, and stops on every exit from the copy (an abort-on-drop guard), including a panic.
  • On a catalog that has not been migrated to V005, CreateBackup, RestoreTableFromBackup, and DeleteBackup are a ValidationException naming extenddb migrate --yes, instead of an internal error; a running server notices the migration without a restart.
  • UpdateTimeToLive enabling TTL on a table with an asynchronously propagated GSI is a ValidationException naming the table, the index, and its delay, instead of an internal error that SDKs retry (the engine now maps a backend's validation refusal in the TTL handlers).
  • extenddb init writes the application role's username and password into the generated config, so serve starts against a PasswordAuthenticator node without editing it. The config file is created with mode 0600 instead of being written and then narrowed.
  • The abandoned-restore sweep runs once at startup and stops on shutdown.
  • The restore registration's lightweight transaction requires backup_status = 'AVAILABLE' AND active_restores = null, so a second restore of a backup whose first restore is CREATING is BackupInUseException and a restore into a CREATING restore target is TableInUseException, as on the other backends. A restore whose registration attempts are all taken by a restore that was just starting answers BackupInUseException ("another restore of this backup is starting; retry") instead of "Backup not found".
• Backup and restore authorization is main's (fix(auth): authorize resource-ARN operations against the resource they act on #419, released in v0.1.14): every backup operation is evaluated on the resource the service evaluates, canonicalized to this deployment. This PR adds one thing on top: a BackupArn is trimmed of surrounding ASCII whitespace before it is canonicalized, as the engine trims it before resolving the backup, so a Deny on a backup ARN cannot be bypassed by sending the ARN with a trailing newline (unit test backup_arn_with_surrounding_whitespace_authorizes_the_trimmed_backup). The engine resolves a BackupArn of the caller's account under the server's region first and then as written, which is the same table and backup id that authorization evaluated. tests/test_auth_permissions.py::TestBackupAuthorization covers ListBackups per table, the restore's seven item actions, and a tagged existing target (an aws:ResourceTag Allow: TableAlreadyExistsException; without the tag: AccessDeniedException) in authenticated mode on all four backends.
• Catalog version guards (PostgreSQL converge, SQLite apply, extenddb migrate) refuse a stored version they cannot parse instead of overwriting it, and serve prints the same refusal line as migrate instead of wrapping it as an internal error.
• extenddb migrate:
  • The preview without --yes exits 3 when work is pending and 0 when up to date. It used to exit 1 with an error. This covers main's resource-tag condition-key rewrite (fix(auth): evaluate resource tags only as aws:ResourceTag, matching DynamoDB #421) too, whose detection and apply step now sit behind a small ResourceTagStep trait so the exit-status tests can run without a catalog; main's tests/test_resource_tag_migration.py is updated to expect exit 3 for its preview.
  • On PostgreSQL it treats any catalog migration file the ledger does not record as pending, even when the stored version already matches, and applies it with --yes. It lists such files under data migrations as catalog <file>.
  • A failed admin connection is one plain line that names the role it tried and says to pass --pg-user and EXTENDDB_PG_PASSWORD. Before, it read Internal("Internal(\"Cannot connect as admin: ...\")"). init and destroy print bootstrapper errors the same way.
  • It accepts --cassandra-user and --cassandra-pass, and reads the password from EXTENDDB_CASSANDRA_PASSWORD when the flag is absent (so does init). Without them it could not migrate a Cassandra deployment that uses PasswordAuthenticator.
• Build and CI: extenddb-app gains an async-trait dev-dependency. .github/workflows/integration.yml adds --test backup_restore to the PostgreSQL storage-level test step. devtools/run-cassandra-tests runs the fidelity file's large-backup tests with 10,000 items unless EXTENDDB_TEST_BACKUP_ITEMS is set (see Not changed here).
• Docs:
  • docs/differences-from-dynamodb.md: the Backup and Restore section, rewritten for four backends, with a seven-member overrides table; rows that rest on the code rather than a running server are marked. Two Secondary Indexes rows for the new CreateTable/UpdateTable checks. New rows for BackupArn shapes, name validation, the cursor checks, reads on a restore target, UpdateTable on a CREATING target, what a restore carries across, backup contents, and the authorization resources.
  • docs/dynamodb-limits.md: the backup rows; projected attributes are now enforced. docs/technical-debt.md: that limit removed from the unenforced list.
  • docs/manuals/07-upgrade-manual.md: parameterized backup and rollback commands, the migrate credentials and exit codes, and PostgreSQL, SQLite, Cassandra, and MongoDB subsections for 0.0.4 (including serve's refusal of unrecorded catalog files, the 5 s lock wait, and MongoDB's read and write concern).
  • docs/manuals/05-admin-guide.md: [storage.sqlite]; the PostgreSQL restore limits and the session-mode pooler requirement.
  • docs/manuals/01-architecture-guide.md, docs/design/13-storage-mongodb.md (restore flow; catalog read preference and concern), and the version literals in the getting-started and install guides.

Why

Round-2 review by Lee Hannigan, which requested changes. It found four blocking problems:

• The merge with Feat/storage cassandra in tree #338 did not compile, and Cassandra's restore skipped the refusals and the indexes.
• PostgreSQL and SQLite copied inside the request, so a client timeout lost the restore (PostgreSQL) or locked the name and backup until restart (SQLite).
• DescribeBackup returned the pre-PR subset.
• The PR body and the docs stated behaviour the build did not have.

The review also listed wire differences measured against DynamoDB, PostgreSQL robustness issues, tenancy issues (a process-global restore limit, unvalidated stored definitions, and catalog guards that failed open), authorization of backup operations against table/*, and the upgrade-path problems in the manual. A further review of this round found the MongoDB delete-claim release, and the shapes that CreateTable accepted and restore refused.

Round-4 review by Lee Hannigan (sent directly) on the previous revision found 19 further problems, all addressed here: twenty concurrent MongoDB restores wedged the server; a SQLite CreateBackup flood took the whole pool; MongoDB UpdateTable changed a CREATING restore target; DescribeBackup masked database failures; reads on a restored table kept failing for seconds after ACTIVE; PostgreSQL migrate could not find a data database named at init; BackupName, TableName, BackupArn and the ListBackups cursor were not validated as the service validates them; CreateBackup of a missing table, a restore into a CREATING target, and a second restore of a backup gave the wrong errors; DeleteBackup returned a partial description; TableName as an ARN was refused; index and on-demand overrides the service applies were refused, and two requests the service refuses were accepted; TableClass was carried across; and the docs and test comments narrated the review. It also listed pre-existing issues, of which the SQLite GSI-throughput storage shape and the authorization resources of ListBackups and RestoreTableFromBackup are fixed here (the others are under Not changed here).

Open questions for the maintainers:

• Migration order. fix(backup): report point-in-time recovery as unsupported #335 and Store U+0000 in strings on PostgreSQL and MongoDB #368 also add a PostgreSQL 003_*.sql that writes catalog 0.0.4. Whichever merges second has to renumber to 004_*.sql and 0.0.5 (and the migrations.rs assertions and the manual's section). The design in docs(backup): design revision for off-host backup and restore [backup/restore 1/4] #396 assumes this PR keeps 003 / 0.0.4 and fix(backup): report point-in-time recovery as unsupported #335 is rebased to 004 / 0.0.5, and reserves 005 / 0.0.6 for a later PR of the series; it does not mention Store U+0000 in strings on PostgreSQL and MongoDB #368. Is that the order you want, and where does Store U+0000 in strings on PostgreSQL and MongoDB #368 go?
• With this PR, migrate reports an unrecorded catalog migration file as pending even at the current version and applies it, and PostgreSQL serve refuses to start until it has, so after a renumbering the second PR's file is applied rather than skipped. The numbering itself still has to be settled.

Testing done

Release binaries were built from the code in this PR, one per backend feature, and served on 127.0.0.1 against PostgreSQL 16, mongo:7 (single-node replica set), and cassandra:4.1 with PasswordAuthenticator, all on one host. Timings compare backends; they do not predict a networked deployment. The fidelity table below ran on those binaries; the large-backup timings and the further checks after them ran on earlier builds of this branch, as each says.

tests/test_backup_restore_fidelity.py, tests/test_ttl.py, tests/test_abac.py, and tests/test_auth_permissions.py, each run in full on release binaries built from this PR's head after its rebase onto main (b5bffa38, v0.1.14), plus main's tests/test_authz_resource_arn.py (Cassandra with index_propagation_delay_ms set to 0, as CI runs it; see #407):

Backend	test_backup_restore_fidelity.py (37)	test_ttl.py (11)	test_abac.py (69)	test_auth_permissions.py (63)	test_authz_resource_arn.py (43)
PostgreSQL	37 passed	11 passed	69 passed	63 passed	43 passed
SQLite	37 passed	11 passed	69 passed	63 passed	43 passed
MongoDB	36 passed, 1 skipped (no vector indexes)	11 passed	69 passed	63 passed	43 passed
Cassandra	36 passed, 1 skipped (no vector indexes)	11 passed	69 passed	63 passed	43 passed

test_abac.py and test_authz_resource_arn.py include the tests main's #418 to #421 added. In the full run the three TestBackupAuthorization tests failed on every backend because their message check expected the resource to end the AccessDeniedException text, and main's evaluator now appends the reason ("because no identity-based policy allows ...", "with an explicit deny ..."); with the check loosened to accept a suffix, the class passed 3/3 on all four backends on the same binaries. tests/test_resource_tag_migration.py (main's, SQLite only) passed 2/2 under Python 3.12 once its preview assertion was changed to this PR's contract (exit 3, "run again with --yes to apply"); the host's Python 3.7 sqlite3 module cannot read the catalog's partial index.

Earlier passes in this round found three defects, fixed before the run above: test_first_read_after_active_succeeds raced the CREATING to ACTIVE transition on PostgreSQL (its copy of 60,000 items takes about 1.5 s; the test now accepts a read that succeeds once DescribeTable confirms ACTIVE); and SQLite, MongoDB, and Cassandra answered BackupInUseException where the service answers TableInUseException for a restore of a backup into its own CREATING target's name, because they checked the backup before the name (PostgreSQL checked the name first). All four now check the name at the storage layer before the backup and before a restore slot is taken; storage-level tests on each pin the same-backup case and a missing backup restored into the name, and on PostgreSQL and MongoDB a retry answered at once while every account slot is held. The engine still reads the backup before the storage call, so at the wire a missing backup is BackupNotFoundException whatever the name; the storage order matters when a backup is deleted between the engine's read and the registration, and for a retry. One Cassandra pass ran against a node that had accumulated about 1,640 tables over two days of test runs and was pausing over 80 s per young-generation GC (from the node's own log); four of its five failures were copies or teardowns that timed out, the fifth was the ordering defect. The runs above used a cassandra:4.1 container created the day before them.

The large-backup tests use a 60,000-item on-demand backup of a table with only a partition key and no secondary indexes (secondary indexes are covered by the smaller tests; with GSIs on 60,000 items the fixture kept a backend's GSI queue or backfill worker busy for minutes and slowed other modules' GSI tests):

• test_restore_response_arrives_before_its_copy_finishes: a client with a 10 s read timeout and no retries gets the CREATING response within the timeout, the target is seen CREATING while it is polled, and it reaches ACTIVE with every item. It shows the response does not wait for the copy; it does not drop the client.

  Measured on an earlier build of this PR:

  Backend	Response	ACTIVE after
  PostgreSQL	0.031 s	1.52 s
  SQLite	0.017 s	4.67 s
  MongoDB	0.082 s	3.90 s
  Cassandra	0.147 s	10.65 s

  Cassandra's copy writes every item through the ordinary put path (a read and a lightweight-transaction write per item, 8 at a time, with the index writes synchronous); that is why it is several times slower than the others. A restore-only plain-insert path is storage-cassandra: restore copy writes every item through the LWT put path #404.

  On Cassandra the response waits for the target's schema. With five restores at once it took 0.75 to 1.3 s on an earlier build, and inside the full Cassandra pytest suite it once exceeded 1 s, which is why the test's client timeout is 10 s; the check that the copy runs after the response is that the target is still CREATING when polled.

• test_restore_survives_a_client_that_disconnects_before_the_response: a client with a 1 ms read timeout sends the request and closes the connection before any response. The test accepts either outcome that leaves no half-made state: the restore was registered and finishes whole, or the request was cancelled before registration and a new restore into the same name from the same backup is accepted and finishes whole. It prints which. On all four backends the restore was registered and finished. The engine unit test a_restore_request_dropped_while_the_backup_is_read_still_restores drops the request future while the backup is being read and checks that the restore still runs.

• test_target_and_backup_are_held_while_the_copy_runs: DeleteBackup (ExtendDB only) and DeleteTable during the copy. The status is read immediately before each probe; a probe on a CREATING target must be refused, and each probe must land in the window. On all four backends both probes were refused while the target was CREATING, and the restore then finished whole.

Other new or changed tests: DescribeBackup feature details from an on-demand source and from a PROVISIONED source with SSE (throughput values; the SSEDescription assertion is ExtendDB only); ListBackups with Limit=1 over three backups (exactly one summary per page and three pages, ExtendDB only; the service's contract allows a cursor on the last page); TableAlreadyExistsException; malformed and foreign BackupArn; empty index overrides, the SSE override, and no SSEDescription without it; CreateBackup of a vector table; no RestoreSummary on UpdateTable of an ACTIVE restored table; (ExtendDB only) restore of a PROVISIONED backup whose GSI has no throughput; ListBackups filtered by BackupType and a time range; index overrides built from DescribeBackup (as the console does) and an equal OnDemandThroughputOverride, both restoring; a BackupArn in another partition; and no DeletionProtectionEnabled in the restore response. The DynamoDB-branch message in test_provisioned_restore_override_requiring_gsi_override_leaves_no_target is the service's reply as measured during review (Invalid Request: Must override ProvisionedThroughput if BillingMode is overridden to PROVISIONED for index by_gpk). tests/test_abac.py::TestBackupResourceTags reproduces the TagResource case from the review in authenticated mode: a user allowed DescribeBackup only on aws:ResourceTag/classification = dev cannot tag the backup and stays denied (main's #419 refuses the tag operations on a backup ARN; the test pins that against this PR's backups). tests/test_ttl.py gains a TTL-with-GSI test: it requires TTL to be enabled on PostgreSQL, SQLite, and MongoDB (passed on each, with the backend given by the new EXTENDDB_TEST_BACKEND that devtools/run-tests --backend exports), and on Cassandra accepts the refusal, which it got at the default delay (ValidationException naming by_gpk and 10 ms). The Cassandra refusal is pinned by the live storage-level test test_ttl_on_a_table_with_an_async_gsi_is_refused_as_a_validation_error (refused at 50 ms; enabled once the GSI is synchronous). test_index_overrides_listing_the_backups_indexes_restore_them runs for a PROVISIONED and an on-demand backup and passed on all four backends. tests/test_ttl.py::test_disable_ttl now waits for ENABLING to end before it disables TTL; it failed once on a loaded PostgreSQL run with "Time to live is being updated".

Checks that only ExtendDB can satisfy, and assertions not measured on the service, are skipped against DynamoDB. The file was not run against DynamoDB in this round.

Against older builds: on the round-2 build (8473dd0e), on PostgreSQL, 15 of 19 earlier tests failed, among them the response test with ReadTimeoutError. On SQLite, the round-3 build before this pass's engine fixes fails the UpdateTable test (RestoreSummary with RestoreInProgress: false in the response) and the GSI-throughput test (ProvisionedThroughput must be specified for index).

Further checks with built binaries, all four backends unless noted, on earlier builds of this branch (not repeated on the final binaries):

• A SIGKILL mid-restore followed by an immediate restart. The target was removed after 60 s on PostgreSQL, 0.1 s after the restart on SQLite, 60 s on MongoDB, and 62 s on Cassandra. Afterwards DeleteBackup and a new restore into the name succeeded.

• A writer updating an early item and then a late item during CreateBackup. 0 pairs out of order on PostgreSQL and SQLite; 16 of 66 on MongoDB and 123 of 800 on Cassandra. Neither MongoDB nor Cassandra takes a point-in-time backup; the docs say so.

• Concurrent restores on PostgreSQL: with four restores running in one account, a fifth in the same account waited for a slot (3.1 s) while another account's restore answered in 0.05 s.

• Upgrades from origin/main (catalog 0.0.3) with extenddb migrate:

  • PostgreSQL: preview exit 3, --yes exit 0, preview again exit 0. A table left CREATING was removed at startup and logged by name. Rolling back with the manual's parameterized dump and restore brought the 0.0.3 binary back. A catalog in the pre-release 0.0.4 shape was converged by serve and by migrate --yes.
  • SQLite: the rebuild was named in the preview with its row count and logged on apply. Rolling back by restoring the copied files brought the 0.0.3 binary back.
  • Cassandra: on an unmigrated catalog, a 0.0.4 server served every table and item operation tried, and ListBackups and DescribeBackup worked; CreateBackup, RestoreTableFromBackup, and DeleteBackup answered ValidationException "Backups are unavailable until the ExtendDB catalog is migrated: it does not record catalog migration V005..." naming extenddb migrate --yes (re-measured on the final binary; an earlier build returned internal errors). After migrate --yes, all of them worked.

  Backups taken before the upgrade restored afterwards on PostgreSQL, SQLite, and Cassandra.

• PostgreSQL serve with a debug build of this round's PostgreSQL changes: on a catalog whose ledger names another 003_*.sql at 0.0.4, it exits 1 naming 003_backup_definitions.sql and extenddb migrate --yes; after migrate --yes it starts. On a pre-release 0.0.4 shape with another session holding a transaction on table_restores, it exits 1 after about 6 s with could not lock the table within 5 s; once the lock is free it adds the column, keeps the restore_date_time default, and starts, and migrate then reports only the default step.

• The migrate error text with no admin credentials and with a wrong password (PostgreSQL), and the SQLite pool_size log line.

The docs mark which statements rest on the code or on storage-level tests rather than on a running server, and which numbers come from an earlier build.

Rust, on the final tree:

• cargo test --workspace without the PostgreSQL storage crate and the Cassandra live test targets: all passed. extenddb 1, extenddb-app 28, extenddb-auth 219, extenddb-cache 27, extenddb-config 7, extenddb-core 519, extenddb-engine 178, extenddb-server 202, extenddb-storage 125, extenddb-storage-mongodb lib 122 (22 live and model tests ignored), its interpreter_selftests 18 and pushdown_parity 20, extenddb-storage-sqlite 141 (1 ignored); doc-tests 4.
• Against PostgreSQL 16, cargo test -p extenddb-storage-postgres --tests: lib 53, backup_restore 30, catalog_converge 5, doc_version_literals 3, key_collation 3, migrate_data_db_name 2, put_create_race 9, stream_shard_retention 1, twi_conflict 8, vector_control_plane 42, all passed.
• Against mongo:7, cargo test -p extenddb-storage-mongodb --lib -- --ignored --skip protocol_model live_: 18 passed under default test parallelism (three runs in a row before the rebase, one after); the protocol model's tests run with the workspace.
• Against cassandra:4.1, cargo test -p extenddb-storage-cassandra --lib: 61 passed; --test direct_integration: 149 passed. test_restore_sweep_worker_sweeps_at_startup_and_stops_on_shutdown, which this PR adds, missed its 8 s deadline on an earlier head on a node that had run tests for about a day (in the full run and alone); on a freshly created node it passed, and it passed in this head's full run. The sweep it exercises has not changed since then.
• cargo fmt --all -- --check, cargo +1.97.0 clippy --workspace --all-targets -- -D warnings, and cargo +1.88.0 check --workspace --locked --all-targets: clean. In one workspace run extenddb-cache's stale_hit_serves_cached_and_spawns_refresh, a timing test in a crate this PR does not touch, failed once and then passed five times alone and in the full rerun.

The engine, authorization, and the first round of backend fixes were each checked with a mutant (the fix reverted or broken, the new test run; every mutant was caught); of the later backend fixes, the name-before-backup order on SQLite and MongoDB and the cursor and key-info fixes were mutant-checked, the others were not. The MongoDB wedge was reproduced on the build before its fix (20 restores of 20 distinct backups: 6 answered, 14 timed out, every later request timed out) and the build with the fix answered all 20 in 1.1 s with every target ACTIVE after 6.9 s (both before the later admission change; the storage-level test live_twenty_concurrent_restores_of_distinct_backups_do_not_wedge_the_pool runs in the live suite above); the SQLite backup flood was checked at storage level (16 backups, at most 4 CREATING at once, another account's reads unaffected).

CI-equivalent legs (devtools/run-tests): tests/rust integration passed 512 on PostgreSQL, SQLite, and MongoDB, and the PostgreSQL no-vector leg passed 7, on an earlier build of this branch. The pytest legs (--pytest --comprehensive --parallel) fail exactly the tests that fail on origin/main's binaries on the same host (host Python 3.7 and botocore 1.33): PostgreSQL, SQLite, and MongoDB on the previous build of this branch, and Cassandra on the final one, with 2 workers as CI's 4-core runner gives: 6 failed against main's 6 failed and 1 error (main's extra error is a fixture failure in test_stream_table_reuse.py), plus 4 in tests/python on both. The PR's Cassandra CI job has failed on two runs, on different tests each time. In the first, test_gsi_configured_delay_range saw about 1 s GSI delays; that came from the GSI worker change this revision takes out (#407). In the second, three FGAC tests' teardown DeleteTable returned 500 because their account's keyspace was missing tables: a race on main when two accounts are created at once (#408), which this PR's extra tests made more likely by changing how the parallel suite is split. Until #408 is fixed, that leg can fail intermittently.

Not run: the CLI migration pytest suites (tests/test_cli_*migrat*.py), because psycopg2 is not installed on the test host. tests/test_put_item_create_race.py needs Python 3.8 and was not run on the 3.7 host.

Not changed here

• A client that disconnects while its RestoreTableFromBackup is still being authenticated or authorized (SigV4 verification and policy evaluation, including the target table's tag reads, run in the request before the engine handler) cancels the request: nothing is created and no error is reported to anyone. From the engine handler on, the request runs in a task of its own. The fidelity suite runs the server in dev mode, where these steps take almost no time, so its 1 ms disconnect test does not exercise this window.
• backup.rs maps storage Transient and Connection errors to 500 InternalServerError, where the generic mapping (for example in create_table.rs) gives 503 ServiceUnavailable. Both are retried by the SDKs; worth aligning separately.
• UpdateTable checks the 100 projected NonKeyAttributes total against a description read before, and outside, the backend's lock, so two concurrent GSI Creates can each pass and together exceed it (whether a backend keeps the table ACTIVE while the first index is CREATING was not checked). A GSI still DELETING is also counted.
• UpdateTable GSI Create refuses ProvisionedThroughput below 1 on a PAY_PER_REQUEST table, while {5, 5} on the same table is still accepted; the "should not be specified when BillingMode is PAY_PER_REQUEST" rule exists only in CreateTable validation. The ">= 1" rule also has three message texts.
• Error __type namespaces for ValidationException and AccessDeniedException still differ from the service's.
• Every ListBackups request reads all of the account's (or table's) backups and pages in memory.
• A GSI override carrying ProvisionedThroughput {0, 0} for an on-demand backup, which is what DescribeBackup reports, is refused (leave the member out), as on the service. Duplicate NonKeyAttributes in an index override are accepted (compared as a set); whether the service refuses them was not measured. A GSI entry's OnDemandThroughput is accepted and dropped: GsiInput has no field for a per-index on-demand maximum, and adding one touches every backend.
• GlobalSecondaryIndexOverride and LocalSecondaryIndexOverride are accepted only for indexes the backup has, with their key schema and projection unchanged. The reviewer measured the service treating the override list as a redefinition of the restored index set: it accepts and restores an entry naming an index the backup did not have, and one that re-keys an existing index. ExtendDB refuses both with ValidationException ("is not in the backup", "KeySchema differs"), which is narrower than the service but never produces wrong data. The reviewer is tracking the widening, and the error texts that read differently from the service's while the class is right (among them the TableInUseException text for a different backup restored into a CREATING restore target's name), separately from this PR.
• /health answers 200 as soon as the listener is up; it does not probe the storage backend (extenddb healthcheck documents it as liveness, not readiness). The condition under which it answered 200 while every request hung (the MongoDB wedge) is removed, and the admission bound keeps half the pool free, but a readiness probe should still issue a DescribeTable or ListTables with a client timeout.
• UpdateTable on a CREATING restore target is ResourceInUseException on all four backends; the service answers ResourceNotFoundException.
• ListBackups.TableName as a table ARN is refused with the pattern clause while CreateBackup accepts the ARN form; the service's behaviour for ListBackups was not measured.
• backup_id_epoch_millis reads the time only from the 14-digit id; the 13-digit ids of earlier builds carry no time for the ListBackups cursor placement.
• Pre-existing, reported in the round-4 review and left for separate PRs: a write racing a DeleteTable can answer 500; a Cassandra GSI backfill can fail with "Batch too large"; the Cassandra GSI workers under serve (storage-cassandra: asynchronous GSI writes are never applied under serve (default 10 ms delay) #407).
• MongoDB: a restore whose base copy finished but whose index backfill fails on every attempt stays CREATING; the backfill worker retries it every 5 s with a warning and has no give-up, and while it is CREATING DeleteTable refuses the target and DeleteBackup refuses its source backup. Not reproduced (no item was found that makes a backfill batch fail persistently); the remedy is to fix the cause and let the worker finish, or to remove the target's catalog document by hand.
• MongoDB: a DeleteBackup that returned an error can still take effect later. One that failed after claiming the backup (for example on finding a restore from it CREATING) and whose claim could then not be released, neither in the request nor by the claim guard on drop, is finished by the stale-claim sweep once no restore from the backup is CREATING. If that restore is one that never becomes ACTIVE (the backfill case above), the claim also blocks new restores of the backup. Not reproduced; it needs two consecutive release failures.
• MongoDB still uses the fixed database names extenddb_catalog and extenddb_data and has no serve lock. The sweeps only act on work whose heartbeat is stale.
• MongoDB reports BackupSizeBytes 0.
• Follow-up issues filed from this round: storage-cassandra: cdrs-tokio drops its only connection under sustained load and never reconnects #403 (cdrs-tokio drops its only connection under sustained load and does not reconnect), storage-cassandra: restore copy writes every item through the LWT put path #404 (the Cassandra restore copy goes through the LWT put path), storage-cassandra: asynchronous GSI writes are never applied under serve (default 10 ms delay) #407 (on main, Cassandra serve never applies asynchronous GSI writes, and the default delay is 10 ms, so served GSIs miss every write; CI sets the delay to 0), and storage-cassandra: concurrent account creation puts one account's tables in another's keyspace #408 (on main, concurrent Cassandra account creation can leave an account without its TTL and GSI-queue tables; this is what failed the FGAC tests in this PR's Cassandra CI run). A fix for the GSI worker lifetime was in an earlier revision of this PR and was taken out: reviving the workers exposes defects that belong in storage-cassandra: asynchronous GSI writes are never applied under serve (default 10 ms delay) #407.
• Cassandra under heavy load: with the fidelity file's default 60,000-item backup, the Cassandra pytest leg lost its only driver connection about 25 minutes in (Unmatched stream id, then No nodes available in query plan! on every request until the end), and 90 tests failed and 314 errored. cdrs-tokio 9.0.2 reuses stream ids after 32,767 requests without checking that the old one has been answered, and closes the connection when a second response arrives for an id; it then did not reconnect to the single node. That reading is from the driver's code and the server log, not a reproduction in isolation. devtools/run-cassandra-tests now sets EXTENDDB_TEST_BACKUP_ITEMS to 10,000 (unless it is already set), which keeps the copy running past the response; the driver behaviour itself is a production risk for a busy server: storage-cassandra: cdrs-tokio drops its only connection under sustained load and never reconnects #403.
• Cassandra:
  • It has no startup catalog-version gate; an unmigrated server serves tables and items and refuses only the backup operations (above).
  • UpdateTimeToLive returns an internal error when the backend's TTL change is still blocked by an expiration sweep after its retries (IndexesInUse); SDKs retry it.
  • The generated config always says replication_factor = 1 and datacenter = "datacenter1", whatever --replication-factor was given.
  • The control-plane poller, transaction recovery, GSI delay poller, and GSI backfill recovery workers ignore shutdown, so serve logs that background workers did not drain within 5 s.
  • The restore heartbeat shows the process is alive, not that the copy is progressing; a copy that hangs keeps its target CREATING until the process stops.
• PostgreSQL:
  • Restore is not supported behind a transaction-mode pooler: the restore lock is a session advisory lock. The admin guide says so; docs/design/11-high-availability.md still suggests transaction-mode pooling for large deployments.
  • The startup shape step's wait for the table lock (at most 5 s) still queues later statements on table_restores (every DescribeTable) behind it on every server while it waits.
  • A restore's copy holds one REPEATABLE READ catalog snapshot, and so one catalog pool connection and the catalog's xmin, from the first item read until the last full batch is written.
• PostgreSQL and SQLite: the "restore target" predicate (CREATING with no scheduled transition) and the fallback table ARN built in SQL are written in several SQL statements per backend rather than in one place.
• test_backup_arn_shape_and_account_are_checked runs in dev mode, so its texts and its foreign-account AccessDeniedException come from the engine. With authorization on, main's pre-authorization check answers "Invalid Backup ARN" for a BackupArn that does not parse as a backup ARN at all on DescribeBackup and DeleteBackup, before the engine's texts are reached.
• UpdateContinuousBackups accepts enabling point-in-time recovery and reports a 35-day restorable window, although nothing is recorded. This is pre-existing and now documented.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace). Not run as one command over every crate: the workspace run without the PostgreSQL storage crate and the Cassandra live targets, and the PostgreSQL, MongoDB, and Cassandra live runs, are listed above and passed. The pytest legs are reported in Testing done.
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic). cargo +1.97.0 clippy --workspace --all-targets -- -D warnings is clean; pedantic was not re-run this round.
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below
• If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Not met; see the next line.

ADR / RFC: none accepted. This PR changes the Storage trait (BackupEngine) and the on-disk format (catalog 0.0.4, Cassandra V005, SQLite backup_items and table_restores). docs/rfcs/README.md requires an accepted RFC for both, and none exists for these changes. #396 is a design PR, not an RFC: it revises docs/design/15-backup-restore.md for off-host backup and restore, a series in which it lists this PR as PR 0, to merge first. It describes later changes to BackupEngine (moving create, describe, list, delete, and restore into the engine) and does not specify this PR's change to BackupEngine::restore_table_from_backup. An RFC is required before merge.

Breaking changes

• Restore overrides that were previously ignored are now applied or checked. SSESpecificationOverride, GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, and OnDemandThroughputOverride are applied as the service applies them (an empty index list restores no indexes of that kind; a subset restores those); an unknown or changed index definition, a throughput member contradicting the billing mode, {0, 0}, and BillingModeOverride=PROVISIONED with a GSI whose throughput no list entry supplies are ValidationException; VectorIndexOverride is a ValidationException. A restore no longer carries TableClass: the target is STANDARD. ListBackups requests that set BackupType SYSTEM or AWS_BACKUP, or a time range, now return only the matching backups.
• BackupName and TableName on CreateBackup and ListBackups are validated (3 to 255 characters, [a-zA-Z0-9_.-]+; TableName takes the configured table-name limits); a BackupArn or ExclusiveStartBackupArn that is not a well-formed backup ARN is ValidationException where it was BackupNotFoundException; a cursor from another account, region, or table, a time bound before the epoch, and a lower bound above the upper bound are ValidationException. An ARN of the caller's account in another region now resolves (it was BackupNotFoundException); a statement in a policy must allow both spellings (below).
• CreateBackup of a table that does not exist or is CREATING is TableNotFoundException "Table not found: " (it was ResourceNotFoundException). DescribeBackup answers an internal error when the definition cannot be read (it answered a description without feature details). DeleteBackup returns the full description with BackupStatus DELETED.
• Restoring into a name that is a CREATING restore target is TableInUseException (it was TableAlreadyExistsException), and a second restore of a backup while a restore from it is CREATING is BackupInUseException on every backend (PostgreSQL, SQLite, and Cassandra accepted it).
• CreateBackup on SQLite is bounded to pool_size / 2 - 1 at once (4 by default), with LimitExceededException after 30 s. Restores on MongoDB are bounded to 4 per account and min(16, pool/2) per server (5 with the driver's default pool), with the same refusal, and a registration that takes longer than 60 s answers an internal error while the restore continues. UpdateTable on a MongoDB CREATING restore target is ResourceInUseException (it was accepted).
• CreateTable of a PROVISIONED table with a GSI ProvisionedThroughput below 1 (including 0/0), and UpdateTable GSI Create with one, are now ValidationException. So are a CreateTable with more than 100 projected NonKeyAttributes across its indexes and an UpdateTable GSI Create that would take the table over 100.
• Restoring into an existing table name is TableAlreadyExistsException, where it was ResourceInUseException.
• No table description carries RestoreSummary once the table is not CREATING (DescribeTable used to keep it with RestoreInProgress: false, and UpdateTable returned it too). RestoreDateTime is now the backup's creation time, not the time the restore ran.
• RestoreTableFromBackup returns before the copy. On PostgreSQL and SQLite the table used to be ACTIVE when the response arrived; now it is CREATING until the copy ends, so callers must wait for ACTIVE.
• On PostgreSQL a server process runs at most half of pool_size restores in total (capped at 16), besides 4 per account; more wait up to 30 s and then get LimitExceededException. extenddb migrate on PostgreSQL migrates the data database init recorded (or --data-db), not one derived from the catalog database name.
• On MongoDB, a restore of a backup that another restore or a DeleteBackup holds is BackupInUseException, and so is a DeleteBackup of a backup another delete holds, where both were BackupNotFoundException.
• On MongoDB, a DeleteBackup that returns an internal error or is cancelled after it claimed the backup can still complete: the backup reads AVAILABLE, DeleteBackup and RestoreTableFromBackup return BackupInUseException, and the stale-claim sweep marks it DELETED about 45 to 80 s later (later still if a restore from it is CREATING). Before, such an error released the claim and left the backup available, with no data if the drop had already run. A client that treats an error from DeleteBackup as "not deleted" is now wrong on MongoDB.
• The backup ARN id is 14-digit zero-padded epoch milliseconds, then 8 hex digits, on every backend. It used to be 13 digits (with or without the hex suffix); those ids still resolve but cannot place a ListBackups cursor once deleted or filtered out. BackupCreationDateTime has millisecond precision.
• On MongoDB, a connection string with w=0, a numeric or tagged w, or journal=false is refused at startup.
• The catalog version is now 0.0.4 for PostgreSQL, SQLite, and Cassandra, and extenddb migrate --yes is required before serving. PostgreSQL and SQLite servers refuse an unmigrated catalog, and a PostgreSQL server also refuses a catalog whose ledger lacks one of its catalog migration files, even at the current version, and refuses to start a pre-release 0.0.4 catalog if its startup shape step cannot lock table_restores within 5 s. SQLite serve also refuses a file database in a pre-release 0.0.4 shape until migrate --yes has run. A Cassandra server on an unmigrated catalog serves tables and items, but CreateBackup, RestoreTableFromBackup, and DeleteBackup fail until it is migrated. A stored catalog version that cannot be parsed is now refused rather than overwritten.
• The Storage trait changed. BackupEngine::restore_table_from_backup takes a resolved RestoreRequest, backup_restore_source is new, and restore_table_to_point_in_time is removed. BackupSummary gains table_id.
• The extenddb migrate preview without --yes exits 3 when work is pending; it used to exit 1. It exits 0 when everything is up to date.

---

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

[yesyayen]

Blocking on two things, both inline:

1. the Postgres CreateBackup barrier doesn't pin the snapshot the way the code and docs say it does, and
2. SQLite CreateBackup now holds the write lock for the whole read, which is a server-wide write stall on a big table. The rest are non-blocking.

[yesyayen]

Blocking. This barrier doesn't hold. LOCK TABLE is a utility statement and doesn't take a snapshot in Postgres. The REPEATABLE READ snapshot is taken by the first SELECT, which here runs after meta.commit() has already released the FOR SHARE row. So the definition is captured at one instant and the items at a later one, with UpdateTable free to commit in between. I checked this on PG 15: BEGIN ... REPEATABLE READ; LOCK TABLE t ...; then an insert from another session, then SELECT count(*) sees the new row.

[robinnsc]

Yeah good catch, swapped that to BEGIN RR RO; LOCK TABLE … ACCESS SHARE; SELECT 1 FROM <data> LIMIT 1; and only then releases the FOR SHARE row

[yesyayen]

Blocking. This now holds the engine write lock for the whole read and write of the table. On main the item read ran before the lock, only the inserts were under it. The pool is WAL mode, so a deferred read transaction on a second connection already gives a stable snapshot without blocking writers.

[robinnsc]

FIxed, the lock is now held only to read the definition, insert the backup row as CREATING, and open the deferred read transaction (snapshot fixed under the lock, so definition and items agree), items stream unlocked and are written in batches under short holds

[yesyayen]

Now that the definition is in hand, BillingModeOverride and ProvisionedThroughputOverride are a few lines on CreateTableInput. Refusing them is a wire regression for clients that pass them (the console does). I'd implement at least those two and refuse only the index and SSE overrides.

[robinnsc]

Done

[yesyayen]

This says "Matches on a restore target" without a backend qualifier, but the MongoDB diff only adds RestoreSummary and the DeleteBackup refusal. DeleteTable during a Mongo restore is not refused, and Mongo has no abandoned-restore sweep. Either qualify these rows or add the behaviour.

[robinnsc]

Qualified every row by backend, and implemented the Mongo DeleteTable, and clarified that Mongo still has no abandoned restore sweep

[yesyayen]

Worth a sentence here: any Postgres target the old bug left in CREATING matches the sweep predicate and gets removed on the first control-plane pass after upgrade. Right outcome, but operators will see tables disappear from ListTables and should know why.

[robinnsc]

Added

[yesyayen]

The restore cursor is the implicit rowid of backup_items, read across separately committed batches. backup_items has no INTEGER PRIMARY KEY, so a VACUUM between batches can renumber rowids and the restore skips or duplicates rows. Nothing in-tree runs VACUUM and #391 keeps a second extenddb out, but not a sqlite3 CLI. Give backup_items an INTEGER PRIMARY KEY or cursor on something stable.

[robinnsc]

Done, backup_items has id INTEGER PRIMARY KEY and the restore cursors on it. An existing table is rebuilt in place by migrate, and the fixture test now checks id is present with rows in order.

[yesyayen]

If this SET fails (managed instance that rejects it), every restore becomes a 500. Make it best-effort with a warning.

[robinnsc]

Done, now best-effort with a warning

[yesyayen]

This pass is notify-driven with a 60 s idle timeout, so an abandoned target is removed between 60 and ~120 s, not "after 60 s". Each pass also opens a fresh out-of-pool connection per still-locked candidate. Fine, but say so in the comment.

[robinnsc]

Corrected the comment

[yesyayen]

This lock is now held across the read and the interleaved catalog writes. A DeleteTable issued during the backup reaches the control-plane worker, whose DROP TABLE blocks behind it and stalls the whole worker pass until the backup finishes. Main had the same shape for the read only. Either SET LOCAL lock_timeout in the worker's drop or document it.

[robinnsc]

Went with using SET LOCAL lock_timeout

[yesyayen]

SQLite stores pk = '' while Postgres stores the real pk. Harmless today but the #348 migration-from-rows will want it. Also COPY_BATCH_ITEMS is imported but this file uses its own COPY_BATCH.

[robinnsc]

Swapped to storing the Real pk

[robinnsc]

#395 (SnapshotEngine) is branched from this PR and depends on it; its diff shows this PR's commit until this merges.

[LeeroyHannigan]

Requesting changes. The storage engineering under this PR held under reproduction: both of yesyayen's blocking items are fixed and each fix fails its test when reverted; restore atomicity against DeleteBackup held on PostgreSQL in both orders and across 100 race iterations on MongoDB; the migration, the sweep predicate, the backup_items cursor, the downgrade gate, and crash cleanup all behaved as described; cross-account access to backups is refused on every backend under every ARN variation tried; the restored data matched DynamoDB item for item on every index and projection. The problems are in the surface customers and operators touch, and four of them block the merge.

Blocking

1. The merged tree does not compile. #338 landed the Cassandra backend on main this morning with the pre-PR four-argument restore_table_from_backup. cargo check --workspace --all-targets on the merge of origin/main and ba9f869 fails in one crate, extenddb-storage-cassandra, E0050 (4 parameters, trait has 5). GitHub reports MERGEABLE because no file overlaps. Adding the parameter alone leaves three defects: the Cassandra restore hardcodes global_secondary_indexes: None and local_secondary_indexes: None, ignores both overrides, and never calls ensure_single_part_base_key or ensure_restorable, because those refusals live in each SQL storage backend rather than in the engine. Either port the definition, the overrides, and both refusals into Cassandra, or add a Cassandra row to docs/differences-from-dynamodb.md saying what its restore does not honour. In both cases move the two refusals into crates/engine/src/backup.rs so a backend cannot skip them.

2. RestoreTableFromBackup copies every item inside the request on PostgreSQL and SQLite, and a client timeout cancels it. Both backends await copy_backup_items and only then return CREATING for a table that is already ACTIVE (the comment at backup_engine.rs:1282 says so). Reproduced with 40,000 items and two GSIs, client read timeout 3 s, no retries. PostgreSQL: the target sat CREATING with zero items, then the sweep removed it 64 s after the disconnect; the restore was lost. SQLite: the target is still CREATING four minutes later, a retry into the same name returns ResourceInUseException, DeleteTable on it returns ResourceInUseException ("Table is being restored"), and DeleteBackup on the source returns BackupInUseException. The customer is locked out of the name and the backup until the process restarts. boto3 and the CLI default to a 60 s read timeout, and with one INSERT per item per index over a network a few hundred thousand items cross it, so large restores cannot succeed with stock clients. SQLite CreateBackup in this PR already detaches its copy with tokio::spawn for this reason (backup.rs:799). Do the same here on both backends: commit the CREATING target and its table_restores row, spawn the copy, return at once. Add a live test that drops the client mid-restore and asserts ACTIVE with the full count. This also makes the DeleteBackup and DeleteTable refusals observable the way the docs describe them.

3. DescribeBackup still returns the pre-PR subset. Against DynamoDB, a backup of a table with a GSI, an LSI, and TTL enabled returns SourceTableFeatureDetails with GlobalSecondaryIndexes, LocalSecondaryIndexes, and TimeToLiveDescription, and SourceTableDetails.ProvisionedThroughput ({0, 0} for on-demand; the member is required in the shape). ExtendDB omits all of them on all three backends, although backup_definitions now holds the data. A customer auditing a backup is told it has no indexes; a strongly typed SDK can fail to deserialize the response. Populate both from the stored definition.

4. The PR body and the docs state behaviour the build does not have. Checked on servers built from ba9f869: "DeleteTable on a table that is being restored returns ResourceInUseException on all three backends" (only MongoDB; see item 2); "SQLite similarly exposes CREATING while it copies" (the caller blocks until AVAILABLE; only a concurrent observer sees CREATING); "a backup is never stuck undeletable" (false after a MongoDB crash during $out, and after item 2 on SQLite); "backups of tables with vector indexes are refused" (only the restore is refused; CreateBackup returns an AVAILABLE backup). Every claim in the body and in differences-from-dynamodb.md needs a live check per backend before the next revision.

Wire behaviour, measured against DynamoDB

Each of these was run on the service and on all three ExtendDB backends.

• RestoreSummary.RestoreDateTime is the restore wall clock. The service reports the source backup's creation time (measured: RestoreDateTime 1791976431.438 equal to BackupCreationDateTime, restore issued 74 s later). A client reading it to learn which point in time was restored gets the wrong instant.
• The service drops RestoreSummary from DescribeTable once the table is ACTIVE (absent at ACTIVE and 30 s later). ExtendDB keeps it with RestoreInProgress: false, and tests/test_backup_restore_fidelity.py:252-254 pins that.
• RestoreSummary.SourceTableArn is never emitted; the service sets it.
• The CREATING restore response carries GSI IndexStatus: ACTIVE; the service omits the member.
• A malformed BackupArn returns AccessDeniedException; the service returns ValidationException.
• Restoring into an existing table name returns ResourceInUseException; the service returns TableAlreadyExistsException.
• An empty GlobalSecondaryIndexOverride or LocalSecondaryIndexOverride list and an SSESpecificationOverride are refused; the service accepts them as no-ops. The blanket refusal is documented, the no-op case is not.
• The PROVISIONED-with-GSIs refusal tells the caller to pass GlobalSecondaryIndexOverride, which ExtendDB then refuses. The fidelity test pins that text at line 649.
• ListBackups ignores Limit and never returns LastEvaluatedBackupArn; every row comes back in one response. The service caps at 100 with a token.

PostgreSQL

• The restore's ownership is a session advisory lock on a dedicated connection that issues no statement for the whole copy. SET idle_session_timeout = 0 is best-effort and the comment at backup_engine.rs:118-123 says managed services may reject it. On RDS, Aurora, or behind a transaction-mode pooler the idle session can end while the copy continues; after 60 s the sweep flips the target to DELETING and the restore fails with "deleted while it was being restored". Heartbeat the lock session, or gate the sweep on a liveness signal that does not depend on an idle connection. Item 2 does not remove this; a long detached copy has the same exposure.
• "Bounded memory" bounds the Rust buffer. The copy is one PostgreSQL transaction (begin at line 325, one commit after the loop), and secondary-index rows are written one INSERT per item per index in the nested loop at line 464 while base rows are batched. A million-item table with four GSIs is one transaction with four million round trips, WAL held throughout, target pinned CREATING throughout.
• A failure to open the lock connection (max_connections reached) maps to an Internal error. Map it to LimitExceeded so clients back off.
• restore_table_to_point_in_time (lines 1393-1420) is unreachable, uses a fixed backup name, and discards its cleanup error. Delete it.

SQLite and MongoDB

• CreateBackup on a table with a vector index returns an AVAILABLE backup that RestoreTableFromBackup always refuses. The customer finds out at restore time. Refuse at CreateBackup with the same message.
• MongoDB: a crash during the base $out copy, before restore_backfill_pending is set, leaves a CREATING target that nothing sweeps or resumes. DeleteTable refuses it and DeleteBackup of the source returns BackupInUse until the catalog is edited by hand. Crashes after $out self-heal. Add a sweep for CREATING targets carrying restore_source_backup_arn with no pending flag and no live operation, or let DeleteTable remove them.
• MongoDB ignores --data-db and --catalog-db and has no serve lock, so two processes on one replica set share one catalog, and sweep_stale_backup_claims at startup clears the other process's in-flight markers. Pre-existing, but the sweep makes it matter.
• pool_size = 1 is raised to 2 for a file-backed database with no log line and no mention in the config docs. A file-backed backup also pins one pooled connection for its duration, so pool_size >= 3 is wanted for undisturbed concurrency; say so.
• BackupMetadataClaim's Drop release only runs when a runtime handle is current; a comment should say the startup sweep is the backstop.
• The MongoDB backup is not a point-in-time snapshot ($out reads the live collection). Line 93 says so in the general consistency row; the Backup and Restore section should state the per-item mixed-state consequence where an operator choosing MongoDB will read it.

Tenancy and validation

• BackupTableDefinition::apply_to applies the stored definition without validate_create_table. A definition with 30 GSIs, a GSI on an undefined attribute, 101 projected attributes, or negative throughput restores to an ACTIVE table that CreateTable would reject. Triggering it needs catalog write access; it is still wrong data stored, and the service re-validates.
• RESTORE_SLOTS is a process-global semaphore of four. One account can hold all four and every other account's restores wait 30 s and get LimitExceededException. Make the limit per account.
• converge_catalog_version and refuse_newer_catalog treat a stored catalog_version the binary cannot parse as "proceed" and stamp the compiled version over it (older_or_equal_or_absent_is_allowed asserts Some("garbage") is allowed). That defeats the downgrade guard in the one case it exists for. Fail closed.
• Pre-existing, outside this diff: RestoreTableFromBackup, DescribeBackup, and DeleteBackup are authorized against arn:...:table/* because extract_table_name reads only TableName (crates/server/src/request_helpers.rs:42-46, 289-294; the server log shows resource="...:table/*" on a denial). A policy scoped to a target table or a backup ARN can never match. The service authorizes on the backup ARN and the target table. This PR makes restore a first-class feature on top of that gap; fix it here or file an issue; either way it needs recording.

Upgrade path and docs

Walked literally from docs/manuals/07-upgrade-manual.md on a 0.0.3 deployment created by the 0.1.13 binary.

• extenddb migrate --config extenddb.toml as written fails: it needs PostgreSQL admin credentials the manual never mentions, and the error is Error: Internal("Internal(\"Cannot connect as admin: error returned from database: password authentication failed for user \"lhnng\"\")"). Document --pg-user and EXTENDDB_PG_PASSWORD in the command, flatten the nested error, and say what to pass.
• The manual says to run migrate without --yes first to see what is pending; the tool prints the plan and then Error: --yes is required to apply migrations with exit status 1, so a scripted upgrade under set -e stops at the preview. Exit 0 for the preview, or a documented distinct code.
• On SQLite the upgrade rebuilds backup_items by copying every row into a new table in one write transaction (schema.rs:494-553). No manual mentions it, the dry run does not name it, and the function logs nothing. Document it, name it in the dry run when the old shape is present, and log start and finish with the row count.
• The sweep that removes targets the old bug left in CREATING logs nothing. Operators will see tables vanish from ListTables with no server record of why. Log each removal by name.
• The manual's backup and rollback commands hardcode extenddb_catalog, extenddb, and OWNER extenddb; init lets the operator choose all three. There is no SQLite subsection (stop, copy the file, migrate, roll back by restoring the copy).
• 0.0.4 and migration 003 are also claimed by #368 and #335. The migration ledger keys on filename and the gate compares version numbers, so if #368 merges first and stamps 0.0.4, this binary applies its own 003 and converges with no refusal: two schemas named 0.0.4. Whoever merges second renumbers to 004 and 0.0.5 (and the two assertions in migrations.rs tests, and the manual's 0.0.4 section). The order should be agreed now.
• differences-from-dynamodb.md names three backends; the merged tree ships four.
• The Storage trait and the on-disk format changed. CONTRIBUTING.md and docs/rfcs/README.md require an accepted RFC for both; the PR links #348, a design PR, not a numbered accepted RFC.
• The PR template asks for a ## Breaking changes section; the changes are spread through the prose, and the one customers will hit first (overrides that were silently ignored are now hard errors) deserves its own line.

Minor

Backup ARN timestamp is 13 digits where the service zero-pads to 14; BackupCreationDateTime is whole seconds where the service is sub-second; the restore response adds LastUpdateToPayPerRequestDateTime, which the service omits; ListBackups summaries omit TableId; error __type namespace differs for ValidationException and AccessDeniedException; with two invalid inputs the service checks the ARN account before the target name.

[LeeroyHannigan]

main now has crates/storage-cassandra/src/backup_engine.rs (#338, merged this morning) with the old four-argument signature, so the merge fails to compile: cargo check --workspace --all-targets on origin/main plus this branch gives E0050 in extenddb-storage-cassandra and nothing else. Adding the parameter there leaves three defects: the Cassandra restore hardcodes global_secondary_indexes: None and local_secondary_indexes: None, ignores the overrides, and never calls ensure_single_part_base_key or ensure_restorable, which this PR enforces inside the PostgreSQL and SQLite backends rather than in the engine. Port the definition and both refusals into Cassandra, or document in the differences matrix what its restore does not honour, and move the two refusals into the engine so a backend cannot skip them.

[robinnsc]

Fixed. The trait now takes a resolved RestoreRequest. The engine runs ensure_restorable, restore_input and validate_create_table for every backend, and Cassandra restores GSIs, LSIs and the overrides through its V005 migration. cargo check --workspace --all-targets is clean on the merge.

[LeeroyHannigan]

The copy is awaited inside the request future, so the response at line 1282 says CREATING for a table that is already ACTIVE, and a client that times out drops the future mid-copy. Reproduced with 40,000 items and two GSIs, read timeout 3 s, no retries: the target sat CREATING with zero items and the sweep removed it 64 s after the disconnect. The restore was lost. boto3 and the CLI default to 60 s, so any restore whose copy takes longer cannot succeed with a stock client. Commit the CREATING target and its table_restores row, spawn the copy, and return at once, the way SQLite CreateBackup does at backup.rs:799. Add a live test that drops the client and asserts the target reaches ACTIVE with the full count.

[robinnsc]

Fixed. Registration and copy run in a spawned task; the request waits only for the CREATING description. Once the request is authenticated and authorized, the whole handler (backup read, validation, registration) also runs in its own task, so a disconnect does not cancel it part-way. Live tests: a 1 ms-read-timeout disconnect and the copy window (DeleteTable/DeleteBackup refused while CREATING) on all four backends, plus storage-level tests that drop the request future.

[LeeroyHannigan]

The SQLite path has the same shape. With the client gone after 2 s on a 40,000-item restore, the target is still CREATING with RestoreInProgress: true four minutes later; a retry into the same name returns ResourceInUseException, DeleteTable on the target returns ResourceInUseException ("Table is being restored"), and DeleteBackup on the source returns BackupInUseException. SQLite sweeps abandoned restores only at startup, so the name and the backup are locked until the process restarts. create_backup at line 799 already detaches its copy with tokio::spawn and the comment there names this hazard; the restore path needs the same treatment.

[robinnsc]

Fixed the same way as PostgreSQL. serve also now refuses a file database left in a pre-release 0.0.4 shape and points to extenddb migrate --yes.

[LeeroyHannigan]

This connection issues no statement after pg_try_advisory_lock for the whole copy. When the SET is refused (the comment says managed services refuse it) or the connection sits behind a transaction-mode pooler, the idle session can end while the copy continues, the lock frees, and after 60 s the sweep flips the target to DELETING; the restore then fails with "deleted while it was being restored". That covers RDS, Aurora, and PgBouncer. Heartbeat the session while the copy runs, or gate the sweep on a liveness signal that does not depend on an idle connection.

[robinnsc]

The lock session now runs SELECT 1 every 15 s and the copy stops if that fails, which covers idle-session reaping on RDS/Aurora. A transaction-mode pooler is not supported for restores (the lock is a session advisory lock): the admin guide says so, and the comments that implied poolers were handled are removed. Replacing the lock with a lease row (as MongoDB and Cassandra use) is the durable fix if transaction pooling matters; I left that out of this PR.

[LeeroyHannigan]

The whole copy is one transaction: begin here, one commit after the loop. The 500-item / 4 MiB bound limits the Rust buffer only. In write_restore_batch the base rows go in one multi-row INSERT, but the loop at line 464 writes secondary-index rows one INSERT per item per index. A million-item table with four GSIs is one transaction with four million round trips, WAL held for the duration, and the target pinned CREATING throughout, which widens the sweep exposure above. Batch the index rows per index per batch, and either commit per batch with cleanup on failure or document the single-transaction cost for large restores.

[robinnsc]

Fixed: one commit per batch, one multi-row INSERT per index per batch. The catalog side still holds one REPEATABLE READ snapshot for the copy (so the backup rows are read whole); that is documented in the rustdoc.

[LeeroyHannigan]

DynamoDB does not return RestoreSummary on DescribeTable once the table is ACTIVE (checked at ACTIVE and 30 s later). This assertion pins the ExtendDB behaviour as correct. The test should assert the member is absent after ACTIVE, and the engine should stop emitting it.

[robinnsc]

The test now asserts the member is absent once ACTIVE, and the engine no longer emits it.

[LeeroyHannigan]

"A backup is never stuck undeletable" is false in two cases on this branch: a MongoDB crash during the base $out copy leaves a CREATING target that blocks DeleteBackup with BackupInUse until the catalog is hand-edited, and on SQLite a restore whose client disconnected leaves the same state until restart. The next row's "PostgreSQL, SQLite, and MongoDB match" is only observable on MongoDB: on the SQL backends the restore completes inside the request, so a sequential client's DeleteTable or DeleteBackup after the response succeeds every time.

[robinnsc]

Rewritten for four backends. Rows that rest on code reading or storage-level tests are marked (from the code), numbers from earlier builds of the branch are marked (earlier build), and the header says so.

[LeeroyHannigan]

"SQLite similarly exposes CREATING while it copies" reads as if the caller gets a CREATING response. The caller blocks until AVAILABLE on all three backends (1.4 s for 50,000 items; minutes for a large table), and only a concurrent observer sees the CREATING row. Say that, and note that CreateBackup blocks for the copy duration, or return CREATING and let the detached task finish, which the SQLite code already supports.

[robinnsc]

Rewritten: CreateBackup blocks the caller for the copy on all four backends; only a concurrent observer sees CREATING. Detaching CreateBackup is left for a follow-up.

[LeeroyHannigan]

Followed literally on a 0.0.3 deployment this fails: migrate connects as a PostgreSQL admin and the manual gives no credentials, so it falls back to the OS user and prints Error: Internal("Internal(\"Cannot connect as admin: ... password authentication failed for user \"lhnng\"\")"). Show --pg-user and EXTENDDB_PG_PASSWORD here, and flatten that nested error in the tool. The preview step on line 203 also exits 1 with Error: --yes is required, so a scripted upgrade under set -e stops at the preview; exit 0 for the preview or say the non-zero exit is intended. Nothing here tells a SQLite operator about the backup_items rebuild, and the backup and rollback steps above hardcode extenddb_catalog, extenddb, and OWNER extenddb.

[robinnsc]

--pg-user and EXTENDDB_PG_PASSWORD are shown; the error is one plain line naming the role it tried; the preview exits 3 when work is pending and 0 when up to date (documented); the commands are parameterized; and there are SQLite, Cassandra and MongoDB subsections.

[LeeroyHannigan]

#368 and #335 also ship a 003_*.sql that writes 0.0.4. The ledger keys on filename and converge_catalog_version compares numbers, so if #368 merges first and stamps 0.0.4, this binary applies its own 003 and converges with no refusal: two different schemas both called 0.0.4, and the startup gate cannot tell them apart. Whoever merges second renumbers to 004 and 0.0.5 (plus the two assertions in the migrations.rs tests and the manual's 0.0.4 section). The order should be agreed before any of the three lands.

[robinnsc]

Not renumbered; I need your call on the order (open question in the PR body: #396 assumes this PR keeps 003/0.0.4 and #335 moves to 004/0.0.5, and does not place #368). Meanwhile migrate applies an unrecorded catalog migration file even when the version already matches, and PostgreSQL serve refuses to start on a catalog missing one of its files, so a second 003 can no longer be skipped silently.