Repository navigation
feat(core): add portable backup manifest and DynamoDB export layout types - #337
Closed
LeeroyHannigan wants to merge 2 commits into
Closed
LeeroyHannigan wants to merge 2 commits into
LeeroyHannigan wants to merge 2 commits into
Conversation
…ypes
Add crates/core/src/backup_format, the pure-type layer of the off-host
backup format: no I/O, no storage dependency.
extenddb-manifest.json (ExtenddbManifest): format_version gated by
check_supported, backup identity fields with backup_creation_date_time as
f64 epoch seconds matching BackupDetails, snapshot descriptor, a
source_table section reusing the existing wire types (KeySchemaElement,
AttributeDefinition, BillingMode, ProvisionedThroughput,
OnDemandThroughput, SseDescription, GsiInput, LsiInput,
VectorIndexDescription), per-data-file entries (key, item_count, sha256,
bytes), and manifest_sha256 computed over the manifest's compact JSON with
the checksum field set to the empty string, with compute, seal, and verify
helpers. Unknown fields are tolerated on read so a newer writer does not
break an older reader; the version gate is the explicit refusal point. The
manifest doc records two constraints the checksum implies: an additive
field requires a format_version bump because the digest covers all fields
and an older reader drops unknown ones, and the digest depends on
serde_json f64 formatting of the epoch-seconds timestamps, so a
non-serde_json verifier must reproduce that formatting.
DynamoDB export layout (ExportSummaryManifest, ExportFileEntry): the
manifest-summary.json and manifest-files.json shapes Amazon DynamoDB
writes for ExportTableToPointInTime with DYNAMODB_JSON output, per the
documented output format, including the explicit null s3SseKmsKeyId, the
exportType member appearing only on incremental exports, and JSON-lines
serialization for the files manifest. Both a serializer and a parser are
provided; the parser also accepts the documented incremental-export
members it does not model.
Data files (encode_data_file, DataFileDecoder): gzip newline-delimited
{"Item": ...} objects using the same json!({"Item": item}) line encoding
the engine's export handler writes, so the two agree byte for byte. The
encoder reports item count, sha256 (hex) and md5 (base64) over the
compressed bytes, and the uncompressed byte count; the decoder verifies
whichever expectations it is given and streams items back. The decoder is
bounded: each line is read through a fill_buf/consume loop capped at
MAX_LINE_BYTES (4 MiB, derived from the 400 KB item limit plus JSON and
base64 overhead) so a compressed bomb cannot drive unbounded memory, it
returns a typed LineTooLong on breach, it enforces the recorded
uncompressed byte count incrementally rather than only at end of stream,
and it uses MultiGzDecoder so a multi-member gzip decodes fully instead of
silently stopping after the first member.
DataFileEntry.key is validated at manifest parse time
(validate_data_files) and again at decode time (DataFileEntry::decode):
the key must be data/ plus a single <digits>.json.gz segment, rejecting
traversal, absolute paths, wrong prefixes, and extra directory levels.
Key layout (backup_prefix, data_file_key, parse_backup_arn): the
account/table/backup_id store prefix, fixed object names, zero-padded
data file names, and parsing of the ARN shape the engine's backup
handlers use, unchanged.
Golden files: a small DynamoDB export checked in under
crates/core/testdata/ddb-export, constructed to the documented format
(provenance in its README); tests assert the parser reads it and the
serializers reproduce both manifests byte for byte, and that decoded
items re-encode to identical uncompressed lines.
Dependencies flate2 1.1.9, md-5 0.10.6, sha2 0.10.9 are the versions
already resolved in Cargo.lock.
Testing: cargo fmt --all -- --check, cargo clippy --all-targets -- -D
warnings, the new backup_format module additionally clippy::pedantic
clean, cargo test -p extenddb-core (546 passed, 54 in backup_format),
cargo test --workspace (1212 passed), cargo +1.88.0 check --workspace
--locked clean. The bound test feeds a 64 MiB single-line gzip bomb and
measures peak resident memory from /proc/self/status: it grew about 4 MiB
across the decode, held to the line cap.
Signed-off-by: Lee Hannigan <lhnng@amazon.com>
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
September 10, 2026 19:03
robinnsc
reviewed
Sep 19, 2026
robinnsc
left a comment
robinnsc
left a comment
Collaborator
There was a problem hiding this comment.
Read through the whole module. Overall this is in good shape: the bounded line reader, MultiGzDecoder, the key validation, and the golden fixtures are all solid, and the checksum canonicalization tradeoffs are documented rather than hidden. One real bug (the line cap boundary), one design question I'd like settled before format_version 1 freezes (committing the two DynamoDB manifests), and a few smaller things inline.
| } | ||
| if let Some(nl) = available.iter().position(|&b| b == b'\n') { | ||
| let want = nl + 1; | ||
| if line.len() + want > MAX_LINE_BYTES { |
Collaborator
There was a problem hiding this comment.
Off-by-one against the doc: the comment on MAX_LINE_BYTES says "a line at or over the cap yields LineTooLong", but this check is >, so a line of exactly MAX_LINE_BYTES is accepted. Related: line_exactly_at_cap_boundary_is_rejected writes 1 MiB blocks while written <= MAX_LINE_BYTES, which ends at 5 MiB, so despite the name it tests one block past the cap rather than the boundary. Either behavior is defensible, but the doc, the check, and the test should agree, and the boundary test should feed exactly MAX_LINE_BYTES bytes.
| /// Lowercase hex sha256 of the manifest's canonical JSON, computed with | ||
| /// this field set to the empty string. See | ||
| /// [`Self::compute_manifest_sha256`]. | ||
| pub manifest_sha256: String, |
Collaborator
There was a problem hiding this comment.
Design question, worth settling before format_version 1 freezes: the manifest checksums itself and records a sha256 per data file, but nothing commits manifest-summary.json or manifest-files.json. Those two can be tampered with or corrupted without any ExtendDB-side verification failing, and the files manifest carries the md5s a plain ImportTable-style consumer would trust. Since this manifest is the commit record, recording the sha256 of both DynamoDB manifests here would make the whole backup verifiable from one sealed document. Adding it later costs a version bump, so now is the cheap time.
| /// Returns [`FormatError::InvalidDataFileKey`] when the key is not a | ||
| /// single data-file entry under the prefix, and any error | ||
| /// [`crate::backup_format::decode_data_file`] can produce. | ||
| pub fn decode(&self, compressed: &[u8]) -> Result<Vec<crate::types::Item>, FormatError> { |
Collaborator
There was a problem hiding this comment.
The entry records bytes (compressed size) but decode never enforces it, only sha256 and item_count. A length check is nearly free and fails fast before hashing a wrong-sized object, so it seems worth passing through as an expectation (or checking directly here).
| /// | ||
| /// Returns [`FormatError::InvalidDataFileKey`] for the first offending | ||
| /// key. | ||
| pub fn validate_data_files(&self) -> Result<(), FormatError> { |
Collaborator
There was a problem hiding this comment.
This doesn't reject duplicate keys, so a manifest listing the same data file twice validates cleanly and would restore its items twice. A uniqueness check over data_files here would close that.
|
|
||
| # Crypto & checksums | ||
| crc32fast = "1" | ||
| # Pinned to the versions already resolved in Cargo.lock. |
Collaborator
There was a problem hiding this comment.
The comment says "pinned", but these are caret requirements, so any compatible 1.x can resolve on a lockfile refresh. Either use = versions if pinning is the intent, or match the workspace's existing loose style (crc32fast = "1") and drop the comment.
| /// The key of the `index`-th data file within a backup prefix: | ||
| /// `data/000001.json.gz` for index 1. | ||
| #[must_use] | ||
| pub fn data_file_key(index: u32) -> String { |
Collaborator
There was a problem hiding this comment.
Take or leave: zero-padding to six digits means file 1,000,000 breaks lexicographic ordering (the test even shows data/1000000.json.gz). If anything ever lists the data prefix and relies on name order matching index order, that's a trap. If order never matters, a one-line doc note saying so would close the question.
robinnsc
added a commit
that referenced
this pull request
Oct 6, 2026
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. `create_backup` and `restore_table_from_backup` on PostgreSQL checked for a column named `sk`, which data tables do not have (`sk_s`, `sk_n`, `sk_b`). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store `pk` and `item_data` only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in `item_data` and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add `backup_definitions` (one row per backup, the definition as JSON behind a version marker; `extenddb_storage::backup_definition`) and `table_restores` (one row per restored table: source backup ARN and restore time, not a foreign key to `backups` so the summary outlives the backup, as on the service). Backups and restores from before the upgrade have no rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency: - PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot (REPEATABLE READ) is taken, and locks the source data table ACCESS SHARE before releasing the row. UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. A GSI whose data table UpdateTable has not built yet is left out. - SQLite CreateBackup reads the table row, its definition, and its items in one write transaction. Every other write on that server waits for it. - A restore registers itself in the same transaction that creates its target (the `table_restores` row on SQLite, the provenance fields on the target's catalog document on MongoDB; PostgreSQL holds the backup row FOR SHARE while it records itself), so DeleteBackup can never observe the target without its source and returns BackupInUseException (HTTP 400) for the whole restore. A restore reads from a snapshot (PostgreSQL) or re-checks under the write lock for each batch (SQLite), so it copies all of a backup or fails with BackupNotFoundException. - DeleteTable on a table being restored returns ResourceInUseException. - DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. MongoDB gets the same two behaviours. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. SQLite restores commit per batch. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way, including when target creation itself fails part-way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection; the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free. SQLite removes the same targets at startup, which is safe only because #391 lets one server hold the file; this change should merge after #391. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and RestoreTableFromBackup requests carrying BillingModeOverride, GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, ProvisionedThroughputOverride, or SSESpecificationOverride, which were silently ignored before. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported; docs/differences-from-dynamodb.md gains a Backup and Restore section (what is restored, the refusals, where backups live, the SQLite write-lock cost) and its vector-index row matches the code; the upgrade manual has a 0.0.4 section and notes the SQLite lock; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (8, any backend: S/N/B hash and sort keys compared item by item, a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads, RestoreSummary, and a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput on the table and its GSI); crates/storage-postgres/tests/backup_restore.rs (15, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier, half-built GSIs, DeleteTable and DeleteBackup during a restore, the abandoned-restore sweep, RestoreSummary); SQLite `restore_tests` (11, including the restore intent committed with its target and a 0.0.3 database in the old row format); unit tests for the definition normalization, the override rejections, CatalogVersion ordering, and the newer-catalog refusals in the migrate command and both schema runners. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
5 of 8 tasks
robinnsc
added a commit
that referenced
this pull request
Oct 6, 2026
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. `create_backup` and `restore_table_from_backup` on PostgreSQL checked for a column named `sk`, which data tables do not have (`sk_s`, `sk_n`, `sk_b`). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store `pk` and `item_data` only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in `item_data` and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add `backup_definitions` (one row per backup, the definition as JSON behind a version marker; `extenddb_storage::backup_definition`) and `table_restores` (one row per restored table: source backup ARN and restore time, not a foreign key to `backups` so the summary outlives the backup, as on the service). Backups and restores from before the upgrade have no rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency: - PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot (REPEATABLE READ) is taken, and locks the source data table ACCESS SHARE before releasing the row. UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. A GSI whose data table UpdateTable has not built yet is left out. - SQLite CreateBackup reads the table row, its definition, and its items in one write transaction. Every other write on that server waits for it. - A restore registers itself in the same transaction that creates its target (the `table_restores` row on SQLite, the provenance fields on the target's catalog document on MongoDB; PostgreSQL holds the backup row FOR SHARE while it records itself), so DeleteBackup can never observe the target without its source and returns BackupInUseException (HTTP 400) for the whole restore. A restore reads from a snapshot (PostgreSQL) or re-checks under the write lock for each batch (SQLite), so it copies all of a backup or fails with BackupNotFoundException. - DeleteTable on a table being restored returns ResourceInUseException. - DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. MongoDB gets the same two behaviours. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. SQLite restores commit per batch. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way, including when target creation itself fails part-way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection; the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free. SQLite removes the same targets at startup, which is safe only because #391 lets one server hold the file; this change should merge after #391. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and RestoreTableFromBackup requests carrying BillingModeOverride, GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, ProvisionedThroughputOverride, or SSESpecificationOverride, which were silently ignored before. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported; docs/differences-from-dynamodb.md gains a Backup and Restore section (what is restored, the refusals, where backups live, the SQLite write-lock cost) and its vector-index row matches the code; the upgrade manual has a 0.0.4 section and notes the SQLite lock; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (8, any backend: S/N/B hash and sort keys compared item by item, a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads, RestoreSummary, and a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput on the table and its GSI); crates/storage-postgres/tests/backup_restore.rs (15, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier, half-built GSIs, DeleteTable and DeleteBackup during a restore, the abandoned-restore sweep, RestoreSummary); SQLite `restore_tests` (11, including the restore intent committed with its target and a 0.0.3 database in the old row format); unit tests for the definition normalization, the override rejections, CatalogVersion ordering, and the newer-catalog refusals in the migrate command and both schema runners. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 6, 2026
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. `create_backup` and `restore_table_from_backup` on PostgreSQL checked for a column named `sk`, which data tables do not have (`sk_s`, `sk_n`, `sk_b`). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store `pk` and `item_data` only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in `item_data` and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add `backup_definitions` (one row per backup, the definition as JSON behind a version marker; `extenddb_storage::backup_definition`) and `table_restores` (one row per restored table: source backup ARN and restore time, not a foreign key to `backups` so the summary outlives the backup, as on the service). Backups and restores from before the upgrade have no rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency: - PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot (REPEATABLE READ) is taken, and locks the source data table ACCESS SHARE before releasing the row. UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. A GSI whose data table UpdateTable has not built yet is left out. - SQLite CreateBackup reads the table row, its definition, and its items in one write transaction. Every other write on that server waits for it. - A restore registers itself in the same transaction that creates its target (the `table_restores` row on SQLite, the provenance fields on the target's catalog document on MongoDB; PostgreSQL holds the backup row FOR SHARE while it records itself), so DeleteBackup can never observe the target without its source and returns BackupInUseException (HTTP 400) for the whole restore. A restore reads from a snapshot (PostgreSQL) or re-checks under the write lock for each batch (SQLite), so it copies all of a backup or fails with BackupNotFoundException. - DeleteTable on a table being restored returns ResourceInUseException. - DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. MongoDB gets the same two behaviours. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. SQLite restores commit per batch. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way, including when target creation itself fails part-way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection; the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free. SQLite removes the same targets at startup, which is safe only because #391 lets one server hold the file; this change should merge after #391. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and RestoreTableFromBackup requests carrying BillingModeOverride, GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, ProvisionedThroughputOverride, or SSESpecificationOverride, which were silently ignored before. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported; docs/differences-from-dynamodb.md gains a Backup and Restore section (what is restored, the refusals, where backups live, the SQLite write-lock cost) and its vector-index row matches the code; the upgrade manual has a 0.0.4 section and notes the SQLite lock; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (8, any backend: S/N/B hash and sort keys compared item by item, a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads, RestoreSummary, and a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput on the table and its GSI); crates/storage-postgres/tests/backup_restore.rs (15, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier, half-built GSIs, DeleteTable and DeleteBackup during a restore, the abandoned-restore sweep, RestoreSummary); SQLite `restore_tests` (11, including the restore intent committed with its target and a 0.0.3 database in the old row format); unit tests for the definition normalization, the override rejections, CatalogVersion ordering, and the newer-catalog refusals in the migrate command and both schema runners. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 9, 2026
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. create_backup and restore_table_from_backup on PostgreSQL checked for a column named `sk`, which data tables do not have (sk_s, sk_n, sk_b). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store pk and item_data only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in item_data and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. RestoreTableFromBackup overrides. BillingModeOverride and ProvisionedThroughputOverride are implemented on all three backends and passed to storage as an explicit `overrides: RestoreTableOverrides` parameter on BackupEngine::restore_table_from_backup (a trait change). BackupTableDefinition::apply_to applies them before the PAY_PER_REQUEST normalization and refuses, before any target is created, a PROVISIONED restore of a backup whose GSIs carry no throughput (the service needs GlobalSecondaryIndexOverride there). GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, SSESpecificationOverride, OnDemandThroughputOverride, and VectorIndexOverride are refused with ValidationException before any storage read; they were silently ignored. Throughput values reuse CreateTable's validator so the error text matches. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add backup_definitions (one row per backup, the definition as JSON behind a version marker; extenddb_storage::backup_definition) and table_restores (one row per restored table: source backup ARN and restore time, not a foreign key to backups so the summary outlives the backup, as on the service). SQLite's backup_items gains `id INTEGER PRIMARY KEY`; the restore cursors on it rather than the implicit rowid, which a VACUUM between batches could renumber. An existing backup_items is rebuilt in place (copy in rowid order, idempotent). backup_items.pk now holds the real partition key on SQLite as it does on PostgreSQL. Backups and restores from before the upgrade have no definition rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency. PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot is taken, and the snapshot is pinned by a real read of the table (LOCK TABLE is a utility statement and does not establish a REPEATABLE READ snapshot; the first revision relied on it and the definition and the items could come from different instants). UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. SQLite CreateBackup takes the engine write lock only to read the definition, insert the backup row as CREATING, and open the item reader's WAL snapshot; the items are then streamed without the lock and written in batches under short lock holds, so other writers wait for a batch rather than the whole backup (a 100,000-item backup previously stalled every write for about twelve seconds). The copy runs detached from the request, so a client that disconnects does not leave a half-written CREATING backup; one left by a crash is removed at startup. A CREATING backup is visible to ListBackups and DescribeBackup, as on the service, and DeleteBackup refuses it. The long reader pins WAL checkpointing for the backup's duration. A file-backed SQLite pool is clamped to at least two connections so the reader and the batch writer can coexist. A restore registers itself in the same transaction that creates its target on PostgreSQL (backup row FOR SHARE, table_restores inserted with the tables row) and SQLite, so DeleteBackup orders strictly before (the restore then sees no backup) or after (it sees the restore, BackupInUseException). MongoDB has no cross-collection transaction and uses a claim marker on the backup's metadata instead: a restore claims before reading and releases once its CREATING target exists; DeleteBackup claims the same way before dropping data; stale claims from a crashed process are cleared at startup. DeleteTable on a table being restored returns ResourceInUseException on all three backends. DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection (its idle_session_timeout SET is best-effort, for managed instances that refuse it); the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free, which with the pass's 60 s idle timeout means between 60 and about 120 s. SQLite removes the same targets at startup, which is safe because one server at a time holds the file (#391). The control-plane worker's DeleteTable drop runs under a 3 s lock_timeout and drops data tables before deleting the catalog row, so a drop blocked behind a backup's ACCESS SHARE lock fails fast and the row stays DELETING for the next pass (the previous order deleted the row first, so a failed drop was never retried). The stale-vector-index recovery skips indexes whose parent table is DELETING and cleans up if the parent vanishes under it, so the reordered drop cannot leave an orphan vector table. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and the unsupported overrides above. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported and its summary row is recounted; docs/differences-from- dynamodb.md gains a Backup and Restore section with every row qualified by backend (what is restored, the seven overrides, the refusals, CREATING visibility, the delete race, crash recovery, where backups live, the consistency model and WAL cost); the upgrade manual has a 0.0.4 section that warns PostgreSQL operators that targets the old bug left in CREATING are removed on the first control-plane pass after the upgrade, describes the SQLite lock, and notes WAL headroom; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (11, any backend: S/N/B hash and sort keys compared item by item; a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads; RestoreSummary; a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput; BillingModeOverride in both directions; a refused PROVISIONED override on a PAY_PER_REQUEST backup with a GSI leaving no target). crates/storage-postgres/tests/backup_restore.rs (21, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier including a production-path test that fails if the snapshot-pinning read is removed and the two transaction shapes that show why, half-built GSIs, DeleteTable and DeleteBackup racing a restore in both orders, the abandoned-restore sweep, the drop lock_timeout leaving a row DELETING for retry, RestoreSummary). SQLite restore_tests (16, incl. a writer that keeps making progress during a backup, items written after the snapshot excluded, a cancelled request finishing with every item, the startup sweeps, a 0.0.3 database in the old row format, the backup_items rebuild). Unit tests for the definition normalization and override preflight (12), the engine's override parsing and refusals (15), Mongo's claim predicates and GSI throughput handling, CatalogVersion ordering, the newer-catalog refusals, and the vector-recovery predicate. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 12, 2026
…; recover abandoned restores PostgreSQL and SQLite backup and restore now reproduce the table that was backed up, with bounded memory and crash recovery. This closes the two restore defects of the 1.0 readiness review's P0-2 (sort keys lost on PostgreSQL, indexes dropped on both). It does not add point-in-time recovery or off-host backup storage; those stay with #335, #336, and #337. Sort keys. create_backup and restore_table_from_backup on PostgreSQL checked for a column named `sk`, which data tables do not have (sk_s, sk_n, sk_b). Every backup of a table with a sort key lost its sort keys and restoring one left the target in CREATING for good. Backups now store pk and item_data only; restore derives every key column from the item. Backups written by earlier binaries carry the whole item in item_data and restore correctly. Restored, from a new per-backup table definition: global and local secondary indexes with key schemas, projections, and GSI throughput, filled during the copy; billing mode and provisioned throughput (was always 5/5); table class, SSE specification, on-demand throughput. Not restored, as on the service: streams, TTL, tags, deletion protection. A definition captured from a PAY_PER_REQUEST table carries no provisioned throughput for the table or its indexes, even when the catalog still holds values from before a switch from PROVISIONED, so a restored on-demand table reports what a freshly created one does. RestoreTableFromBackup overrides. BillingModeOverride and ProvisionedThroughputOverride are implemented on all three backends and passed to storage as an explicit `overrides: RestoreTableOverrides` parameter on BackupEngine::restore_table_from_backup (a trait change). BackupTableDefinition::apply_to applies them before the PAY_PER_REQUEST normalization and refuses, before any target is created, a PROVISIONED restore of a backup whose GSIs carry no throughput (the service needs GlobalSecondaryIndexOverride there). GlobalSecondaryIndexOverride, LocalSecondaryIndexOverride, SSESpecificationOverride, OnDemandThroughputOverride, and VectorIndexOverride are refused with ValidationException before any storage read; they were silently ignored. Throughput values reuse CreateTable's validator so the error text matches. Catalog 0.0.4. PostgreSQL migration 003 and the SQLite schema add backup_definitions (one row per backup, the definition as JSON behind a version marker; extenddb_storage::backup_definition) and table_restores (one row per restored table: source backup ARN and restore time, not a foreign key to backups so the summary outlives the backup, as on the service). SQLite's backup_items gains `id INTEGER PRIMARY KEY`; the restore cursors on it rather than the implicit rowid, which a VACUUM between batches could renumber. An existing backup_items is rebuilt in place (copy in rowid order, idempotent). backup_items.pk now holds the real partition key on SQLite as it does on PostgreSQL. Backups and restores from before the upgrade have no definition rows and behave as before. The migration runners write the compiled catalog version after every walk, so a replayed earlier migration cannot leave the version behind the schema; they refuse to write it when the stored version is newer, and `extenddb migrate` refuses a catalog newer than the binary outright, so an older binary can no longer stamp its version onto a newer schema and then pass its own startup gate. #335 also claims 0.0.4 and migration 003; whichever merges second renumbers. Consistency. PostgreSQL CreateBackup holds the source table row FOR SHARE from its first catalog read until its data snapshot is taken, and the snapshot is pinned by a real read of the table (LOCK TABLE is a utility statement and does not establish a REPEATABLE READ snapshot; the first revision relied on it and the definition and the items could come from different instants). UpdateTable and DeleteTable take the row FOR UPDATE, so the definition recorded is the one in force when the items were read. SQLite CreateBackup takes the engine write lock only to read the definition, insert the backup row as CREATING, and open the item reader's WAL snapshot; the items are then streamed without the lock and written in batches under short lock holds, so other writers wait for a batch rather than the whole backup (a 100,000-item backup previously stalled every write for about twelve seconds). The copy runs detached from the request, so a client that disconnects does not leave a half-written CREATING backup; one left by a crash is removed at startup. A CREATING backup is visible to ListBackups and DescribeBackup, as on the service, and DeleteBackup refuses it. The long reader pins WAL checkpointing for the backup's duration. A file-backed SQLite pool is clamped to at least two connections so the reader and the batch writer can coexist. A restore registers itself in the same transaction that creates its target on PostgreSQL (backup row FOR SHARE, table_restores inserted with the tables row) and SQLite, so DeleteBackup orders strictly before (the restore then sees no backup) or after (it sees the restore, BackupInUseException). MongoDB has no cross-collection transaction and uses a claim marker on the backup's metadata instead: a restore claims before reading and releases once its CREATING target exists; DeleteBackup claims the same way before dropping data; stale claims from a crashed process are cleared at startup. DeleteTable on a table being restored returns ResourceInUseException on all three backends. DescribeTable reports RestoreSummary for a restored table, in progress while CREATING; the restore response and later DescribeTable calls carry the same RestoreDateTime. Memory and concurrency: backup and restore stream items and buffer at most 500 items or 4 MiB of stored JSON. PostgreSQL runs up to four restores at once per process; a fifth waits up to 30 s and is refused with LimitExceededException. Failure and crash: a failed copy removes its target by table id, claiming it CREATING -> DELETING in one statement so removal and activation cannot both happen; MongoDB removes a partial target the same way. PostgreSQL restores hold a session advisory lock keyed by account and target name on a dedicated connection (its idle_session_timeout SET is best-effort, for managed instances that refuse it); the control-plane pass removes CREATING targets with no scheduled transition, older than 60 s, whose lock is free, which with the pass's 60 s idle timeout means between 60 and about 120 s. SQLite removes the same targets at startup, which is safe because one server at a time holds the file (#391). The control-plane worker's DeleteTable drop runs under a 3 s lock_timeout and drops data tables before deleting the catalog row, so a drop blocked behind a backup's ACCESS SHARE lock fails fast and the row stays DELETING for the next pass (the previous order deleted the row first, so a failed drop was never retried). The stale-vector-index recovery skips indexes whose parent table is DELETING and cleans up if the parent vanishes under it, so the reordered drop cannot leave an orphan vector table. Refused rather than restored wrongly: backups of tables with vector indexes (both backends now refuse; SQLite previously restored without them), backups of multi-part-key tables, definitions with an unknown version, and the unsupported overrides above. RestoreTableFromBackup validates TargetTableName. New DynamoDbError::BackupInUseException and StorageError::BackupInUse. Docs: docs/dynamodb-limits.md no longer says backup and restore are unsupported and its summary row is recounted; docs/differences-from- dynamodb.md gains a Backup and Restore section with every row qualified by backend (what is restored, the seven overrides, the refusals, CREATING visibility, the delete race, crash recovery, where backups live, the consistency model and WAL cost); the upgrade manual has a 0.0.4 section that warns PostgreSQL operators that targets the old bug left in CREATING are removed on the first control-plane pass after the upgrade, describes the SQLite lock, and notes WAL headroom; the admin guide's version literal and migrate guidance are current. Tests: tests/test_backup_restore_fidelity.py (11, any backend: S/N/B hash and sort keys compared item by item; a provisioned table with four GSIs and an LSI including projections, throughput, and ordered index reads; RestoreSummary; a PROVISIONED table switched to PAY_PER_REQUEST restoring with zero throughput; BillingModeOverride in both directions; a refused PROVISIONED override on a PAY_PER_REQUEST backup with a GSI leaving no target). crates/storage-postgres/tests/backup_restore.rs (21, in the PostgreSQL storage-level CI step: old formats, failure cleanup, the consistency barrier including a production-path test that fails if the snapshot-pinning read is removed and the two transaction shapes that show why, half-built GSIs, DeleteTable and DeleteBackup racing a restore in both orders, the abandoned-restore sweep, the drop lock_timeout leaving a row DELETING for retry, RestoreSummary). SQLite restore_tests (16, incl. a writer that keeps making progress during a backup, items written after the snapshot excluded, a cancelled request finishing with every item, the startup sweeps, a 0.0.3 database in the old row format, the backup_items rebuild). Unit tests for the definition normalization and override preflight (12), the engine's override parsing and refusals (15), Mongo's claim predicates and GSI throughput handling, CatalogVersion ordering, the newer-catalog refusals, and the vector-recovery predicate. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
Collaborator
|
Superseded by #394, which includes this branch's commits unchanged and adds one on top with the changes from the design review (see that PR's description). Leaving this open for the author to close or to pick the changes back here, whichever is preferred. |
robinnsc
added a commit
that referenced
this pull request
Oct 13, 2026
…hash, typed backup ARN Adds crates/core/src/backup_format, the on-disk format for off-host backups, built from #337's draft with the changes the design review asked for before v1 freezes. Data files: - DataFileEncoder<W: Write> hashes compressed bytes as they leave the compressor, so a writer streaming to a file or an upload holds the gzip window and one line, not the whole file. encode_data_file() remains as a convenience over it. A write error poisons the encoder (every later call is Encode); try_finish keeps the sink so an upload can be aborted, finish returns it; dropping a poisoned encoder takes the sink out first so flate2's Drop cannot push buffered data into it. A sink that accepts zero bytes is WriteZero, not a spin. is_finished and is_poisoned are distinct. GZIP_LEVEL = 6 is pinned so two runs of one build produce the same bytes. - DataFileDecoder<R: Read> hashes the compressed bytes it consumes and checks digests and counts at end of stream; lines are capped at MAX_LINE_BYTES and the recorded uncompressed size is enforced incrementally. Any error is sticky (later next_item calls return Decode naming the earlier failure, never Ok(None)); the Iterator impl is fused on top so error-dropping consumers terminate. Items are checked for JSON shape only; the restore validates them before writing. verified() keeps the slice path that checks digests before the first item. CompressedHasher is the incremental hasher a restore runs while spooling. Manifest: - Integrity is a sidecar, extenddb-manifest.sha256, in sha256sum format over the stored bytes (manifest_checksum / verify_manifest_checksum), replacing an in-document digest over a re-serialized struct that no other implementation could reproduce. The sidecar is written before the manifest, so the manifest's presence still marks a complete backup. - from_json_bytes gates on format_version before parsing the document, then checks identity (backup_arn parses, names source_table.table_name, and source_table.table_arn is a table ARN for the same account and table) and the data-file list (keys valid and unique, backup_size_bytes equals the sum, key_ordered manifests carry a key_range per file whose endpoints are key-only items of the schema's shape with first <= last and ranges ascending without overlap; unordered manifests carry none). - key_order.rs is the one definition of base-table key order: hash key by hash_key_text (S as is, N normalized, B base64) compared bytewise, range key S/B bytewise and N numerically. N endpoints must be valid DynamoDB numbers in normalized form. KeyOrder::resolve refuses multi-part keys and duplicate attribute definitions. The storage layer's pk_to_text delegates to hash_key_text. The module doc gives the per-backend ORDER BY recipe, including COLLATE "C" on Postgres text keys. - Optional source_table members serialize as explicit null; vector_indexes is three-state (null / [] / list) and holds the creation shape; sse_description, item_order, per-file md5 and uncompressed_bytes. - Golden fixture testdata/extenddb-manifest/v1.json + v1.sha256, compared byte for byte; regenerated only by an #[ignore]d test. DynamoDB export manifest: ExportFileEntry.etag is Option<String>; dataFileS3Key documented as the full bucket key. Layout: BackupArn is partition-aware, validates every component with the table-name charset rule (no minimum length beyond non-empty), and owns the ARN string and the store prefix. The backends' own ARN formatting and the engine's split-by-colon account check are not moved here yet. Tests: 90 in the module (89 run, 1 ignored fixture writer), plus a storage-sqlite test binding encode_orderable_number's byte order to KeyOrder::compare over a fixed pseudo-random corpus of DynamoDB numbers, so a change to either side fails. The ddb-export fixture is constructed from the documented output format, not captured from the service. Co-authored-by: Lee <leeroyhannigan@yahoo.ie>
robinnsc
added a commit
that referenced
this pull request
Oct 13, 2026
…t code Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture, and two review rounds against the companion PRs #393, #394 and #395. The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). The items-only signature in the draft allowed a GSI change or DeleteTable between the definition read and the data snapshot. Indexes are reported in whatever build state they had and only an index being deleted is dropped; an ACTIVE-only filter would have left a GSI mid-backfill out of the backup for good. - The snapshot is opened in the CreateBackup request so the backup represents the request time, as DynamoDB documents; backup_creation_date_time is the instant the snapshot pinned. - Per-job claims with fencing replace the single-instance lease the draft assumed; PostgreSQL and MongoDB have no such lease. The claim is an advisory lock on a dedicated catalog connection on PostgreSQL (the seed #384's restore lock already uses), the row itself on MongoDB, an in-process slot on SQLite; no row ever exists unowned. Heartbeats are a timer, staleness is measured on the database's clock, and a failed heartbeat aborts the job. The reconciler resolves a dead CREATING row at once under a fresh generation conditioned on status = CREATING and deletes its prefix only after the stale window. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty; backup sync is scoped to the configured store and never adopts a CREATING, DELETING, DELETED or FAILED prefix. Failed creations and deleted backups are BackupNotFound, as on the service, and share one retention. - Restore returns CREATING from the request (table and restore_jobs row created atomically; ResourceInUseException on a name race, LimitExceededException over the job limit), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data: a data-database row per file on PostgreSQL (data migration 005 reserved), the same transaction per 10k-item batch on SQLite, idempotent re-application of one file on MongoDB followed by a document-count check against the manifest. Duplicate detection for unordered backups is the target's primary key, not an in-memory key set. The ACTIVE flip is conditional; DeleteTable on a restore target fails the job. - The manifest self-checksum is replaced by a sidecar over the stored bytes; the draft's canonical form depended on serde_json formatting. backup_arn is the backup's identity and source_table must agree with it. N key values are normalized before key text or key order is computed; the SQL ORDER BY recipe includes COLLATE "C" on PostgreSQL. - Index population on restore stays per backend via bulk_load; the shared restore-backfill worker the draft described exists only on MongoDB. - Format additions: item_order, per-file key_range, md5 and uncompressed_bytes, explicit nulls, three-state vector_indexes of creation specs, optional etag, dataFileS3Key as the full object key, partition-aware BackupArn as the single ARN implementation. - Filesystem store: owner-only root re-checked by every operation, owner-only rule on every descended directory including list and delete_prefix, ancestor ownership rule, objects 0600 and directories 0700 at creation, put returns object metadata; the consequence for backup pickup agents is stated. The S3 conformance suite is mandatory in CI against moto (MinIO no longer publishes images). - ListBackups paging and filtering, the restore override contract, sqlite-memory behavior, extenddb destroy leaving the store intact, a restore spool outside the store root, max_concurrent_jobs, heartbeat_interval_secs and a concrete pool-size rule are specified. - Phase 1 is rewritten as the PR series with acceptance criteria and GitHub numbers. Migration numbers are reserved up front so the PITR PR is not on the critical path. Co-authored-by: Lee Hannigan <lhnng@amazon.com>
Collaborator
Author
|
Closing in favour of #394, which carries this branch's commit. |
robinnsc
added a commit
that referenced
this pull request
Oct 26, 2026
…hash, typed backup ARN Adds crates/core/src/backup_format, the on-disk format for off-host backups, built from #337's draft with the changes the design review asked for before v1 freezes. Data files: - DataFileEncoder<W: Write> hashes compressed bytes as they leave the compressor, so a writer streaming to a file or an upload holds the gzip window and one line, not the whole file. encode_data_file() remains as a convenience over it. A write error poisons the encoder (every later call is Encode); try_finish keeps the sink so an upload can be aborted, finish returns it; dropping a poisoned encoder takes the sink out first so flate2's Drop cannot push buffered data into it. A sink that accepts zero bytes is WriteZero, not a spin. is_finished and is_poisoned are distinct. GZIP_LEVEL = 6 is pinned so two runs of one build produce the same bytes. - DataFileDecoder<R: Read> hashes the compressed bytes it consumes and checks digests and counts at end of stream; lines are capped at MAX_LINE_BYTES and the recorded uncompressed size is enforced incrementally. Any error is sticky (later next_item calls return Decode naming the earlier failure, never Ok(None)); the Iterator impl is fused on top so error-dropping consumers terminate. Items are checked for JSON shape only; the restore validates them before writing. verified() keeps the slice path that checks digests before the first item. CompressedHasher is the incremental hasher a restore runs while spooling. Manifest: - Integrity is a sidecar, extenddb-manifest.sha256, in sha256sum format over the stored bytes (manifest_checksum / verify_manifest_checksum), replacing an in-document digest over a re-serialized struct that no other implementation could reproduce. The sidecar is written before the manifest, so the manifest's presence still marks a complete backup. - from_json_bytes gates on format_version before parsing the document, then checks identity (backup_arn parses, names source_table.table_name, and source_table.table_arn is a table ARN for the same account and table) and the data-file list (keys valid and unique, backup_size_bytes equals the sum, key_ordered manifests carry a key_range per file whose endpoints are key-only items of the schema's shape with first <= last and ranges ascending without overlap; unordered manifests carry none). - key_order.rs is the one definition of base-table key order: hash key by hash_key_text (S as is, N normalized, B base64) compared bytewise, range key S/B bytewise and N numerically. N endpoints must be valid DynamoDB numbers in normalized form. KeyOrder::resolve refuses multi-part keys and duplicate attribute definitions. The storage layer's pk_to_text delegates to hash_key_text. The module doc gives the per-backend ORDER BY recipe, including COLLATE "C" on Postgres text keys. - Optional source_table members serialize as explicit null; vector_indexes is three-state (null / [] / list) and holds the creation shape; sse_description, item_order, per-file md5 and uncompressed_bytes. - Golden fixture testdata/extenddb-manifest/v1.json + v1.sha256, compared byte for byte; regenerated only by an #[ignore]d test. DynamoDB export manifest: ExportFileEntry.etag is Option<String>; dataFileS3Key documented as the full bucket key. Layout: the backup id and ARN implementation moves from extenddb_storage::util::backup_id into extenddb_core::backup_format::layout (format_backup_id, is_backup_id, backup_id_epoch_millis, backup_creation_date_time, check_backup_arn, parse_backup_arn, BackupArnParts, BackupArnError, MAX_BACKUP_ARN_LEN), unchanged in behaviour; the field split is core::arn::fields, the one authorization uses. extenddb_storage::util re-exports them so the engine and the four backends keep their import path, and keeps new_backup_id (needs rand) and backup_arn (infallible, this deployment's partition). BackupArn is the owned form the format uses: check_backup_arn plus the store-prefix rules (table name at most 255 bytes and not only dots), with the ARN string and the store prefix. A test pins that backup_arn() and BackupArn::new() format the same string and that check_backup_arn and BackupArn::parse agree on every input except the dot-only name. Tests: 92 in the module (91 run, 1 ignored fixture writer), plus a storage-sqlite test binding encode_orderable_number's byte order to KeyOrder::compare over a fixed pseudo-random corpus of DynamoDB numbers, so a change to either side fails. The ddb-export fixture is constructed from the documented output format, not captured from the service. Co-authored-by: Lee <leeroyhannigan@yahoo.ie>
robinnsc
added a commit
that referenced
this pull request
Oct 26, 2026
…t code Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture, two review rounds against the companion PRs #393, #394 and #395, and a reconciliation against main 744ea29 after #384 merged in its final form and the Cassandra backend landed. The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). The items-only signature in the draft allowed a GSI change or DeleteTable between the definition read and the data snapshot. Indexes are reported in whatever build state they had and only an index being deleted is dropped; an ACTIVE-only filter would have left a GSI mid-backfill out of the backup for good. - The snapshot is opened in the CreateBackup request so the backup represents the request time, as DynamoDB documents; backup_creation_date_time is the instant the snapshot pinned. - Per-job claims with fencing replace the single-instance lease the draft assumed; PostgreSQL and MongoDB have no such lease. The claim is an advisory lock on a dedicated catalog connection on PostgreSQL (the seed #384's restore lock already uses), the row itself on MongoDB, an in-process slot on SQLite; no row ever exists unowned. Heartbeats are a timer, staleness is measured on the database's clock, and a failed heartbeat aborts the job. The reconciler resolves a dead CREATING row at once under a fresh generation conditioned on status = CREATING and deletes its prefix only after the stale window. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty; backup sync is scoped to the configured store and never adopts a CREATING, DELETING, DELETED or FAILED prefix. Failed creations and deleted backups are BackupNotFound, as on the service, and share one retention. - Restore returns CREATING from the request (table and restore_jobs row created atomically; ResourceInUseException on a name race, LimitExceededException over the job limit), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data: a data-database row per file on PostgreSQL (data migration 005 reserved), the same transaction per 10k-item batch on SQLite, idempotent re-application of one file on MongoDB followed by a document-count check against the manifest. Duplicate detection for unordered backups is the target's primary key, not an in-memory key set. The ACTIVE flip is conditional; DeleteTable on a restore target fails the job. - The manifest self-checksum is replaced by a sidecar over the stored bytes; the draft's canonical form depended on serde_json formatting. backup_arn is the backup's identity and source_table must agree with it. N key values are normalized before key text or key order is computed; the SQL ORDER BY recipe includes COLLATE "C" on PostgreSQL. - Index population on restore stays per backend via bulk_load; the shared restore-backfill worker the draft described exists only on MongoDB. - Format additions: item_order, per-file key_range, md5 and uncompressed_bytes, explicit nulls, three-state vector_indexes of creation specs, optional etag, dataFileS3Key as the full object key, partition-aware BackupArn as the single ARN implementation. - Filesystem store: owner-only root re-checked by every operation, owner-only rule on every descended directory including list and delete_prefix, ancestor ownership rule, objects 0600 and directories 0700 at creation, put returns object metadata; the consequence for backup pickup agents is stated. The S3 conformance suite is mandatory in CI against moto (MinIO no longer publishes images). - ListBackups paging and filtering, the restore override contract, sqlite-memory behavior, extenddb destroy leaving the store intact, a restore spool outside the store root, max_concurrent_jobs, heartbeat_interval_secs and a concrete pool-size rule are specified. - Reconciled with #384 as merged: the restore job record extends #384's table_restores / provenance fields rather than adding a table; the backup id (14-digit, new_backup_id) and ARN parser #384 introduced are the one implementation, moved into extenddb-core by PR 4 and re-exported by storage; the claim names #384's RESTORE_LOCK_SEED, 15 s heartbeat and slot semaphores as what it replaces; defaults follow it (15 s heartbeat, 300 s stale window). Cassandra (#338) implements SnapshotEngine as Unsupported and keeps its logical backup until a token-range snapshot is accepted or the backend's backups are documented as unavailable, decided before PR 14. - Phase 1 is rewritten as the PR series with acceptance criteria and GitHub numbers. Migration numbers are reserved up front so the PITR PR is not on the critical path. Co-authored-by: Lee Hannigan <lhnng@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 26, 2026
…hash, typed backup ARN Adds crates/core/src/backup_format, the on-disk format for off-host backups, built from #337's draft with the changes the design review asked for before v1 freezes. Data files: - DataFileEncoder<W: Write> hashes compressed bytes as they leave the compressor, so a writer streaming to a file or an upload holds the gzip window and one line, not the whole file. encode_data_file() remains as a convenience over it. encode_item_line is the one encoding of the {"Item": ...} line; the engine's ExportTableToPointInTime writer calls it. write_item refuses a line over MAX_LINE_BYTES (4 MiB) with the encoder still usable, so a backup cannot contain a line the decoder would refuse at restore; the cap's sizing (7.5x worst-case DynamoDB JSON expansion of a 400 KB item, 1.37x margin, operator-raised item limits above about 559 KB can exceed it) is in its doc. A write error poisons the encoder (every later call is Encode); try_finish keeps the sink so an upload can be aborted, finish returns it; dropping a poisoned encoder takes the sink out first so flate2's Drop cannot push buffered data into it. A sink that accepts zero bytes is WriteZero, not a spin; an Interrupted sink is retried on every path. is_finished and is_poisoned are distinct. GZIP_LEVEL = 6 is pinned so two runs of one build produce the same bytes. - DataFileDecoder<R: Read> hashes the compressed bytes it consumes and checks digests and counts at end of stream; lines are capped at MAX_LINE_BYTES and the recorded uncompressed size is enforced incrementally. Any error is sticky (later next_item calls return Decode naming the earlier failure, never Ok(None)); the Iterator impl is fused on top so error-dropping consumers terminate. Items are checked for JSON shape only; the restore validates them before writing. verified() keeps the slice path that checks digests before the first item. CompressedHasher is the incremental hasher a restore runs while spooling. Manifest: - Integrity is a sidecar, extenddb-manifest.sha256, in sha256sum format over the stored bytes (manifest_checksum / verify_manifest_checksum), replacing an in-document digest over a re-serialized struct that no other implementation could reproduce. The sidecar is written before the manifest, so the manifest's presence still marks a complete backup. - from_json_bytes gates on format_version before parsing the document, then checks identity (backup_arn is in canonical form and parses; its id's instant equals backup_creation_date_time; backup_name is 3-255 of [a-zA-Z0-9_.-]; source_table.table_name is the ARN's table; source_table.table_arn, read with core::arn::parse, is a table ARN in the same partition for the same account and table) and the data-file list (keys valid and unique, sha256 64 lowercase hex, md5 24 base64 characters, a file with items has non-zero sizes, backup_size_bytes equals the sum, key_ordered manifests carry a key_range per file whose endpoints are key-only items of the schema's shape with first <= last and ranges ascending without overlap; unordered manifests carry none; an empty table is a manifest with no files). - key_order.rs is the one definition of base-table key order: hash key by hash_key_text (S as is, N normalized, B base64) compared bytewise, range key S/B bytewise and N numerically. N endpoints must be valid DynamoDB numbers; parsed input arrives normalized (the deserializer rewrites 5.0 to 5) and items built in code are checked for normalized form. SnapshotDescriptor carries the three KIND_* constants the backends report, so the manifest and the storage layer share the strings. KeyOrder::resolve refuses multi-part keys and duplicate attribute definitions. The storage layer's pk_to_text delegates to hash_key_text. The module doc gives the per-backend ORDER BY recipe, including COLLATE "C" on Postgres text keys. - Optional source_table members serialize as explicit null; vector_indexes is three-state (null / [] / list) and holds the creation shape; sse_description, item_order, per-file md5 and uncompressed_bytes. - Golden fixture testdata/extenddb-manifest/v1.json + v1.sha256, compared byte for byte; regenerated only by an #[ignore]d test. DynamoDB export manifest: ExportFileEntry.etag is Option<String>; dataFileS3Key documented as the full bucket key. Layout: the backup id and ARN implementation moves from extenddb_storage::util::backup_id into extenddb_core::backup_format::layout (format_backup_id, is_backup_id, backup_id_epoch_millis, backup_creation_date_time, check_backup_arn, parse_backup_arn, BackupArnParts, BackupArnError, MAX_BACKUP_ARN_LEN), unchanged in behaviour (verified by a reviewer's differential run over 5.9M inputs); the field split is core::arn::fields and the table-name alphabet is validation::validate_table_name_chars, so authorization, CreateTable and this parser share them. format_backup_arn is the one formatter (backup_arn() in storage and BackupArn call it) and trim_backup_arn the one whitespace rule (check_backup_arn and the two authorization sites call it). extenddb_storage::util re-exports the rest so the engine and the four backends keep their import path, and keeps new_backup_id (needs rand), backup_arn, and is_backup_resource_arn (the authorization predicate). BackupArn is the owned form the format uses: check_backup_arn plus the two store-prefix rules (table name at most 255 bytes and not only dots), with the ARN string and the store prefix; a test enumerates both divergences and the id-shape grid. Tests: 105 in the module (104 run, 1 ignored fixture writer), plus a storage-sqlite test binding encode_orderable_number's byte order to KeyOrder::compare over a fixed pseudo-random corpus of DynamoDB numbers, so a change to either side fails, and a row-key contract test in storage pinning pk_to_text's literal output (standard base64, "+/+/") so a change to hash_key_text cannot silently re-key B rows. The ddb-export fixture is constructed from the documented output format, not captured from the service. Round-3 review additions: the streaming decoder's end-of-stream digest check (level-1 re-encode of the same items), DataFileEntry::decoder carrying every expectation, the incremental byte cap firing early, try_finish failure poisoning, Interrupted retry, invalid UTF-8 as Decode, an unterminated final line as an item, the hash-key type check in compare, and the identity and digest-shape rules above. Co-authored-by: Lee <leeroyhannigan@yahoo.ie>
robinnsc
added a commit
that referenced
this pull request
Oct 26, 2026
Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), three review rounds against the companion PRs #393, #394 and #395, and a reconciliation against main 744ea29 after #384 merged in its final form and the Cassandra backend landed. The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). The items-only signature in the draft allowed a GSI change or DeleteTable between the definition read and the data snapshot. Indexes are reported in whatever build state they had and only an index being deleted is dropped; an ACTIVE- only filter would have left a GSI mid-backfill out of the backup for good. - The snapshot is opened in the CreateBackup request so the backup represents the request time (stricter than the service's two- minute window). The backup id is minted from the instant the snapshot pinned, so backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed; PostgreSQL and MongoDB have no such lease. The claim is an advisory lock on a dedicated catalog connection on PostgreSQL (the seed #384's restore lock already uses), the row itself on MongoDB, an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets only); no row ever exists unowned (the MongoDB row is born through an upsert pipeline so its first heartbeat is the server's clock). Heartbeats are a timer, staleness is measured on the database's clock, and any heartbeat that does not confirm the claim aborts the job. A job runs the failure path's delete_prefix only after its FAILED write matched a row; otherwise the claim holder owns the prefix. The reconciler resolves a dead CREATING row at once under a fresh generation conditioned on status = CREATING, leaves heartbeat_at untouched, and deletes the prefix only after the stale window; its duties are listed once, in section 4. A sink error fails the backup; there is no per-file restart, which the one-shot stream and the memory bound rule out. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty; backup sync is scoped to the configured store and never adopts a CREATING, DELETING, DELETED or FAILED prefix. Failed creations and deleted backups are BackupNotFound, as on the service, and share one retention. - Restore returns CREATING from the request (table and restore_jobs row created atomically in a detached task; TableAlreadyExistsException or TableInUseException on a name race, as #384 already returns; LimitExceededException over the job limit), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data: a data-database row per file on PostgreSQL (data migration 005 reserved), the same transaction per 10k-item batch on SQLite, idempotent re-application of one file on MongoDB followed by a document-count check against the manifest. Duplicate detection for unordered backups is the target's primary key, not an in-memory key set. The ACTIVE flip is conditional and precedes the deletion of the PostgreSQL restore_progress rows, which live in the data database and cannot share its transaction. DeleteTable on a restore target fails the job (a change from #384, which refuses it); multi-part-key tables are restored (#384 refuses them). The override paragraph now states what #384 ships: every override but VectorIndexOverride is applied, and the new path keeps that. - The manifest self-checksum is replaced by a sidecar over the stored bytes; the draft's canonical form depended on serde_json formatting. backup_arn is the backup's identity and source_table must agree with it. N key values are normalized before key text or key order is computed; #395's PostgreSQL ORDER BY carries COLLATE "C" so tables created before #361 stream in the format's order. - Index population on restore stays per backend via bulk_load; the shared restore-backfill worker the draft described exists only on MongoDB. - Format additions: item_order, per-file key_range, md5 and uncompressed_bytes, explicit nulls, three-state vector_indexes of creation specs, optional etag, dataFileS3Key as the full object key, partition-aware BackupArn as the single ARN implementation. - Filesystem store: owner-only root re-checked by every operation, owner-only rule on every descended directory including list and delete_prefix, ancestor ownership rule, objects 0600 and directories 0700 at creation, put returns object metadata; the consequence for backup pickup agents is stated; list and delete_prefix resolve their prefix with the same no-follow walk as a key and tolerate entries a concurrent deleter removed. The S3 conformance suite is mandatory in CI against moto, with the 301/403/abort paths driven through an interceptor. Store settings live under [backup.store], matching #393. - ListBackups keeps #384's handler (full request already on main) over the catalog index merged with the legacy reader; sqlite-memory behavior, extenddb destroy leaving the store intact, a restore spool outside the store root, max_concurrent_jobs, heartbeat_interval_secs, a memory formula in bytes with the item-size term, and a pool-size rule that clamps and warns are specified. Restore and Describe never read the store without a catalog row; backup sync is the path in. - Reconciled with #384 as merged: the restore job record extends #384's table_restores / provenance fields rather than adding a table; the backup ARN parser and formatter #384 introduced are the one implementation, moved into extenddb-core by PR 4 and re-exported by storage (new_backup_id stays in storage and calls the shared formatter); the claim names #384's RESTORE_LOCK_SEED, lock key, 15 s heartbeat and slot semaphores as what it keeps or replaces; the 300 s stale window is new (#384 uses 60 s). Cassandra (#338) implements SnapshotEngine as Unsupported and keeps its logical backup until a token-range snapshot is accepted or the backend's backups are documented as unavailable, decided before PR 14. - Phase 1 is rewritten as the PR series with acceptance criteria and GitHub numbers. Migration numbers are reserved up front so the PITR PR is not on the critical path (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend. Also adds the missing rows for designs 12 and 13 to docs/design/README.md. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 26, 2026
…hash, typed backup ARN Adds crates/core/src/backup_format, the on-disk format for off-host backups, built from #337's draft with the changes the design review asked for before v1 freezes. Data files: - DataFileEncoder<W: Write> hashes compressed bytes as they leave the compressor, so a writer streaming to a file or an upload holds the gzip window and one line, not the whole file. encode_data_file() remains as a convenience over it. encode_item_line is the one encoding of the {"Item": ...} line; the engine's ExportTableToPointInTime writer calls it. write_item refuses a line over MAX_LINE_BYTES (4 MiB) with the encoder still usable, so a backup cannot contain a line the decoder would refuse at restore; the cap's sizing (a list of empty strings is the worst case, 9x: a 400 KB item becomes a 3.69 MB line, a 1.14x margin; operator-raised item limits above about 466 KB can exceed it) is in its doc and pinned by a test. A write error poisons the encoder (every later call is Encode); try_finish keeps the sink so an upload can be aborted, finish returns it. Only a successful finish writes the gzip trailer: dropping an encoder in any other state (fresh, open, poisoned) takes the sink out first, so flate2's Drop cannot complete a valid file of the items so far in a sink the caller abandoned. A sink that accepts zero bytes is WriteZero, not a spin; Interrupted is retried on every sink write and flush, and on the decoder's source reads. is_finished and is_poisoned are distinct. GZIP_LEVEL = 6 is pinned so two runs of one build produce the same bytes. - DataFileDecoder<R: Read> hashes the compressed bytes it consumes and checks digests and counts at end of stream; lines are capped at MAX_LINE_BYTES and the recorded uncompressed size is enforced incrementally. Any error is sticky (later next_item calls return Decode naming the earlier failure, never Ok(None)); the Iterator impl is fused on top so error-dropping consumers terminate. Items are checked for JSON shape only; the restore validates them before writing. verified() keeps the slice path that checks digests before the first item. A line ends in \n and holds one object with the one member Item (an unknown member is refused); JSON whitespace, including a \r before the newline, is accepted as JSON accepts it. CompressedHasher is the incremental hasher a restore runs while spooling. Manifest: - Integrity is a sidecar, extenddb-manifest.sha256, in sha256sum format over the stored bytes (manifest_checksum / verify_manifest_checksum), replacing an in-document digest over a re-serialized struct that no other implementation could reproduce. The sidecar is written before the manifest, so the manifest's presence still marks a complete backup. - from_json_bytes gates on format_version before parsing the document, then checks identity (backup_arn is in canonical form and parses; backup_creation_date_time is exactly the f64 its id's instant gives; backup_name is 3-255 of the table-name alphabet; source_table.table_name is the ARN's table; source_table.table_arn, read with core::arn::parse, is a table ARN in the same partition for the same account and table) and the data-file list (keys valid and unique, sha256 64 lowercase hex, md5 24 base64 characters ending "==", every file has items and non-zero sizes, backup_size_bytes equals the sum, key_ordered manifests carry a key_range per file whose endpoints are key-only items of the schema's shape with first <= last and ranges ascending without overlap; unordered manifests carry none; an empty table is a manifest with no files). A key_range endpoint naming an attribute twice is refused at parse time. Error messages carry at most 80 bytes of an untrusted field. - key_order.rs is the one definition of base-table key order: hash key by hash_key_text (S as is, N as held, B base64) compared bytewise, range key S/B bytewise and N numerically. N endpoints must be valid DynamoDB numbers; parsed input arrives normalized (the deserializer rewrites 5.0 to 5) and items built in code are checked for normalized form. SnapshotDescriptor carries the three KIND_* constants the backends report, so the manifest and the storage layer share the strings. KeyOrder::resolve refuses multi-part keys and duplicate attribute definitions. The storage layer's pk_to_text delegates to hash_key_text. The module doc gives the per-backend ORDER BY recipe, including COLLATE "C" on Postgres text keys. - Optional source_table members serialize as explicit null; vector_indexes is three-state (null / [] / list) and holds the creation shape; sse_description, item_order, per-file md5 and uncompressed_bytes. - Golden fixture testdata/extenddb-manifest/v1.json + v1.sha256, compared byte for byte; regenerated only by an #[ignore]d test. DynamoDB export manifest: ExportFileEntry.etag is Option<String>; dataFileS3Key documented as the full bucket key. Layout: the backup id and ARN implementation moves from extenddb_storage::util::backup_id into extenddb_core::backup_format::layout (format_backup_id, is_backup_id, backup_id_epoch_millis, backup_creation_date_time, check_backup_arn, parse_backup_arn, BackupArnParts, BackupArnError, MAX_BACKUP_ARN_LEN), unchanged in behaviour (verified by a reviewer's differential run over 5.9M inputs); the field split is core::arn::fields and the table-name alphabet is validation::validate_table_name_chars, so authorization, CreateTable and this parser share them. format_backup_arn is the one backup-ARN formatter (backup_arn() in storage, BackupArn, and the server's canonical resource ARN call it) and trim_backup_arn the one whitespace rule (check_backup_arn and the two authorization sites call it). extenddb_storage::util re-exports the rest so the engine and the four backends keep their import path, and keeps new_backup_id (needs rand), backup_arn, and is_backup_resource_arn (the authorization predicate). BackupArn is the owned form the format uses: check_backup_arn plus the two store-prefix rules (table name at most 255 bytes and not only dots), with the ARN string and the store prefix; a test enumerates both divergences and the id-shape grid. The 255-byte bound is narrower than the engine's max_table_name_length when an operator raises it; the layout doc says so. Other copies of ARN facts in the server and storage crates (is_backup_resource_arn's own split, a partition constant) predate this PR and are not changed here. Tests: 105 in the module (104 run, 1 ignored fixture writer), plus a storage-sqlite test binding encode_orderable_number's byte order to KeyOrder::compare over a fixed pseudo-random corpus of DynamoDB numbers, so a change to either side fails, and a row-key contract test in storage pinning pk_to_text's literal output (standard base64, "+/+/") so a change to hash_key_text cannot silently re-key B rows. The ddb-export fixture is constructed from the documented output format, not captured from the service. Round-3 review additions: the streaming decoder's end-of-stream digest check (level-1 re-encode of the same items), DataFileEntry::decoder carrying every expectation, the incremental byte cap firing early, try_finish failure poisoning, Interrupted retry, invalid UTF-8 as Decode, an unterminated final line as an item, the hash-key type check in compare, and the identity and digest-shape rules above. Round-4 additions: the 9x worst-case item at the default limit fits the cap; dropping a fresh or open encoder writes nothing; Interrupted on the trailer write, on flush and on the source; a failing flush poisons; the end-of-stream checks on a zero-item file; an unknown member refused and CRLF accepted; range-key pairs that tell bytewise from length-first and from base64 order; md5 shape cells, an empty table-ARN region, a lowercase table name, a slash in backup_name, last-endpoint validation, version 0, a missing array member, zero-item files, duplicate key_range attributes, and bounded error messages. The golden fixture writer runs only with EXTENDDB_WRITE_GOLDEN set. Co-authored-by: Lee <leeroyhannigan@yahoo.ie>
robinnsc
added a commit
that referenced
this pull request
Oct 26, 2026
Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), four review rounds against the companion PRs #393, #394 and #395, and a reconciliation against main abf8082 (#384 in its final form, the Cassandra backend #338, the PostgreSQL GSI build #401). The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). Indexes are reported in whatever build state they had and only an index being deleted is dropped; a GSI without its data table is left out with the attribute definitions only it used. PostgreSQL bounds open snapshots per server and refuses more with LimitExceeded; DeleteTable takes its asynchronous path while a reader holds the data table. - The snapshot is opened in the CreateBackup request (stricter than the service's two-minute window), and the backup id is minted from the instant it pinned, so backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (#384's seed and key), the row itself on MongoDB (born through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read. A job deletes a prefix, or a restore target, only after its FAILED write matched a row. The reconciler's duties are listed once, in section 4, and include the prefixes of DELETED rows, orphan restore targets and stale spools. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty. backup sync is scoped to the configured store, never adopts a CREATING, DELETING, DELETED or FAILED prefix, and marks a row whose manifest head is gone DELETING. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. - Restore returns CREATING from the request (TableAlreadyExists or TableInUse on a name race, as #384 returns), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data on each backend. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as #384 and the service refuse it; the restore record is the table's child. Multi-part-key tables are restored (#384 refuses them). Table class is not restored, as on the service and #384; every override but VectorIndexOverride is applied, as #384 applies them. - The manifest self-checksum is replaced by a sidecar over the stored bytes; backup_arn is the backup's identity and source_table must agree with it; N key values are normalized before key text or key order is computed; #395's PostgreSQL ORDER BY carries COLLATE "C". - Store (#393): owner-only root and directories re-checked by every operation, the prefix resolved with the same no-follow walk as a key, listings that report only keys passing the key rule, multipart uploads aborted when a put is dropped and swept by delete_prefix (a lifecycle rule as backstop), validate by a one-key listing so a 403 keeps S3's code, settings under [backup.store]. - Configuration: the default store root is per backend (beside the database file on SQLite); PostgreSQL, MongoDB and Cassandra refuse CreateBackup until [backup.store] is set, and keep serving legacy backups. Cassandra keeps creating in-catalog backups through the legacy reader until PR 14. The memory formula counts the fetch buffer and caps a restore batch at one file; purge holds every job permit. - Phase 1 is the PR series with acceptance criteria and GitHub numbers; migration numbers are reserved up front (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend and for DeleteBackup. Also adds the missing rows for designs 12 and 13 to docs/design/README.md and marks 15 as a draft. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 27, 2026
…hash, typed backup ARN Adds crates/core/src/backup_format, the on-disk format for off-host backups, built from #337's draft with the changes the design review asked for before v1 freezes. Data files: - DataFileEncoder<W: Write> hashes compressed bytes as they leave the compressor, so a writer streaming to a file or an upload holds the gzip window and one line, not the whole file. encode_data_file() remains as a convenience over it. encode_item_line is the one encoding of the {"Item": ...} line; the engine's ExportTableToPointInTime writer calls it. write_item refuses a line over MAX_LINE_BYTES (32 MiB) with the encoder still usable, so a backup cannot contain a line the decoder would refuse at restore. The worst case at the default 400 KB item limit is a list of number sets, 67.6x (a 27.7 MB line, a 1.21x margin); operator-raised item limits above about 496 KB can exceed the cap. The doc carries the measured table and a test pins the worst case. A write error poisons the encoder (every later call is Encode); try_finish keeps the sink so an upload can be aborted, finish returns it. Only a successful finish writes the gzip trailer: dropping an encoder in any other state (fresh, open, poisoned) takes the sink out first, so flate2's Drop cannot complete a valid file of the items so far in a sink the caller abandoned. A sink that accepts zero bytes is WriteZero, not a spin; Interrupted is retried on every sink write and flush, and on the decoder's source reads. is_finished and is_poisoned are distinct. GZIP_LEVEL = 6 is pinned, and a test pins the output length and sha256 of a fixed input, so two runs of one build produce the same bytes. - DataFileDecoder<R: Read> hashes the compressed bytes it consumes. The recorded compressed size and uncompressed size are enforced while reading (a source longer than its recorded compressed size fails with CompressedSizeMismatch after one byte past it); digests and the item count are checked at end of stream. Lines are capped at MAX_LINE_BYTES. Decoding one line holds about 82 bytes of memory per line byte (2.56 GiB at the cap, measured by an ignored test), because the AttributeValue deserializer builds a serde_json::Value of the whole value; the doc says so and states the trust boundary (only a principal that can write to the store can forge such a file). Any error is sticky (later next_item calls return Decode naming the earlier failure, never Ok(None)); the Iterator impl is fused on top so error-dropping consumers terminate. Items are checked for JSON shape only; the restore validates them before writing. verified() keeps the slice path that checks digests before the first item. A line ends in \n and holds one object with the one member Item (an unknown member or a repeated top-level attribute is refused; nested maps keep JSON's last-wins); JSON whitespace, including a \r before the newline, is accepted as JSON accepts it. CompressedHasher is the incremental hasher a restore runs while spooling. Manifest: - Integrity is a sidecar, extenddb-manifest.sha256, in sha256sum format over the stored bytes (manifest_checksum / verify_manifest_checksum), replacing an in-document digest over a re-serialized struct that no other implementation could reproduce. The sidecar is accepted in one shape only: 64 lowercase hex digits, optionally followed by two spaces and extenddb-manifest.json, optionally one \n; anything else is InvalidManifest. The sidecar is written before the manifest, so the manifest's presence still marks a complete backup. - from_json_bytes gates on format_version before parsing the document, then checks identity (backup_arn is in canonical form and parses; backup_creation_date_time is exactly the f64 its id's instant gives; backup_name is 3-255 of the table-name alphabet; source_table.table_name is the ARN's table; source_table.table_arn, read with core::arn::parse, is a table ARN in the same partition for the same account and table) and the data-file list (keys valid and unique, sha256 64 lowercase hex, md5 the standard base64 of 16 bytes, every file has items and non-zero sizes, backup_size_bytes equals the sum without overflow, key_ordered manifests carry a key_range per file whose endpoints are key-only items of the schema's shape with first <= last and ranges ascending without overlap against every earlier file; unordered manifests carry none; an empty table is a manifest with no files). A key_range endpoint naming an attribute twice is refused at parse time. - Error messages carry a bounded slice of any untrusted field: at most 80 bytes of its escaped rendering, otherwise a 48-byte prefix and the total length. FormatError::Json bounds serde's text (which quotes the input) to 120 bytes and does not expose the inner error as source(). - key_order.rs is the one definition of base-table key order: hash key by hash_key_text (S as is, N as held, B base64) compared bytewise, range key S/B bytewise and N numerically. N endpoints must be valid DynamoDB numbers; parsed input arrives normalized (the deserializer rewrites 5.0 to 5) and items built in code are checked for normalized form. SnapshotDescriptor carries the three KIND_* constants the backends report, so the manifest and the storage layer share the strings. KeyOrder::resolve refuses multi-part keys and duplicate attribute definitions. The storage layer's pk_to_text delegates to hash_key_text. The module doc gives the per-backend ORDER BY recipe, including COLLATE "C" on Postgres text keys. - Optional source_table members serialize as explicit null; vector_indexes is three-state (null / [] / list) and holds the creation shape; sse_description, item_order, per-file md5 and uncompressed_bytes; the per-file compressed size feeds the decoder's expectations. - Golden fixtures testdata/extenddb-manifest/v1.json and v1-minimal.json (every optional member absent, every list empty), each with its .sha256, compared byte for byte; regenerated only by an #[ignore]d test run with EXTENDDB_WRITE_GOLDEN set. DynamoDB export manifest: ExportFileEntry.etag is Option<String>; dataFileS3Key documented as the full bucket key (<s3Prefix>/AWSDynamoDB/<export id>/data/<file>). Layout: the backup id and ARN implementation moves from extenddb_storage::util::backup_id into extenddb_core::backup_format::layout (format_backup_id, is_backup_id, backup_id_epoch_millis, backup_creation_date_time, check_backup_arn, parse_backup_arn, BackupArnParts, BackupArnError, MAX_BACKUP_ARN_LEN), unchanged in behaviour (verified by a reviewer's differential run over 5.9M inputs); the field split is core::arn::fields and the table-name alphabet is validation::validate_table_name_chars, so authorization, CreateTable and this parser share them. format_backup_arn is the one backup-ARN formatter (backup_arn() in storage, BackupArn, and the server's canonical resource ARN call it) and trim_backup_arn the one whitespace rule (check_backup_arn and the two authorization sites call it). It trims both ends as main did; the doc notes that the service resolves a trailing newline but refuses a leading space or tab. extenddb_storage::util re-exports the rest so the engine and the four backends keep their import path, and keeps new_backup_id (needs rand), backup_arn, and is_backup_resource_arn (the authorization predicate). BackupArn is the owned form the format uses: check_backup_arn plus the two store-prefix rules (table name at most 255 bytes and not only dots), with the ARN string and the store prefix; a test enumerates both divergences and the id-shape grid. The 255-byte bound is narrower than the engine's max_table_name_length when an operator raises it; the layout doc says so. Other copies of ARN facts in the server and storage crates (is_backup_resource_arn's own split, a partition constant) predate this PR and are not changed here. Tests: 131 in the module (130 run, 1 ignored fixture writer), two integration binaries in crates/core/tests (data_file_line_bomb: a 256 MiB single-line bomb stops within twice the cap's memory; data_file_decode_peak: the ignored RSS measurement above), a storage-sqlite test binding encode_orderable_number's byte order to KeyOrder::compare over a fixed pseudo-random corpus of DynamoDB numbers, so a change to either side fails, and a row-key contract test in storage pinning pk_to_text's literal output (standard base64, "+/+/") so a change to hash_key_text cannot silently re-key B rows. The ddb-export fixture is constructed from the documented output format, not captured from the service. Coverage includes the streaming end-of-stream digest check, every expectation a DataFileEntry carries, the uncompressed and compressed size caps firing early, wrong item counts in both directions, try_finish failure poisoning, Interrupted on every sink and source call, invalid UTF-8 as Decode, an unterminated final line as an item, dropped encoders writing nothing, range-key pairs that tell bytewise from length-first and from base64 order, B hash keys in base64 order, overlap between later files, last-endpoint validation, size-sum overflow, md5 and sidecar shapes, identity rules, explicit nulls, and bounded messages at field lengths from 81 bytes to 2 MB. 32 hand mutants of these rules were each killed by the suite. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 27, 2026
Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), five review rounds against the companion PRs #393, #394 and #395, and a reconciliation against main abf8082 (#384 in its final form, the Cassandra backend #338, the PostgreSQL GSI build #401). The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). Indexes are reported in whatever build state they had and only an index being deleted is dropped; a GSI without its data table is left out with the attribute definitions only it used. PostgreSQL bounds open snapshots at half the pool and refuses more with LimitExceeded, streams a table whose key columns predate COLLATE "C" unordered rather than sort it, and DeleteTable takes its asynchronous path while a reader holds any relation it would drop. On sqlite-memory the snapshot copies the table on the one connection, so CreateBackup keeps working there. MongoDB reads without the explicit snapshot level fail, and the document says so with the server's error. - The snapshot is opened in the CreateBackup request (stricter than the service's two-minute window), and the backup id is minted from the instant it pinned, so backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (#384's seed and key), the row itself on MongoDB (born through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read. A job deletes a prefix, or a restore target, only after its FAILED write matched a row. The reconciler's duties are listed once, in section 4, and include the prefixes of DELETED rows, orphan restore targets, stale spools and, on Cassandra, restores_by_backup entries with no RUNNING job (that entry cannot share a lightweight transaction with the claim, so it is written first and is only a hint). Every duty that deletes a prefix acts only on rows of the configured store. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty. backup sync is scoped to the configured store (backup sync --move-store re-points the rows of a copied store), never adopts a CREATING, DELETING, DELETED or FAILED prefix, and marks a row whose manifest head is gone DELETING. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. - Restore returns CREATING from the request (TableAlreadyExists or TableInUse on a name race, as #384 returns), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data on each backend. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as #384 and the service refuse it; the restore record is the table's child. Multi-part-key tables are restored (#384 refuses them). Table class is not restored, as on the service and #384; every override but VectorIndexOverride is applied, as #384 applies them. - The manifest self-checksum is replaced by a sidecar over the stored bytes; backup_arn is the backup's identity and source_table must agree with it; N key values are normalized before key text or key order is computed. The line cap is 32 MiB, sized from the measured worst case, with its decode memory stated; the compressed size is enforced while reading. - Store (#393): owner-only root and directories re-checked by every operation, the prefix resolved with the same no-follow walk as a key, listings that report only keys passing the key rule, multipart uploads aborted when a put is dropped and swept by delete_prefix (a lifecycle rule as backstop), validate by a one-key listing so a 403 keeps S3's code and then by probes of the write grants, a 255-byte component cap and U+FFFE/U+FFFF refused by the key rule, settings under [backup.store]. Section 8 gains a transport rule (a non-loopback http:// endpoint needs allow_insecure_endpoint) and an IAM example with the list actions on the bucket. - Configuration: the default store root is per backend (beside the database file on SQLite); PostgreSQL, MongoDB and Cassandra refuse CreateBackup until [backup.store] is set, and keep serving legacy backups. Cassandra keeps creating in-catalog backups through the legacy reader until PR 14. The memory formula counts the fetch buffer and caps a restore batch at one file; purge holds every job permit. - Phase 1 is the PR series with acceptance criteria and GitHub numbers; migration numbers are reserved up front (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend and for DeleteBackup. Also adds the missing rows for designs 12 and 13 to docs/design/README.md and marks 15 as a draft. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 28, 2026
…hash, typed backup ARN Adds crates/core/src/backup_format, the on-disk format for off-host backups, built from #337's draft with the changes the design review asked for before v1 freezes. Data files: - DataFileEncoder<W: Write> hashes compressed bytes as they leave the compressor, so a writer streaming to a file or an upload holds the gzip window and one line, not the whole file. encode_data_file() remains as a convenience over it. encode_item_line is the one encoding of the {"Item": ...} line; the engine's ExportTableToPointInTime writer calls it. write_item refuses a line over MAX_LINE_BYTES (32 MiB) with the encoder still usable, so a backup cannot contain a line the decoder would refuse at restore. The worst case at the default 400 KB item limit is a list of number sets, 67.6x (a 27.7 MB line, a 1.21x margin); operator-raised item limits above about 496 KB can exceed the cap. The doc carries the measured table and a test pins the worst case. A write error poisons the encoder (every later call is Encode); try_finish keeps the sink so an upload can be aborted, finish returns it. Only a successful finish writes the gzip trailer: dropping an encoder in any other state (fresh, open, poisoned) takes the sink out first, so flate2's Drop cannot complete a valid file of the items so far in a sink the caller abandoned. A sink that accepts zero bytes is WriteZero, not a spin; Interrupted is retried on every sink write and flush, and on the decoder's source reads. is_finished and is_poisoned are distinct. GZIP_LEVEL = 6 is pinned, and a test pins the output length and sha256 of a fixed input, so two runs of one build produce the same bytes. - DataFileDecoder<R: Read> hashes the compressed bytes it consumes. The recorded compressed size and uncompressed size are enforced while reading (a source longer than its recorded compressed size fails with CompressedSizeMismatch after one byte past it); digests and the item count are checked at end of stream. Lines are capped at MAX_LINE_BYTES. Decoding one line peaks at about 76 bytes of heap per line byte for a list of empty strings, and at most about 128 for any shape measured (115 for the worst item, lists of one-member map chains; 127 for nested objects that are refused once built), whatever the nesting depth: about 4 GiB at the cap. DataFileDecoder's doc states the bound once. Any error is sticky (later next_item calls return Decode naming the earlier failure, never Ok(None)); the Iterator impl is fused on top so error-dropping consumers terminate. Line-level errors name the 1-based line in the file, and serde's column is labelled as within the line. Items are checked for JSON shape only; the restore validates them before writing. verified() keeps the slice path that checks digests before the first item. A line ends in \n and holds one object with the one member Item (an unknown member or a repeated top-level attribute is refused; nested maps keep JSON's last-wins); JSON whitespace, including a \r before the newline, is accepted as JSON accepts it. CompressedHasher is the incremental hasher a restore runs while spooling. AttributeValue deserializer (crates/core/src/types/attribute_value.rs, main's wire deserializer): the L and M arms now consume the serde_json::Value they own instead of cloning and re-deserializing it at each level. That copy had made an item PutItem accepts, 31 L wrappers around 409,470 empty strings, cost 8.6 GB and 12.7 s to decode, and the request path, which shares the deserializer, the same. Results and every API error text are unchanged (a differential run over 600k inputs; nested_error_text_is_unchanged pins six nested texts). data_file_nested_decode_memory, which is not ignored, bounds a 32-level item's decode at 120 bytes per line byte. Manifest: - Integrity is a sidecar, extenddb-manifest.sha256, in sha256sum format over the stored bytes (manifest_checksum / verify_manifest_checksum), replacing an in-document digest over a re-serialized struct that no other implementation could reproduce. The sidecar is accepted in one shape only, manifest_checksum's line "<64 lowercase hex> extenddb-manifest.json" with or without its final \n. A bare digest, which sha256sum -c refuses too, and anything else is InvalidManifest. The sidecar is written before the manifest, so the manifest's presence still marks a complete backup. - from_json_bytes gates on format_version (check_format_version: 1 to CURRENT_FORMAT_VERSION; 0 and later versions are refused with the existing text) before parsing the document. It then checks that the source table's item_count and table_size_bytes are not negative, then identity (backup_arn is in canonical form and parses; backup_creation_date_time is exactly the f64 its id's instant gives; backup_name is 3-255 of the table-name alphabet; source_table.table_name is the ARN's table; source_table.table_arn, read with core::arn::parse, is a table ARN in the same partition for the same account and table) and the data-file list (keys valid and unique, sha256 64 lowercase hex, md5 the standard base64 of 16 bytes, every file has items and non-zero sizes (entry errors are InvalidDataFileEntry, key errors InvalidDataFileKey), backup_size_bytes equals the sum without overflow, key_ordered manifests carry a key_range per file whose endpoints are key-only items of the schema's shape (S and B non-empty, N written in normalized form; the parser reads N endpoints as written, so 5.0 and 6E0 are refused) with first <= last and ranges ascending without overlap against every earlier file; unordered manifests carry none; an empty table is a manifest with no files). A key_range endpoint naming an attribute twice is refused at parse time. Non-overlapping ranges do not by themselves rule out a key in two files. KeyRangeCheck, which the restore runs over each file's items, refuses a first or last item that is not the endpoint, an item not strictly after the one before, an item past last and an empty file; with the ranges, that rules duplicates out. - Error messages carry a bounded slice of any untrusted field: at most 80 bytes of its escaped rendering, otherwise a 48-byte prefix and the total length. FormatError::Json holds a JsonError that bounds serde's text (which quotes the input) to 120 bytes in Display and Debug alike, and does not expose the inner error, as source() or otherwise; ChecksumMismatch prints its expected value with {:?}. - key_order.rs is the one definition of base-table key order: hash key by hash_key_text (S as is, N as held, B base64) compared bytewise, range key S/B bytewise and N numerically. N endpoints must be valid DynamoDB numbers written in normalized form; the parser refuses any other spelling, and items built in code are checked the same way. SnapshotDescriptor carries the three KIND_* constants the backends report, so the manifest and the storage layer share the strings. KeyOrder::resolve refuses multi-part keys, duplicate attribute definitions, a range key listed before the hash key, and a range key with the hash key's name. The storage layer's pk_to_text delegates to hash_key_text. The module doc gives the per-backend ORDER BY recipe, including COLLATE "C" on Postgres text keys. - Optional source_table members serialize as explicit null. global_secondary_indexes, local_secondary_indexes and vector_indexes are required members (vector_indexes may be null). OnDemandThroughput writes null for an absent member, as on the wire, and the module doc records it; vector_indexes is three-state (null / [] / list) and holds the creation shape; sse_description, item_order, per-file md5 and uncompressed_bytes; the per-file compressed size feeds the decoder's expectations. - Golden fixtures testdata/extenddb-manifest/v1.json, v1-minimal.json (every optional member absent, every list empty) and v1-unordered.json ("key_range":null files, a one-member OnDemandThroughput, a vector index with SearchSchema and Projection, an LSI with INCLUDE), each with its .sha256, compared byte for byte. They are permanent read tests, never regenerated once v1 is released; the #[ignore]d writer (EXTENDDB_WRITE_GOLDEN) writes v{CURRENT} fixtures. DynamoDB export manifest: ExportFileEntry.etag is Option<String>; dataFileS3Key documented as the full bucket key (<s3Prefix>/AWSDynamoDB/<export id>/data/<file>). Layout: the backup id and ARN implementation moves from extenddb_storage::util::backup_id into extenddb_core::backup_format::layout (format_backup_id, is_backup_id, backup_id_epoch_millis, backup_creation_date_time, check_backup_arn, parse_backup_arn, BackupArnParts, BackupArnError, MAX_BACKUP_ARN_LEN), unchanged in behaviour (verified by a reviewer's differential run over 5.9M inputs); the field split is core::arn::fields and the table-name alphabet is validation::validate_table_name_chars, so authorization, CreateTable and this parser share them. The engine's validate_name_member and validate_index_name still spell the alphabet themselves. format_backup_arn is the one backup-ARN formatter (backup_arn() in storage, BackupArn, and the server's canonical resource ARN call it) and trim_backup_arn the one whitespace rule (check_backup_arn and the two authorization sites call it). It trims both ends as main did; the doc notes that the service resolves a trailing newline but refuses a leading space or tab. extenddb_storage::util re-exports the rest so the engine and the four backends keep their import path, and keeps new_backup_id (needs rand), backup_arn, and is_backup_resource_arn (the authorization predicate). BackupArn is the owned form the format uses: check_backup_arn plus the two store-prefix rules (table name at most 255 bytes and not only dots), with the ARN string and the store prefix; a test enumerates both divergences and the id-shape grid. The 255-byte bound is narrower than the engine's max_table_name_length when an operator raises it; the layout doc says so. Other copies of ARN facts in the server and storage crates (is_backup_resource_arn's own split, a partition constant) predate this PR and are not changed here. Tests: 147 in the module (146 run, 1 ignored fixture writer), three integration binaries in crates/core/tests (data_file_line_bomb: a 256 MiB single-line bomb stops within twice the cap's memory; data_file_nested_decode_memory: a 32-level item decodes within 120 bytes of heap per line byte; data_file_decode_peak: the ignored RSS measurement), a storage-sqlite test binding encode_orderable_number's byte order to KeyOrder::compare over a fixed pseudo-random corpus of DynamoDB numbers, so a change to either side fails, and a row-key contract test in storage pinning pk_to_text's literal output (standard base64, "+/+/") so a change to hash_key_text cannot silently re-key B rows. The ddb-export fixture is constructed from the documented output format, not captured from the service. Coverage includes the streaming end-of-stream digest check, every expectation a DataFileEntry carries, the uncompressed and compressed size caps firing early, wrong item counts in both directions, try_finish failure poisoning, Interrupted on every sink and source call, invalid UTF-8 as Decode, an unterminated final line as an item, dropped encoders writing nothing, line numbers in data-line errors, range-key pairs that tell bytewise from length-first and from base64 order, B hash keys in base64 order, N range keys differing beyond 17 digits, overlap between later files, last-endpoint validation, the per-file key-range check's refusals and two non-overlapping files each holding every key, un-normalized and empty endpoints, resolve's key-order rules, required index members, negative source counts, the version gate's range, size-sum overflow, md5 and sidecar shapes, identity rules, explicit nulls, bounded messages at field lengths from 81 bytes to 2 MB, a bounded Debug of FormatError::Json, an escaped ChecksumMismatch, and nested deserializer error texts. 32 hand mutants of these rules, and 39 more of the round-5 fixes, were each killed by the suite. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 28, 2026
Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), five review rounds (the fifth on 2026-10-27, Lee's reviews among them) against the companion PRs #393, #394 and #395, and a reconciliation against main abf8082 (#384 in its final form, the Cassandra backend #338, the PostgreSQL GSI build #401). The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). Indexes are reported in whatever build state they had and only an index being deleted is dropped; a GSI without its data table is left out with the attribute definitions only it used. On PostgreSQL's shared layout the open is one connection and one REPEATABLE READ transaction; on the separate layout the catalog row is held FOR SHARE until the data snapshot pins, and an index that becomes ACTIVE between the definition read and the pin is reported CREATING. PostgreSQL bounds open snapshots at half the pool and refuses more with LimitExceeded, so open snapshots leave half the pool to requests; restores hold their own half-pool slots. It streams a table whose key columns predate COLLATE "C" unordered rather than sort it, and DeleteTable takes its asynchronous path while a reader holds any relation it would drop, so its DROP does not wait behind a snapshot or backup, with two stated gaps: a short GetItem, Query or Scan between the probe and the DROP, and on the shared layout a probe between an open's snapshot and its LOCK. On sqlite-memory the snapshot copies the table on the one connection inside the request and keeps a backup slot until its stream ends, so CreateBackup keeps working there once [backup.store] is set. MongoDB reads without the explicit snapshot level fail, and the document says so with the server's error. - The snapshot is opened in the CreateBackup request (stricter than the service's two-minute window), and the backup id is minted from the instant the open returned, just after the pin, so the backup holds every write acknowledged before the request was received, and backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (#384's seed and key), the row itself on MongoDB (born through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read. A job deletes a prefix, or a restore target, only after its FAILED write matched a row. The reconciler's duties are listed once, in section 4, and include the prefixes of DELETED rows, orphan restore targets, stale spools and, on Cassandra, restores_by_backup entries older than the stale window with no RUNNING job (that entry cannot share a lightweight transaction with the claim, so it is written first, carries its write time and is only a hint). Every duty that reads or writes a backup row acts only on rows of the configured store (store_kind, store_root and the store id); a CREATING or DELETING row of another store is reported and left alone. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty. backup sync is scoped to the configured store and checks the store id in extenddb-store.json, so an unmounted volume or a recreated bucket fails loudly; it never adopts a CREATING, DELETING, DELETED or FAILED prefix, and marks a row whose manifest head is gone DELETING only with --prune-missing. --move-store resolves the old root's CREATING and DELETING rows, compares every data file, refuses to carry a tombstone over a verified manifest and offers --forget-backup; --forget-store deletes the rows of a root. purge, --move-store, --forget-store and --prune-missing take a catalog maintenance claim that every job checks; max_concurrent_jobs is per instance. DeleteBackup and Restore of another store's row return a ValidationException that names no root. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. - Restore returns CREATING from the request (TableAlreadyExists or TableInUse on a name race, as #384 returns), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data on each backend. A key-ordered restore runs #394's per-file key-range check and an unordered one ends with a count of the target, which is Cassandra's only duplicate check. A PostgreSQL restore is resumed, or its target deleted, only after the stale window; the manifest and sidecar are spooled for a resume; a legacy restore with a dead owner is failed, not resumed. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as #384 and the service refuse it; the restore record is the table's child. Multi-part-key tables are restored (#384 refuses them). Table class is not restored, as on the service and #384; every override but VectorIndexOverride is applied, as #384 applies them. - The manifest self-checksum is replaced by a sidecar over the stored bytes, accepted in its sha256sum line form only; backup_arn is the backup's identity and source_table must agree with it; N key values are normalized before key text or key order is computed, and an un-normalized key_range endpoint is refused; the version gate accepts 1 to the current version. The line cap is 32 MiB, sized from the measured worst case, and the compressed size is enforced while reading; both are format rules in section 1. Section 7 states the decode memory from #394's DataFileDecoder doc: about 76 bytes per line byte for flat or nested lists and at most about 128 for any shape, about 4 GiB at the cap, after #394 stopped the deserializer copying nested values. - Store (#393): owner-only root and directories re-checked by every operation, an ancestor writable by group or others without the sticky bit refused, a root whose canonical path changed since open refused, the prefix resolved with the same no-follow walk as a key, listings that report only keys passing the key rule, multipart uploads aborted when a put is dropped and swept by delete_prefix (a lifecycle rule as backstop), validate by a one-key listing so a 403 keeps S3's code and then by probes of the grants, a GetObject of an absent key included, a 255-byte component cap and U+FFFE/U+FFFF refused by the key rule, settings under [backup.store]. Section 8 gains a transport rule (a non-loopback http:// endpoint, from the configuration or from the SDK's own endpoint sources, needs allow_insecure_endpoint; localhost over http needs force_path_style) and an IAM example with the list actions on the bucket, the KMS actions for an SSE-KMS bucket, the rule that a missing key reads as 404 only under s3:ListBucket, and a delete-marker lifecycle rule for versioned buckets. - Configuration: the default store root is per backend (beside the database file on file-backed SQLite); PostgreSQL, MongoDB and sqlite-memory refuse CreateBackup until [backup.store] is set, and keep serving legacy backups. Cassandra keeps creating in-catalog backups through the legacy create until PR 14, whether or not [backup.store] is set. The memory formula is in item text bytes, and PR 8a adds a byte budget to the PostgreSQL and SQLite snapshot streams; the PostgreSQL pool rule is max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10); the spool default is per backend. - Phase 1 is the PR series with acceptance criteria and GitHub numbers; migration numbers are reserved up front (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend and for DeleteBackup. The Phasing section names the companion heads the copied values come from. Also adds the missing rows for designs 12 and 13 to docs/design/README.md and marks 15 as a draft. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 28, 2026
…hash, typed backup ARN Adds crates/core/src/backup_format, the on-disk format for off-host backups, built from #337's draft with the changes the design review asked for before v1 freezes. Data files: - DataFileEncoder<W: Write> hashes compressed bytes as they leave the compressor, so a writer streaming to a file or an upload holds the gzip window and one line, not the whole file. encode_data_file() remains as a convenience over it. encode_item_line is the one encoding of the {"Item": ...} line; the engine's ExportTableToPointInTime writer calls it. write_item refuses a line over MAX_LINE_BYTES (32 MiB) with the encoder still usable, so a backup cannot contain a line the decoder would refuse at restore. The worst case at the default 400 KB item limit is a list of number sets, 67.6x (a 27.7 MB line, a 1.21x margin); operator-raised item limits above about 496 KB can exceed the cap. The doc carries the measured table and a test pins the worst case. A write error poisons the encoder (every later call is Encode); try_finish keeps the sink so an upload can be aborted, finish returns it. Only a successful finish writes the gzip trailer: dropping an encoder in any other state (fresh, open, poisoned) takes the sink out first, so flate2's Drop cannot complete a valid file of the items so far in a sink the caller abandoned. A sink that accepts zero bytes is WriteZero, not a spin; Interrupted is retried on every sink write and flush, and on the decoder's source reads. is_finished and is_poisoned are distinct. GZIP_LEVEL = 6 is pinned, and a test pins the output length and sha256 of a fixed input, so two runs of one build produce the same bytes. The writer emits one gzip member; the reader accepts concatenated members as one stream (a line may span members), which the reader contract states together with UTF-8 without BOM. - DataFileDecoder<R: Read> hashes the compressed bytes it consumes. The recorded compressed size and uncompressed size are enforced while reading (a source longer than its recorded compressed size fails with CompressedSizeMismatch after one byte past it); digests and the item count are checked at end of stream. Lines are capped at MAX_LINE_BYTES. Decoding one line peaks at about 76 bytes of heap per line byte for a list of empty strings, and at most about 128 for any shape measured (115 for the worst item, lists of one-member map chains; 127 for nested objects that are refused once built), whatever the nesting depth: about 4 GiB at the cap (measured without serde_json's preserve_order, the default binary; a build with the MongoDB backend enables it and measures about two thirds). DataFileDecoder's doc states the bound once. Any error is sticky (later next_item calls return Decode naming the earlier failure, never Ok(None)); the Iterator impl is fused on top so error-dropping consumers terminate. Line-level errors name the 1-based line in the file, and serde's column is labelled as within the line. Items are checked for JSON shape only; the restore validates them before writing. verified() keeps the slice path that checks digests before the first item. A line ends in \n and holds one object with the one member Item (an unknown member or a repeated top-level attribute is refused; a repeated type descriptor at the top of an attribute value is refused on this path and stays last-wins on the request path; nested maps keep JSON's last-wins); JSON whitespace, including a \r before the newline, is accepted as JSON accepts it. CompressedHasher is the incremental hasher a restore runs while spooling. AttributeValue deserializer (crates/core/src/types/attribute_value.rs, main's wire deserializer): the L and M arms now consume the serde_json::Value they own instead of cloning and re-deserializing it at each level. That copy had made an item PutItem accepts, 30 L wrappers around a list of 409,470 empty strings (31 containers, the maximum), cost 8.6 GB and 12.7 s to decode, and the request path, which shares the deserializer, the same. Results and every API error text are unchanged (a reviewer-run differential over 600k inputs, not committed; nested_error_text_is_unchanged pins six nested texts). data_file_nested_decode_memory, which is not ignored, bounds a 32-level item's decode at 120 bytes per line byte. Manifest: - Integrity is a sidecar, extenddb-manifest.sha256, in sha256sum format over the stored bytes (manifest_checksum / verify_manifest_checksum), replacing an in-document digest over a re-serialized struct that no other implementation could reproduce. The sidecar is accepted in one shape only, manifest_checksum's line "<64 lowercase hex> extenddb-manifest.json" with or without its final \n. A bare digest, which sha256sum -c refuses too, and anything else is InvalidManifest. The sidecar is written before the manifest, so the manifest's presence still marks a complete backup. - from_json_bytes requires a JSON document whose format_version is a u32, gates on it (check_format_version: 1 to CURRENT_FORMAT_VERSION; 0 and later versions are refused with a message naming the range this build reads) before any other member is interpreted, parses the document, then runs ExtenddbManifest::validate: the version gate, validate_source_table (item_count and table_size_bytes not negative; the key schema has at least one HASH, HASH before RANGE, each key attribute named once and defined exactly once, on every manifest; the count of HASH and RANGE parts is left to the restore's CreateTable validation, and a multi-part key is backed up unordered), validate_identity (backup_arn is in canonical form and parses; backup_creation_date_time is exactly the f64 its id's instant gives; backup_name is 3-255 of the table-name alphabet; source_table.table_name is the ARN's table; source_table.table_arn, read with core::arn::parse, is a table ARN in the same partition for the same account and table) and validate_data_files (keys valid and unique, sha256 64 lowercase hex, md5 the standard base64 of 16 bytes, every file has items and non-zero sizes (entry errors are InvalidDataFileEntry, key errors InvalidDataFileKey), backup_size_bytes equals the sum without overflow, key_ordered manifests carry a key_range per file whose endpoints are key-only items of the schema's shape (S and B non-empty, N written in normalized form; the parser reads N endpoints as written, so 5.0 and 6E0 are refused) with first <= last and ranges ascending without overlap against every earlier file; unordered manifests carry none; an empty table is a manifest with no files). to_json_bytes runs validate too, so a writer cannot produce bytes the reader refuses. The document is parsed twice (once for the version, once for the members); the doc accepts the cost. A key_range endpoint naming an attribute twice is refused at parse time. Non-overlapping ranges do not by themselves rule out a key in two files. KeyRangeCheck, which the restore runs over each file's items, refuses a first or last item that is not the endpoint, an item not strictly after the one before, an item past last and an empty file; with the ranges, that rules duplicates out. - Error messages carry a bounded slice of any untrusted field: at most 80 bytes of its escaped rendering, otherwise a 48-byte prefix and the total length. FormatError::Json holds a JsonError that bounds serde's text (which quotes the input) to 120 bytes in Display and Debug alike, and does not expose the inner error, as source() or otherwise; ChecksumMismatch prints its expected value with {:?}. - key_order.rs is the one definition of base-table key order: hash key by hash_key_text (S as is, N as held, B standard padded base64) compared as UTF-8 bytes, range key S as UTF-8 bytes, B as raw bytes and N by exact decimal value; the module doc states the N normalization rules, the base64 alphabet and the string encoding another implementation must reproduce, and a test pins its examples. N endpoints must be valid DynamoDB numbers written in normalized form; the parser refuses any other spelling, and items built in code are checked the same way. SnapshotDescriptor carries the three KIND_* constants the backends report, so the manifest and the storage layer share the strings; kind is a free string the reader does not validate and a restore must not branch on. KeyOrder::resolve refuses multi-part keys, duplicate attribute definitions, a range key listed before the hash key, and a range key with the hash key's name. The storage layer's pk_to_text delegates to hash_key_text. The module doc gives the per-backend ORDER BY recipe, including COLLATE "C" on Postgres text keys. - Optional source_table members serialize as explicit null. global_secondary_indexes, local_secondary_indexes, vector_indexes, stream_specification and time_to_live are required members (the last three may be null); stream_specification and time_to_live record what DescribeTable and DescribeTimeToLive reported, for DescribeBackup's SourceTableFeatureDetails, and are not restored. OnDemandThroughput writes null for an absent member, as on the wire, and the module doc records it; vector_indexes is three-state (null / [] / list) and holds the creation shape; sse_description, item_order, per-file md5 and uncompressed_bytes; the per-file compressed size feeds the decoder's expectations. - Golden fixtures testdata/extenddb-manifest/v1.json (every optional member set, a stream specification and a TTL description included), v1-minimal.json (every optional member absent, every list empty) and v1-unordered.json ("key_range":null files, a one-member OnDemandThroughput, a vector index with SearchSchema and Projection, an LSI with INCLUDE, a disabled TTL without AttributeName and a null stream specification), each with its .sha256, compared byte for byte, and a non-ignored test binds the writer's three constructors to the checked-in bytes. They are permanent read tests: the #[ignore]d writer (EXTENDDB_WRITE_GOLDEN) writes v{CURRENT} fixtures with create_new and refuses to overwrite an existing file. Any change to the bytes written for the fixtures' inputs is a version bump; a new member gets serde(default) and v{N} fixtures beside the v1 ones. Unknown members are tolerated at every level. DynamoDB export manifests: ExportFileEntry.etag is Option<String>; dataFileS3Key is the full store key, <prefix>/data/<file> (the AWS documentation's two examples differ on the prefix; the module doc says so and names what ExtendDB writes), built by ExportFileEntry::from_data_file from the manifest's DataFileEntry so the two manifests carry one md5 and one item_count; DataFileEntry::from_checksums builds the entry from the encoder's result; ExportSummaryManifest::for_backup fills manifest-summary.json from the manifest (exportArn = the backup ARN, the three instants = the snapshot instant in RFC 3339 UTC with milliseconds, s3SseAlgorithm null, itemCount the sum; the module doc has the table), and s3SseAlgorithm is Option<String>. Layout: the backup id and ARN implementation moves from extenddb_storage::util::backup_id into extenddb_core::backup_format::layout (format_backup_id, is_backup_id, backup_id_epoch_millis, backup_creation_date_time, check_backup_arn, parse_backup_arn, BackupArnParts, BackupArnError, MAX_BACKUP_ARN_LEN), unchanged in behaviour (verified by a reviewer's differential run over 5.9M inputs); the field split is core::arn::fields and the table-name alphabet is validation::validate_table_name_chars, so authorization, CreateTable and this parser share them. The engine's validate_name_member and validate_index_name still spell the alphabet themselves. format_backup_arn is the one backup-ARN formatter (backup_arn() in storage, BackupArn, and the server's canonical resource ARN call it) and trim_backup_arn the one whitespace rule (check_backup_arn and the two authorization sites call it). It trims both ends as main did; the doc notes that the service resolves a trailing newline but refuses a leading space or tab. extenddb_storage::util re-exports the rest so the engine and the four backends keep their import path, and keeps new_backup_id (needs rand), backup_arn, and is_backup_resource_arn (the authorization predicate). BackupArn is the owned form the format uses: check_backup_arn plus the two store-prefix rules (table name at most 255 bytes and not only dots), with the ARN string and the store prefix; a test enumerates both divergences and the id-shape grid. The 255-byte bound is narrower than the engine's max_table_name_length when an operator raises it; the layout doc says so. Other copies of ARN facts in the server and storage crates (is_backup_resource_arn's own split, a partition constant) predate this PR and are not changed here. Tests: 157 in the module (156 run, 1 ignored fixture writer), three integration binaries in crates/core/tests (data_file_line_bomb: a 256 MiB single-line bomb stops within twice the cap's memory; data_file_nested_decode_memory: a 32-level item decodes within 120 bytes of heap per line byte; data_file_decode_peak: the ignored RSS measurement), a storage-sqlite test binding encode_orderable_number's byte order to KeyOrder::compare over a fixed pseudo-random corpus of DynamoDB numbers, so a change to either side fails, and a row-key contract test in storage pinning pk_to_text's literal output (standard base64, "+/+/") so a change to hash_key_text cannot silently re-key B rows. The ddb-export fixture is constructed from the documented output format, not captured from the service. Coverage includes the streaming end-of-stream digest check, every expectation a DataFileEntry carries, the uncompressed and compressed size caps firing early, wrong item counts in both directions, try_finish failure poisoning, Interrupted on every sink and source call, invalid UTF-8 as Decode, an unterminated final line as an item, dropped encoders writing nothing, line numbers in data-line errors, range-key pairs that tell bytewise from length-first and from base64 order, B hash keys in base64 order, N range keys differing beyond 17 digits, overlap between later files, last-endpoint validation, the per-file key-range check's refusals and two non-overlapping files each holding every key, un-normalized and empty endpoints, resolve's key-order rules, required index members, negative source counts, the version gate's range and the gate's message naming the range 1 to CURRENT, unknown members injected at every level of two fixtures, the key-schema shape rules on an unordered manifest, to_json_bytes refusing what from_json_bytes refuses, the two manifest constructors agreeing with the encoder and the manifest entry, the summary manifest's documented values, the key-order doc's normalization examples, size-sum overflow, md5 and sidecar shapes, identity rules, explicit nulls, bounded messages at field lengths from 81 bytes to 2 MB, a bounded Debug of FormatError::Json, an escaped ChecksumMismatch, and nested deserializer error texts. 32 hand mutants of these rules, and 39 more of the round-5 fixes, were each killed by the suite; 20 of the round-6 fixes were killed and one was equivalent (a dead branch in number.rs, replaced by three others, killed). Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 28, 2026
Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), six review rounds (the sixth on 2026-10-28, Lee's reviews among them) against the companion PRs #393, #394 and #395, and a reconciliation against main abf8082 (#384 in its final form, the Cassandra backend #338, the PostgreSQL GSI build #401). The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). Indexes are reported in whatever build state they had and only an index being deleted is dropped; a GSI without its data table is left out with the attribute definitions only it used. On PostgreSQL's shared layout the open is one connection and one REPEATABLE READ transaction; on the separate layout the catalog row is held FOR SHARE until the data snapshot pins, and an index that becomes ACTIVE between the definition read and the pin is reported CREATING. PostgreSQL bounds open snapshots at half the pool and refuses more with LimitExceeded, so open snapshots leave half the pool to requests; from PR 6 on max_concurrent_jobs (default 4) bounds backups and restores together under that permit, and #384's restore slots go. It streams a table whose key columns predate COLLATE "C" unordered rather than sort it, and DeleteTable takes its asynchronous path while a reader holds any relation it would drop, so its DROP does not wait behind a snapshot or backup, with two stated gaps: a short GetItem, Query or Scan between the probe and the DROP, and on the shared layout a probe between an open's snapshot and its LOCK; the control-plane poller stays active while a drop is deferred, so the drop follows any reader's release within about a second. On sqlite-memory the snapshot copies the table on the one connection inside the request and keeps a backup slot until its stream ends, so CreateBackup keeps working there once [backup.store] is set. MongoDB reads without the explicit snapshot level fail, and the document says so with the server's error. - The snapshot is opened in the CreateBackup request (stricter than the service's two-minute window), and the backup id is minted from the instant the open returned, just after the pin, so the backup holds every write acknowledged before the request was received, and backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (#384's seed and key), the row itself on MongoDB (born through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read. A job deletes a prefix, or a restore target, only after its FAILED write matched a row. The reconciler's duties are listed once, in section 4, and include the prefixes of DELETED rows, orphan restore targets, stale spools and, on Cassandra, restores_by_backup entries older than the stale window with no RUNNING job (that entry cannot share a lightweight transaction with the claim, so it is written first, carries its write time and is only a hint). Every duty that reads or writes a backup row acts only on rows of the configured store (store_kind, store_root and the store id); a CREATING or DELETING row of another store is reported and left alone. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty. backup sync is scoped to the configured store and checks the store id in extenddb-store.json, so an unmounted volume or a recreated bucket fails loudly; it never adopts a CREATING, DELETING, DELETED or FAILED prefix, and marks a row whose manifest head is gone DELETING only with --prune-missing. --move-store resolves the old root's CREATING and DELETING rows, compares every data file, refuses to carry a tombstone over a verified manifest and offers --forget-backup; --forget-store deletes the rows of a root. purge, --move-store, --forget-store and --prune-missing take a catalog maintenance claim that every job checks; max_concurrent_jobs is per instance. DeleteBackup and Restore of another store's row return a ValidationException that names no root. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. - Restore returns CREATING from the request (TableAlreadyExists or TableInUse on a name race, as #384 returns), spools and verifies every data file before writing any item, and records progress atomically with the data on PostgreSQL and SQLite and by idempotent re-application of the one unrecorded file or batch on MongoDB and Cassandra; on MongoDB the job writes DONE and the backfill flag in one transaction and the worker's ACTIVE flip is conditioned on the flag, not on the job's generation. A key-ordered restore runs #394's per-file key-range check and an unordered one ends with a count of the target, which is Cassandra's only duplicate check. A PostgreSQL restore is resumed, or its target deleted, only after the stale window; the manifest and sidecar are spooled for a resume; a legacy restore with a dead owner is failed, not resumed. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as #384 and the service refuse it; the restore record is the table's child. Multi-part-key tables are restored (#384 refuses them; bulk_load writes through the backend's own item path, so the table is laid out as a natively created one). Table class is not restored, as on the service and #384; every override but VectorIndexOverride is applied, as #384 applies them. - The engine binds a BackupCatalog trait (PR 6) with a JobClaim for the per-backend claim, heartbeat and conditional status writes; new rows live in backup_index (a new table or collection; Cassandra reuses its tables with store_kind set and adds backup_index_by_status); MongoDB's catalog gets the versioned migration step it lacks; PR 6's migration backfills in-flight #384 restores so the first reconciler pass fails them; every pre-PR-6 server stops before migrate. Vector tables are backed up and restored on the off-host path (three #384 tests change in PR 8a/9a). source_table carries stream_specification and time_to_live for DescribeBackup. Both requests take the job permit last, after every refusal that needs no store access. The format's evolution rule is a version bump for any byte change, with v{N} fixtures beside v1. - The manifest self-checksum is replaced by a sidecar over the stored bytes, accepted in its sha256sum line form only; backup_arn is the backup's identity and source_table must agree with it; N key values are normalized before key text or key order is computed, and an un-normalized key_range endpoint is refused; the version gate accepts 1 to the current version. The line cap is 32 MiB, sized from the measured worst case, and the compressed size is enforced while reading; both are format rules in section 1. Section 7 cites the decode memory from #394's DataFileDecoder doc (76 and 128 bytes per line byte as of #394, measured without preserve_order; about 4 GiB at the cap). - Store (#393): owner-only root and directories re-checked by every operation, an ancestor writable by group or others without the sticky bit refused, a root whose canonical path changed since open refused, the prefix resolved with the same no-follow walk as a key, listings that report only keys passing the key rule, multipart uploads aborted when a put is dropped and swept by delete_prefix (a lifecycle rule as backstop), validate by a one-key listing so a 403 keeps S3's code and then by probes of the grants, a GetObject of an absent key included, a 255-byte component cap and U+FFFE/U+FFFF refused by the key rule, settings under [backup.store]. Section 8 gains a transport rule (a non-loopback http:// endpoint, from the configuration or from the SDK's own endpoint sources, needs allow_insecure_endpoint; localhost over http needs force_path_style) and an IAM example with the list actions on the bucket, the KMS actions for an SSE-KMS bucket, the rule that a missing key reads as 404 only under s3:ListBucket, and a delete-marker lifecycle rule for versioned buckets. - Configuration: the default store root is per backend (beside the database file on file-backed SQLite, where a defaulted root that fails validation is a warning, not a startup error); without [backup.store], CreateBackup on every backend keeps the legacy in-catalog create with a deprecation warning until PR 14 removes it (Cassandra always, having no SnapshotEngine), and kind = "disabled" is the opt-out. A store transport failure at startup degrades the store component instead of stopping the server. The memory formula is in item text bytes, and PR 8a adds the byte budget SNAPSHOT_FETCH_BYTES (64 MiB) to the PostgreSQL and SQLite snapshot streams; the PostgreSQL pool rule is max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10), 18 at the default 4; the spool default is per backend and spool_host is host plus spool path. - Phase 1 is the PR series with acceptance criteria and GitHub numbers; migration numbers are reserved up front (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend and for DeleteBackup. The Phasing section refers to the companion PRs' heads on the revision date rather than naming commits. Also adds the missing rows for designs 12 and 13 to docs/design/README.md and marks 15 as a draft. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 29, 2026
…hash, typed backup ARN Adds crates/core/src/backup_format, the on-disk format for off-host backups, built from #337's draft with the changes the design review asked for before v1 freezes. Data files: - DataFileEncoder<W: Write> hashes compressed bytes as they leave the compressor, so a writer streaming to a file or an upload holds the gzip window and one line, not the whole file. encode_data_file() remains as a convenience over it. encode_item_line_into is the one encoding of the {"Item": ...} line, into a buffer the caller keeps; encode_item_line, which the engine's ExportTableToPointInTime writer calls, and write_item, which reuses one buffer per encoder, both call it. write_item refuses a line over MAX_LINE_BYTES (32 MiB) with the encoder still usable, so a backup cannot contain a line the decoder would refuse at restore. The worst case at the default 400 KB item limit is a list of number sets, 67.6x (a 27.7 MB line, a 1.21x margin); operator-raised item limits above about 496 KB can exceed the cap. The doc carries the measured table and a test pins the worst case. A write error poisons the encoder (every later call is Encode); try_finish keeps the sink so an upload can be aborted, finish returns it. Only a successful finish writes the gzip trailer: dropping an encoder in any other state (fresh, open, poisoned) takes the sink out first, so flate2's Drop cannot complete a valid file of the items so far in a sink the caller abandoned. A sink that accepts zero bytes is WriteZero, not a spin; Interrupted is retried on every sink write and flush, and on the decoder's source reads. is_finished and is_poisoned are distinct. GZIP_LEVEL = 6 is pinned, and a test pins the output length and sha256 of a fixed input, so two runs of one build produce the same bytes. The writer emits one gzip member; the reader accepts concatenated members as one stream (a line may span members), which the reader contract states together with UTF-8 without BOM. - DataFileDecoder<R: Read> hashes the compressed bytes it consumes. The recorded compressed size and uncompressed size are enforced while reading (a source longer than its recorded compressed size fails with CompressedSizeMismatch after one byte past it); digests and the item count are checked at end of stream. Lines are capped at MAX_LINE_BYTES. Decoding one line peaks at about 76 bytes of heap per line byte for a list of empty strings, and at most about 128 for any shape measured (115 for the worst item, lists of one-member map chains; 127 for nested objects that are refused once built), whatever the nesting depth: about 4 GiB at the cap (measured without serde_json's preserve_order, the default binary; a build with the MongoDB backend enables it and measures about two thirds). DataFileDecoder's doc states the bound once. Any error is sticky (later next_item calls return Decode naming the earlier failure, never Ok(None)); the Iterator impl is fused on top so error-dropping consumers terminate. Line-level errors name the 1-based line in the file, and serde's column is labelled as within the line when serde's line is 1; a type descriptor or name ending in ` at line 3 column 4` sets serde's position, and such a message is shown as serde rendered it, with no column of its own. Items are checked for JSON shape only; the decoder does not apply the target's item limits, which the restore applies to each item before writing it. verified() keeps the slice path that checks digests before the first item, and then decodes without hashing again. A line ends in \n, except that the last line's may be absent (that line is an item, counted in uncompressed_bytes without a newline), and holds one object with the one member Item (an unknown member or a repeated top-level attribute is refused; a repeated type descriptor at the top of an attribute value is refused on this path and stays last-wins on the request path; nested maps keep JSON's last-wins); JSON whitespace, including a \r before the newline, is accepted as JSON accepts it. CompressedHasher is the incremental hasher a restore runs while spooling; DataFileEntry::decoder_prehashed and ExtenddbManifest::reader_prehashed then read the spooled copy without hashing it again (about 0.43 µs per 318-byte item, 9% of decoding) and keep every size, count and line check. AttributeValue deserializer (crates/core/src/types/attribute_value.rs, main's wire deserializer): the L and M arms now consume the serde_json::Value they own instead of cloning and re-deserializing it at each level. That copy had made an item PutItem accepts, 30 L wrappers around a list of 409,470 empty strings (31 containers, the maximum), cost 8.6 GB and 12.7 s to decode, and the request path, which shares the deserializer, the same. Results and every API error text are unchanged: a nested error is rendered once per level through serde_json::Error::custom, as main's per-level from_value rendered it, so a message ending in ` at line 0 column 0` loses one such suffix per level as before (a reviewer-run differential over 600k inputs, not committed; nested_error_text_is_unchanged pins thirteen nested texts, seven of that class at one to three levels with main's output, one of them with a suffix more than the levels strip). data_file_nested_decode_memory, which is not ignored, bounds a 32-level item's decode at 120 bytes per line byte. validation/number.rs (main's code, which every format parser and the request path reach) does its exponent arithmetic in i64. An exponent at either end of i32 combined with the digits no longer overflows: a debug build panicked on 10E2147483647, 49E2147483647 and .00E-2147483647, and a release build reported overflow as underflow and the reverse. An invalid N key's InvalidKey message carries the number validation's reason. Manifest: - Integrity is a sidecar, extenddb-manifest.sha256, in sha256sum format over the stored bytes (written by BackupDocuments::build, checked by verify_manifest_checksum), replacing an in-document digest over a re-serialized struct that no other implementation could reproduce. The sidecar is accepted in one shape only, the line the writer stores, "<64 lowercase hex> extenddb-manifest.json", with or without its final \n. A bare digest, which sha256sum -c refuses too, and anything else is InvalidManifest. The sidecar is written before the manifest, so the manifest's presence still marks a complete backup. - from_json_bytes requires a JSON document whose format_version is a u32, gates on it (check_format_version: 1 to CURRENT_FORMAT_VERSION; 0 and later versions are refused with a message naming the range this build reads) before any other member is interpreted, parses the document, then runs ExtenddbManifest::validate: the version gate, validate_source_table (item_count and table_size_bytes not negative; table_creation_date_time finite, since JSON has no NaN or infinity and serde_json writes them as null; the key schema has at least one HASH, HASH before RANGE, each key attribute named once and defined exactly once, on every manifest; the count of HASH and RANGE parts is left to the restore's CreateTable validation, and a multi-part key is backed up unordered), validate_identity (backup_arn is in canonical form and parses; backup_creation_date_time is exactly the f64 its id's instant gives; backup_name is 3-255 of the table-name alphabet; source_table.table_name is the ARN's table; source_table.table_arn, read with core::arn::parse, is a table ARN in the same partition for the same account and table) and validate_data_files (keys valid and unique, sha256 64 lowercase hex, md5 the standard base64 of 16 bytes, every file has items and non-zero sizes (entry errors are InvalidDataFileEntry, key errors InvalidDataFileKey), backup_size_bytes equals the sum without overflow, key_ordered manifests carry a key_range per file whose endpoints are key-only items of the schema's shape (S and B non-empty, N written in normalized form; the parser reads N endpoints as written, so 5.0 and 6E0 are refused) with first <= last and ranges ascending without overlap against every earlier file; unordered manifests carry none; an empty table is a manifest with no files). BackupDocuments::build runs validate too, so a writer cannot produce bytes the reader refuses. The document is parsed twice (once for the version, under 10% of the cost, once for the members); the doc states the cost per data file: about 6 to 12 µs and 1.6 KB retained per file with short keys. A key_range endpoint naming an attribute twice, or whose value repeats its type descriptor ({"S":"0","S":"A"}) or names two, is refused at parse time, as a data file line of that shape is. Non-overlapping ranges do not by themselves rule out a key in two files. ExtenddbManifest::reader (and reader_prehashed) runs KeyRangeCheck over each file's items when the manifest is key-ordered, without the caller constructing it, and its finish after the decoder's end-of-stream checks. The check refuses a first or last item that is not the endpoint, an item not strictly after the one before, an item past last, an item whose key KeyOrder::endpoint_key refuses, and an empty file; with the ranges, that rules duplicates out. KeyRangeCheck is a KeyRangeTracker plus the recorded range. The writer runs the same tracker over the items it encodes, so the endpoint check and the strict-ascent rule exist once, and its finish gives the file's KeyRange. The tracker checks each key in place and keeps the first and latest keys, with an N range key parsed once; the check parses the recorded endpoints once. Per 318-byte item the check went from 1.07 µs and 11.5 allocations to 0.66 µs and 5 (one partition: 1.64 µs and 19.3 to 0.69 µs and 5.3). - Error messages carry a bounded slice of any untrusted field: at most 80 bytes of its escaped rendering, otherwise a 48-byte prefix and the total length. FormatError::Json holds a JsonError that bounds serde's text (which quotes the input) to 120 bytes in Display and Debug alike, and does not expose the inner error, as source() or otherwise; ChecksumMismatch prints its expected value with {:?}. - key_order.rs is the one definition of base-table key order: hash key by hash_key_text (S as is, N as held, B standard padded base64) compared as UTF-8 bytes, range key S as UTF-8 bytes, B as raw bytes and N by exact decimal value; the module doc states the N normalization rules (the exponent a signed 32-bit integer), the base64 alphabet and the string encoding another implementation must reproduce, and a test pins its examples. It states which writers store an invalid N key as written (BatchWriteItem and TransactWriteItems, #618; UpdateItem on an absent item and ImportTable run no number check on the key) and the rule for a key-ordered writer that meets one: the backup is refused with an error naming the key, through KeyRangeTracker::push, the one call per item, which checks the key as KeyOrder::endpoint_key does. N endpoints must be valid DynamoDB numbers written in normalized form; the parser refuses any other spelling, and items built in code are checked the same way. SnapshotDescriptor carries the three KIND_* constants the backends report, so the manifest and the storage layer share the strings; kind is a free string the reader does not validate and a restore must not branch on. KeyOrder::resolve refuses multi-part keys, duplicate attribute definitions, a range key listed before the hash key, and a range key with the hash key's name. The storage layer's pk_to_text delegates to hash_key_text. The module doc gives the per-backend ORDER BY recipe, including COLLATE "C" on Postgres text keys. - Optional source_table members serialize as explicit null. global_secondary_indexes, local_secondary_indexes, vector_indexes, stream_specification and time_to_live are required members (the last three may be null); stream_specification and time_to_live record what DescribeTable and DescribeTimeToLive reported, for DescribeBackup's SourceTableFeatureDetails, and are not restored. OnDemandThroughput writes null for an absent member, as on the wire, and the module doc records it; vector_indexes is three-state (null / [] / list) and holds the creation shape; sse_description, item_order, per-file md5 and uncompressed_bytes; the per-file compressed size feeds the decoder's expectations. - Golden fixtures testdata/extenddb-manifest/v1.json (every optional member set, a stream specification and a TTL description included), v1-minimal.json (every optional member absent, every list empty) and v1-unordered.json ("key_range":null files and the absent state of every nested optional member v1.json sets: an OnDemandThroughput with both limits null, an SseDescription without SSEType or KMSMasterKeyArn, a GSI without ProvisionedThroughput, a disabled stream specification without StreamViewType, a disabled TTL without AttributeName; and a vector index with SearchSchema and Projection, an LSI with INCLUDE), each with its .sha256, compared byte for byte, and a non-ignored test binds the writer's three constructors to the checked-in bytes. They are permanent read tests: the #[ignore]d writer (EXTENDDB_WRITE_GOLDEN) writes v{CURRENT} fixtures with create_new and refuses to overwrite an existing file. Any change to the bytes written for the fixtures' inputs is a version bump; a new member gets serde(default) and v{N} fixtures beside the v1 ones. A new member omitted when absent leaves the bytes unchanged, so nested_types_have_the_pinned_members pins the members every manifest type reads and writes, and fails it until the list changes. A new or renamed enum value that no fixture uses changes no byte either, so enum_values_are_pinned pins the values of every enum in the manifest: the derived ones through serde's variant list, and KeyType, ScalarAttributeType and DistanceFunction, whose readers are hand-written, through tables filled from a match with no wildcard. Unknown members are tolerated at every level. DynamoDB export manifests: ExportFileEntry.etag and s3SseAlgorithm are Option<String>; dataFileS3Key is the full store key, <prefix>/data/<file> (the service writes <s3Prefix>/AWSDynamoDB/<export id>/data/<file> for a full export to a prefix, measured; the documentation's full-export example omits the prefix; the module doc states both and what ExtendDB writes). dataFileS3Key and manifestFilesS3Key are built by one function, so an empty prefix gives data/<file> and manifest-files.json, without a leading /. Every trailing / of the prefix is dropped once, so a// gives a/data/<file> and a/manifest-files.json, as a does. DataFileEntry::from_checksums builds each entry from the encoder's result. BackupDocuments::build(header, data_files, store_key_prefix, bucket, etags) assembles the manifest, its sidecar, manifest-files.json and manifest-summary.json from one input. It derives backup_size_bytes, backup_creation_date_time and the key prefix (the store's key prefix, then BackupArn::store_prefix) once and runs validate once. It builds each manifest-files.json line from the manifest's DataFileEntry, so the two manifests carry one md5 and one item_count, and the summary from the manifest (exportArn = the backup ARN, the three instants = the snapshot instant in RFC 3339 UTC with milliseconds, s3SseAlgorithm null, itemCount the sum; the module doc has the table). Its doc gives the write order, which documents() returns. for_backup, from_data_file, write_files_manifest, to_json_bytes and manifest_checksum are crate-private. Layout: the backup id and ARN implementation moves from extenddb_storage::util::backup_id into extenddb_core::backup_format::layout (format_backup_id, is_backup_id, backup_id_epoch_millis, backup_creation_date_time, check_backup_arn, parse_backup_arn, BackupArnParts, BackupArnError, MAX_BACKUP_ARN_LEN), unchanged in behaviour (verified by a reviewer's differential run over 5.9M inputs); the field split is core::arn::fields and the table-name alphabet is validation::validate_table_name_chars, so authorization, CreateTable and this parser share them. The engine's validate_name_member and validate_index_name still spell the alphabet themselves. format_backup_arn is the one backup-ARN formatter (backup_arn() in storage, BackupArn, and the server's canonical resource ARN call it) and trim_backup_arn the one whitespace rule (check_backup_arn and the two authorization sites call it). It trims both ends as main did; the doc notes that the service resolves a trailing newline but refuses a leading space or tab. extenddb_storage::util re-exports the rest so the engine and the four backends keep their import path, and keeps new_backup_id (needs rand), backup_arn, and is_backup_resource_arn (the authorization predicate). BackupArn is the owned form the format uses: check_backup_arn plus the two store-prefix rules (table name at most 255 bytes and not only dots), with the ARN string and the store prefix; a test enumerates both divergences and the id-shape grid. The 255-byte bound is narrower than the engine's max_table_name_length when an operator raises it; the layout doc says so. Other copies of ARN facts in the server and storage crates (is_backup_resource_arn's own split, a partition constant) predate this PR and are not changed here. Tests: 172 in the module (171 run, 1 ignored fixture writer), three integration binaries in crates/core/tests (data_file_line_bomb: a 256 MiB single-line bomb stops within twice the cap's memory; data_file_nested_decode_memory: a 32-level item decodes within 120 bytes of heap per line byte; data_file_decode_peak: the ignored RSS measurement), a storage-sqlite test binding encode_orderable_number's byte order to KeyOrder::compare over a fixed pseudo-random corpus of DynamoDB numbers, so a change to either side fails, and a row-key contract test in storage pinning pk_to_text's literal output (standard base64, "+/+/") so a change to hash_key_text cannot silently re-key B rows. The ddb-export fixture is constructed from the documented output format, not captured from the service. Coverage includes the streaming end-of-stream digest check, every expectation a DataFileEntry carries, the uncompressed and compressed size caps firing early, wrong item counts in both directions, try_finish failure poisoning, Interrupted on every sink and source call, invalid UTF-8 as Decode, an unterminated final line as an item, dropped encoders writing nothing, line numbers in data-line errors, range-key pairs that tell bytewise from length-first and from base64 order, B hash keys in base64 order, N range keys differing beyond 17 digits, overlap between later files, last-endpoint validation, the per-file key-range check's refusals (a repeated key and an invalid stored N key among them) and two non-overlapping files each holding every key, through ExtenddbManifest::reader as well, the writer's tracker refusing an out-of-order or repeated key and agreeing with KeyOrder::compare for S, N and B keys, the prehashed decoder keeping every check but the digest, BackupDocuments' four documents agreeing on one prefix, the pinned enum values, N exponents at both ends of i32, a forged position in a data-line error, un-normalized and empty endpoints, resolve's key-order rules, required index members, negative source counts, the version gate's range and the gate's message naming the range 1 to CURRENT, unknown members injected at every level of two fixtures, the key-schema shape rules on an unordered manifest, to_json_bytes refusing what from_json_bytes refuses (non-finite timestamps included), endpoints repeating a type descriptor, the pinned member list of every manifest type, the two manifest constructors agreeing with the encoder and the manifest entry, the summary manifest's documented values, the key-order doc's normalization examples, size-sum overflow, md5 and sidecar shapes, identity rules, explicit nulls, bounded messages at field lengths from 81 bytes to 2 MB, a bounded Debug of FormatError::Json, an escaped ChecksumMismatch, and nested deserializer error texts. 32 hand mutants of these rules, and 39 more of the round-5 fixes, were each killed by the suite; 20 of the round-6 fixes were killed and one was equivalent (a dead branch in number.rs, replaced by three others, killed); 13 of the fixes for the maintainer's review of 2026-10-28 were killed, none equivalent; 15 of the round-7 fixes were killed, none equivalent. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 29, 2026
Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), seven review rounds and the maintainer's reviews (on 2026-10-28 the sixth round, the maintainer's review and the seventh round) against the companion PRs #393, #394 and #395, and a reconciliation against main abf8082 (#384 in its final form, the Cassandra backend #338, the PostgreSQL GSI build #401). The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). Indexes are reported in whatever build state they had and only an index being deleted is dropped; a GSI without its data table is left out with the attribute definitions only it used. On PostgreSQL's shared layout the open is one connection and one REPEATABLE READ transaction; on the separate layout the catalog row is held FOR SHARE until the data snapshot pins, and an index that becomes ACTIVE between the definition read and the pin is reported CREATING. PostgreSQL bounds open snapshots at half the pool and refuses more with LimitExceeded, so open snapshots leave half the pool to requests; from PR 6 on max_concurrent_jobs (default 4) bounds backups and restores together under that permit, and #384's restore slots go. It streams a table whose key columns predate COLLATE "C" unordered rather than sort it, and on both layouts DeleteTable commits the catalog removal only after its drops succeed, base table first, under one 1 s budget, taking its asynchronous path with the row kept when its probe finds a reader or a drop fails in a lock wait (lock timeout, deadlock victim or statement timeout), so no DROP waits behind a reader with the row gone; on the separate layout a task the request awaits commits the catalog and then the data transaction; the control-plane poller probes held tables and retries each about once a second, at once on a snapshot's or an in-catalog backup's release, a drop DeleteTable deferred is due at once, and DeleteTable of a DELETING table returns its description on PostgreSQL and SQLite. On sqlite-memory the snapshot copies the table on the one connection inside the request and keeps a backup slot until its stream ends, so CreateBackup keeps working there once [backup.store] is set. MongoDB reads without the explicit snapshot level fail, and the document says so with the server's error. - The snapshot is opened in the CreateBackup request (stricter than the service's two-minute window), and the backup id is minted from the instant the open returned, just after the pin, so the backup holds every write acknowledged before the request was received, and backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (#384's seed and key), the row itself on MongoDB (born through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read. A job deletes a prefix, or a restore target, only after its FAILED write matched a row. The reconciler's duties are listed once, in section 4, and include the prefixes of DELETED rows, orphan restore targets, stale spools and, on Cassandra, active_restores entries whose target is not a CREATING restore target of the backup. Every duty that reads or writes a backup row acts only on rows of the configured store (store_kind, store_root and the store id); a CREATING or DELETING row of another store is reported and left alone. The PostgreSQL staleness test also selects a NULL heartbeat. While a maintenance claim is held the reconciler only finishes DELETING rows, resolves rows whose owner is dead and removes a FAILED restore's CREATING target. The PostgreSQL claim connection sets TCP keepalives, so the loss of the owner's host frees its lock in about a minute. While the configured store is unhealthy the reconciler acts on no row. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty. backup sync is scoped to the configured store; extenddb-store.json records the store id and the owning catalog's catalog_id, and a root another catalog owns, rows without the file, rows with two store ids or a missing newest manifest mark the store unhealthy (store operations refused, the reconciler idle) instead of stopping startup, so an unmounted volume, a recreated bucket or two catalogs on one root are caught; backup sync --adopt-store hands a root to another catalog; instances that share a catalog must share one store root; it never adopts a CREATING, DELETING, DELETED or FAILED prefix, nor one whose listing lacks a data file the manifest lists at its recorded size, and marks a row whose manifest head is gone DELETING only with --prune-missing. --move-store resolves the old root's CREATING and DELETING rows, compares every data file, refuses to carry a tombstone over a verified manifest and offers --forget-backup; --forget-store deletes the rows of a root. purge, --move-store, --forget-store, --prune-missing and --adopt-store take a catalog maintenance claim that every job checks; max_concurrent_jobs is per instance. DeleteBackup and Restore of another store's row return a ValidationException that names no root. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. The store-id file is created with put_if_absent (If-None-Match on S3, O_EXCL and link on the filesystem). A backup is in use while a restore target registered from it is CREATING, as on main: DeleteBackup and a second restore return BackupInUseException. Purge deletes each row's prefix, never a whole account prefix. Every backup row is read by account and backup id and must match the request's account, table and ARN, and no row stores its key prefix. DeleteBackup's in-use check and DELETING write run under the backup row's lock, and a restore's registration re-checks AVAILABLE and in-use under the same lock, so the two serialize as on main (on Cassandra through main's active_restores lightweight transactions, now for new rows too). - Restore returns CREATING from the request (TableAlreadyExists or TableInUse on a name race, as #384 returns), spools and verifies every data file before writing any item, and records progress atomically with the data on PostgreSQL and SQLite and by idempotent re-application of the one unrecorded file or batch on MongoDB and Cassandra; on MongoDB the job writes DONE and the backfill flag in one transaction and the worker's ACTIVE flip is conditioned on the flag, not on the job's generation. A key-ordered restore runs #394's per-file key-range check and an unordered one ends with a count of the target, which is Cassandra's only duplicate check. A PostgreSQL restore is resumed, or its target deleted, only after the stale window; the manifest and sidecar are spooled for a resume; a legacy restore with a dead owner is failed, not resumed. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as #384 and the service refuse it, and the job and the reconciler remove a failed target through the backend-internal remove_restore_target; the restore record is the table's child. Multi-part-key tables are restored (#384 refuses them; bulk_load keys rows with the backend's item-path functions, so the table is laid out as a natively created one). On PostgreSQL and SQLite bulk_load writes GSI, LSI and vector-index rows in its own transaction, and the ACTIVE flip sets every index ACTIVE. Table class is not restored, as on the service and #384; every override but VectorIndexOverride is applied, as #384 applies them. A second restore of a backup in use is refused before the permit and re-checked at registration. A restore stops on shutdown leaving its row RUNNING, download transport errors are retried for 15 minutes, and a takeover rewrites spool_host. Each data file is read through ExtenddbManifest::reader_prehashed, which applies the key-range check without hashing the spooled bytes again. - The engine binds a BackupCatalog trait (PR 6) with a JobClaim for the per-backend claim, heartbeat and conditional status writes; new rows live in backup_index (a new table or collection; Cassandra reuses its tables with store_kind set and adds backup_index_by_status); MongoDB's catalog gets the versioned migration step it lacks; PR 6's migration backfills in-flight #384 restores with heartbeat_at at the epoch so the first reconciler pass fails them; every pre-PR-6 server stops before migrate. Vector tables are backed up and restored on the off-host path, while the legacy create keeps main's refusal (one #384 engine test is rewritten in PR 9a and the wire test follows the configured path). source_table carries stream_specification and time_to_live for DescribeBackup. Both requests take the job permit last, after every refusal that needs no store access. The format's evolution rule is a version bump for any byte change, with v{N} fixtures beside v1. The trait gains listings for ListBackups, the store-id check and the root-matched commands, and backup_in_use. ListBackups accepts a table ARN in TableName from PR 6, as the service does. PR 6 adds the [backup] struct with the job keys. Row and restore-target methods take account-scoped keys, and the trait gains catalog_id and DeleteBackup's lock-holding start_backup_delete. - The manifest self-checksum is replaced by a sidecar over the stored bytes, accepted in its sha256sum line form only; backup_arn is the backup's identity and source_table must agree with it; N key values are normalized before key text or key order is computed, and an un-normalized key_range endpoint is refused; the version gate accepts 1 to the current version. The line cap is 32 MiB, sized from the measured worst case, and the compressed size is enforced while reading; both are format rules in section 1. Section 7 cites the decode memory from #394's DataFileDecoder doc (76 and 128 bytes per line byte as of #394, measured without preserve_order; about 4 GiB at the cap). A reader refuses a manifest without stream_specification or time_to_live. The backup job validates every item with an item validator PR 8a adds to extenddb-core, which the restore binds, and fails the backup on a value #618 let in through any of the four write paths, a regression from main for tables that hold one; it validates the rest of the snapshot first and logs every refused item, and the FAILED reason gives the count and the first. - Store (#393): owner-only root and directories re-checked by every operation, an ancestor writable by group or others without the sticky bit refused, a root whose canonical path changed since open refused, the prefix resolved with the same no-follow walk as a key, listings that report only keys passing the key rule, multipart uploads aborted when a put is dropped and swept by delete_prefix (a lifecycle rule as backstop), validate by a one-key listing so a 403 keeps S3's code and then by probes of the grants, a GetObject of an absent key included, a 255-byte component cap and U+FFFE/U+FFFF refused by the key rule, settings under [backup.store]. Section 8 gains a transport rule (a non-loopback http:// endpoint, from the configuration or from the SDK's own endpoint sources, needs allow_insecure_endpoint; an http:// S3 endpoint needs force_path_style unless allow_insecure_endpoint is set; the credential clients' endpoints (STS, SSO, SSO OIDC and the general AWS_ENDPOINT_URL or profile endpoint_url) are held to the loopback rule, and a refusal names the client and the setting; S3 Express directory buckets are refused; runtime shutdown sends no abort for in-flight puts, so the server's shutdown awaits its jobs' failure paths) and an IAM example with the list actions on the bucket, the KMS actions for an SSE-KMS bucket, the rule that a missing key reads as 404 only under s3:ListBucket, and a delete-marker lifecycle rule for versioned buckets. The endpoint is refused with user information, a query, a fragment or an IPv6 zone id, from any source; the S3 client for an http:// endpoint is built with aws_smithy_http_client's Builder and no proxy; a denied AbortMultipartUpload in delete_prefix's sweep is logged and the deletion succeeds; a put dropped during the temporary's open leaves no temporary. PR 7 adds put_if_absent to the store trait. - Configuration: the default store root is per backend (beside the database file on file-backed SQLite, where a defaulted root that fails validation is a warning, not a startup error); without [backup.store], CreateBackup on every backend keeps the legacy in-catalog create with a deprecation warning until PR 14 removes it (Cassandra always, having no SnapshotEngine), after which it is a ValidationException naming [backup.store], and kind = "disabled" is the opt-out; PR 14's migration refuses while legacy backup rows remain unless forced. A store transport failure at startup degrades the store component instead of stopping the server. The memory formula is in item text bytes, and PR 8a bounds each PostgreSQL and SQLite snapshot page by SNAPSHOT_FETCH_BYTES (64 MiB) in the page's query, PostgreSQL's cursor replaced by a keyset page under the same snapshot, so a page holds at most the budget plus one row; the PostgreSQL page returns its key columns, orders the outer query and sizes its row limit from the previous page's average row, starting at 16; the PostgreSQL pool rule is max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10), 18 at the default 4; the spool default is per backend and spool_host is host plus spool path. - Phase 1 is the PR series with acceptance criteria and GitHub numbers; migration numbers are reserved up front (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend and for DeleteBackup. The Phasing section refers to the companion PRs' heads on the revision date rather than naming commits. migrate-legacy re-reads and validates what it wrote and refuses, keeping the legacy rows, backups it cannot carry. Also adds the missing rows for designs 12 and 13 to docs/design/README.md and marks 15 as a draft. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 29, 2026
…hash, typed backup ARN Adds crates/core/src/backup_format, the on-disk format for off-host backups, built from #337's draft with the changes the design review asked for before v1 freezes. Data files: - DataFileEncoder<W: Write> hashes compressed bytes as they leave the compressor, so a writer streaming to a file or an upload holds the gzip window and one line, not the whole file. encode_data_file() remains as a convenience over it. encode_item_line_into is the one encoding of the {"Item": ...} line, into a buffer the caller keeps; encode_item_line, which the engine's ExportTableToPointInTime writer calls, and write_item, which reuses one buffer per encoder, both call it. write_item refuses a line over MAX_LINE_BYTES (32 MiB) with the encoder still usable, so a backup cannot contain a line the decoder would refuse at restore. The worst case at the default 400 KB item limit is a list of number sets, 67.6x (a 27.7 MB line, a 1.21x margin); operator-raised item limits above about 496 KB can exceed the cap. The doc carries the measured table and a test pins the worst case. A write error poisons the encoder (every later call is Encode); try_finish keeps the sink so an upload can be aborted, finish returns it. Only a successful finish writes the gzip trailer: dropping an encoder in any other state (fresh, open, poisoned) takes the sink out first, so flate2's Drop cannot complete a valid file of the items so far in a sink the caller abandoned. A sink that accepts zero bytes is WriteZero, not a spin; Interrupted is retried on every sink write and flush, and on the decoder's source reads. is_finished and is_poisoned are distinct. GZIP_LEVEL = 6 is pinned, and a test pins the output length and sha256 of a fixed input, so two runs of one build produce the same bytes. The writer emits one gzip member; the reader accepts concatenated members as one stream (a line may span members), which the reader contract states together with UTF-8 without BOM. - DataFileDecoder<R: Read> hashes the compressed bytes it consumes. The recorded compressed size and uncompressed size are enforced while reading (a source longer than its recorded compressed size fails with CompressedSizeMismatch after one byte past it); digests and the item count are checked at end of stream. Lines are capped at MAX_LINE_BYTES and read into one buffer the decoder reuses across items, as the encoder reuses its own; it keeps the longest line's capacity, at most the cap. Decoding one line peaks at about 76 bytes of heap per line byte for a list of empty strings, and at most about 128 for any shape measured (115 for the worst item, lists of one-member map chains; 127 for nested objects that are refused once built), whatever the nesting depth: about 4 GiB at the cap (measured without serde_json's preserve_order, the default binary; a build with the MongoDB backend enables it and measures about two thirds). DataFileDecoder's doc states the bound once. Any error is sticky (later next_item calls return Decode naming the earlier failure, never Ok(None)); the Iterator impl is fused on top so error-dropping consumers terminate. Line-level errors name the 1-based line in the file, and serde's column is labelled as within the line when serde's line is 1; a type descriptor or name ending in ` at line 3 column 4` sets serde's position, and such a message is shown as serde rendered it, with no column of its own. Items are checked for JSON shape only; the decoder does not apply the target's item limits, which the restore applies to each item before writing it. verified() keeps the slice path that checks digests before the first item, and then decodes without hashing again. A line ends in \n, except that the last line's may be absent (that line is an item, counted in uncompressed_bytes without a newline), and holds one object with the one member Item (an unknown member or a repeated top-level attribute is refused; a repeated type descriptor at the top of an attribute value is refused on this path and stays last-wins on the request path; nested maps keep JSON's last-wins); JSON whitespace, including a \r before the newline, is accepted as JSON accepts it. CompressedHasher is the incremental hasher a restore runs while spooling a download or over a file in place under an owner-only store root; DataFileEntry::decoder_prehashed and ExtenddbManifest::reader_prehashed then read that source without hashing it again (about 0.43 µs per 318-byte item, 9% of decoding) and keep every size, count and line check; the digests are not compared again, so a file replaced by one of another length is refused by the compressed-size check and a same-length replacement is not detected. AttributeValue deserializer (crates/core/src/types/attribute_value.rs, main's wire deserializer): the L and M arms now consume the serde_json::Value they own instead of cloning and re-deserializing it at each level. That copy had made an item PutItem accepts, 30 L wrappers around a list of 409,470 empty strings (31 containers, the maximum), cost 8.6 GB and 12.7 s to decode, and the request path, which shares the deserializer, the same. Results and every API error text are unchanged: a nested error is rendered once per level through serde_json::Error::custom, as main's per-level from_value rendered it, so a message ending in ` at line 0 column 0` loses one such suffix per level as before (a reviewer-run differential over 600k inputs, not committed; nested_error_text_is_unchanged pins thirteen nested texts, seven of that class at one to three levels with main's output, one of them with a suffix more than the levels strip). data_file_nested_decode_memory, which is not ignored, bounds a 32-level item's decode at 120 bytes per line byte. validation/number.rs (main's code, which every format parser and the request path reach) does its exponent arithmetic in i64. An exponent at either end of i32 combined with the digits no longer overflows: a debug build panicked on 10E2147483647, 49E2147483647 and .00E-2147483647, and a release build reported overflow as underflow and the reverse. An invalid N key's InvalidKey message carries the number validation's reason. Manifest: - Integrity is a sidecar, extenddb-manifest.sha256, in sha256sum format over the stored bytes (written by BackupDocuments::build, checked by verify_manifest_checksum), replacing an in-document digest over a re-serialized struct that no other implementation could reproduce. verify_manifest_checksum takes the sidecar as the bytes the store returned and accepts one shape only, the line the writer stores, "<64 lowercase hex> extenddb-manifest.json", with or without its final \n. Bytes that are not UTF-8 (refused with their own reason, the bytes not shown), a bare digest, which sha256sum -c refuses too, and anything else is InvalidManifest. The sidecar is written before the manifest, so the manifest's presence still marks a complete backup. - from_json_bytes requires a JSON document whose format_version is a u32, gates on it (check_format_version: 1 to CURRENT_FORMAT_VERSION; 0 and later versions are refused with a message naming the range this build reads) before any other member is interpreted, parses the document, then runs ExtenddbManifest::validate: the version gate, validate_source_table (item_count and table_size_bytes not negative; table_creation_date_time finite, since JSON has no NaN or infinity and serde_json writes them as null; the key schema has at least one HASH, HASH before RANGE, each key attribute named once and defined exactly once, on every manifest; the count of HASH and RANGE parts is left to the restore's CreateTable validation, and a multi-part key is backed up unordered), validate_identity (backup_arn is in canonical form and parses; backup_creation_date_time is exactly the f64 its id's instant gives; backup_name is 3-255 of the table-name alphabet; source_table.table_name is the ARN's table; source_table.table_arn, read with core::arn::parse, is a table ARN in the same partition for the same account and table) and validate_data_files (keys valid and unique, sha256 64 lowercase hex, md5 the standard base64 of 16 bytes, every file has items and non-zero sizes (entry errors are InvalidDataFileEntry, key errors InvalidDataFileKey), backup_size_bytes equals the sum without overflow, key_ordered manifests carry a key_range per file whose endpoints are key-only items of the schema's shape (S and B non-empty, N written in normalized form; the parser reads N endpoints as written, so 5.0 and 6E0 are refused) with first <= last and ranges ascending without overlap against every earlier file; unordered manifests carry none; an empty table is a manifest with no files). BackupDocuments::build runs validate too, so a writer cannot produce bytes the reader refuses. The document is parsed twice (once for the version, under 10% of the cost, once for the members); the doc states the cost per data file: about 6 to 12 µs and 1.6 KB retained per file with short keys. A key_range endpoint naming an attribute twice, or whose value repeats its type descriptor ({"S":"0","S":"A"}) or names two, is refused at parse time, as a data file line of that shape is. Non-overlapping ranges do not by themselves rule out a key in two files. ExtenddbManifest::reader (and reader_prehashed) runs KeyRangeCheck over each file's items when the manifest is key-ordered, without the caller constructing it, and its finish after the decoder's end-of-stream checks. The check refuses a first or last item that is not the endpoint, an item not strictly after the one before, an item past last, an item whose key KeyOrder::endpoint_key refuses, and an empty file; with the ranges, that rules duplicates out. KeyRangeCheck is a KeyRangeTracker plus the recorded range. The writer runs one tracker over the items of every file it encodes, so the endpoint check and the strict-ascent rule exist once: next_file closes a file with its KeyRange and keeps its last key, so a boundary that does not ascend (the next file starting with that key or one before it) is refused at the next file's first item, naming the previous file's last key, with nothing of that file encoded; finish closes the last file. With one tracker per file the same boundary is found only at build, after every file is stored. The tracker checks each key in place and keeps the first and latest keys, with an N range key parsed once; the check parses the recorded endpoints once. Per 318-byte item the check went from 1.07 µs and 11.5 allocations to 0.66 µs and 5 (one partition: 1.64 µs and 19.3 to 0.69 µs and 5.3). - Error messages carry a bounded slice of any untrusted field: at most 80 bytes of its escaped rendering, otherwise a 48-byte prefix and the total length. FormatError::Json holds a JsonError that bounds serde's text (which quotes the input) to 120 bytes in Display and Debug alike, and does not expose the inner error, as source() or otherwise; ChecksumMismatch prints its expected value with {:?}. - key_order.rs is the one definition of base-table key order: hash key by hash_key_text (S as is, N as held, B standard padded base64) compared as UTF-8 bytes, range key S as UTF-8 bytes, B as raw bytes and N by exact decimal value; the module doc states the N normalization rules (the exponent a signed 32-bit integer), the base64 alphabet and the string encoding another implementation must reproduce, and a test pins its examples. It states which writers store an invalid N key as written (BatchWriteItem and TransactWriteItems, #618; UpdateItem on an absent item and ImportTable run no number check on the key) and the rule for a key-ordered writer that meets one: the backup is refused with an error naming the key, through KeyRangeTracker::push, the one call per item, which checks the key as KeyOrder::endpoint_key does. N endpoints must be valid DynamoDB numbers written in normalized form; the parser refuses any other spelling, and items built in code are checked the same way. SnapshotDescriptor carries the three KIND_* constants the backends report, so the manifest and the storage layer share the strings; kind is a free string the reader does not validate and a restore must not branch on. KeyOrder::resolve refuses multi-part keys, duplicate attribute definitions, a range key listed before the hash key, and a range key with the hash key's name. The storage layer's pk_to_text delegates to hash_key_text. The module doc gives the per-backend ORDER BY recipe, including COLLATE "C" on Postgres text keys. - Optional source_table members serialize as explicit null. global_secondary_indexes, local_secondary_indexes, vector_indexes, stream_specification and time_to_live are required members (the last three may be null); stream_specification and time_to_live record what DescribeTable and DescribeTimeToLive reported, for DescribeBackup's SourceTableFeatureDetails, and are not restored. OnDemandThroughput writes null for an absent member, as on the wire, and the module doc records it; vector_indexes is three-state (null / [] / list) and holds the creation shape; sse_description, item_order, per-file md5 and uncompressed_bytes; the per-file compressed size feeds the decoder's expectations. - Golden fixtures testdata/extenddb-manifest/v1.json (every optional member set, a stream specification and a TTL description included), v1-minimal.json (every optional member absent, every list empty) and v1-unordered.json ("key_range":null files and the absent state of every nested optional member v1.json sets: an OnDemandThroughput with both limits null, an SseDescription without SSEType or KMSMasterKeyArn, a GSI without ProvisionedThroughput, a disabled stream specification without StreamViewType, a disabled TTL without AttributeName; and a vector index with SearchSchema and Projection, an LSI with INCLUDE), each with its .sha256, compared byte for byte, and a non-ignored test binds the writer's three constructors to the checked-in bytes. They are permanent read tests: the #[ignore]d writer (EXTENDDB_WRITE_GOLDEN) writes v{CURRENT} fixtures with create_new and refuses to overwrite an existing file. Any change to the bytes written for the fixtures' inputs is a version bump; a new member gets serde(default) and v{N} fixtures beside the v1 ones. A new member omitted when absent leaves the bytes unchanged, so nested_types_have_the_pinned_members pins the members every manifest type reads and writes, and fails it until the list changes. A new or renamed enum value that no fixture uses changes no byte either, so enum_values_are_pinned pins the values of every enum in the manifest: the derived ones through serde's variant list, and KeyType, ScalarAttributeType and DistanceFunction, whose readers are hand-written, through tables filled from a match with no wildcard, and AttributeValue, the value type of every key_range endpoint and data-file line, through its ten type descriptors the same way, plus a check that the descriptors its reader accepts among every string of one to four upper-case letters are exactly those ten. Unknown members are tolerated at every level. DynamoDB export manifests: ExportFileEntry.etag and s3SseAlgorithm are Option<String>; dataFileS3Key is the full store key, <prefix>/data/<file> (the service writes <s3Prefix>/AWSDynamoDB/<export id>/data/<file> for a full export to a prefix, measured; the documentation's full-export example omits the prefix; the module doc states both and what ExtendDB writes). dataFileS3Key and manifestFilesS3Key are built by one function, so an empty prefix gives data/<file> and manifest-files.json, without a leading /. Every leading and trailing / of the prefix is dropped once, so a// and /a/ give a/data/<file> and a/manifest-files.json, as a does; the store drops a leading /, one trailing / and refuses an empty component, so the two rules agree on every prefix the store accepts. DataFileEntry::from_checksums builds each entry from the encoder's result. BackupDocuments::build(header, data_files, store_key_prefix, bucket), with each data file paired with the ETag its put returned (None on a filesystem store), assembles the manifest, its sidecar, manifest-files.json and manifest-summary.json from one input. It derives backup_size_bytes, backup_creation_date_time and the key prefix (the store's key prefix with leading and trailing / dropped, then BackupArn::store_prefix) once and runs validate once. It builds each manifest-files.json line from the manifest's DataFileEntry, so the two manifests carry one md5 and one item_count, and the summary from the manifest (exportArn = the backup ARN, the three instants = the snapshot instant in RFC 3339 UTC with milliseconds, s3SseAlgorithm null, itemCount the sum; the module doc has the table). Its doc gives the write order: documents_before_manifest() returns manifest-files.json, manifest-summary.json and the sidecar, the job re-checks its claim, and manifest_document() is the manifest, written last; documents() is the four in that order. for_backup, from_data_file, write_files_manifest, to_json_bytes and manifest_checksum are private to backup_format. Layout: the backup id and ARN implementation moves from extenddb_storage::util::backup_id into extenddb_core::backup_format::layout (format_backup_id, is_backup_id, backup_id_epoch_millis, backup_creation_date_time, check_backup_arn, parse_backup_arn, BackupArnParts, BackupArnError, MAX_BACKUP_ARN_LEN), unchanged in behaviour (verified by a reviewer's differential run over 5.9M inputs); the field split is core::arn::fields and the table-name alphabet is validation::validate_table_name_chars, so authorization, CreateTable and this parser share them. The engine's validate_name_member and validate_index_name still spell the alphabet themselves. format_backup_arn is the one backup-ARN formatter (backup_arn() in storage, BackupArn, and the server's canonical resource ARN call it) and trim_backup_arn the one whitespace rule (check_backup_arn and the two authorization sites call it). It trims both ends as main did; the doc notes that the service resolves a trailing newline but refuses a leading space or tab. extenddb_storage::util re-exports the rest so the engine and the four backends keep their import path, and keeps new_backup_id (needs rand), backup_arn, and is_backup_resource_arn (the authorization predicate). BackupArn is the owned form the format uses: check_backup_arn plus the two store-prefix rules (table name at most 255 bytes and not only dots), with the ARN string and the store prefix; a test enumerates both divergences and the id-shape grid. The 255-byte bound is narrower than the engine's max_table_name_length when an operator raises it; the layout doc says so. Other copies of ARN facts in the server and storage crates (is_backup_resource_arn's own split, a partition constant) predate this PR and are not changed here. Tests: 175 in the module (174 run, 1 ignored fixture writer), three integration binaries in crates/core/tests (data_file_line_bomb: a 256 MiB single-line bomb stops within twice the cap's memory; data_file_nested_decode_memory: a 32-level item decodes within 120 bytes of heap per line byte; data_file_decode_peak: the ignored RSS measurement), a storage-sqlite test binding encode_orderable_number's byte order to KeyOrder::compare over a fixed pseudo-random corpus of DynamoDB numbers, so a change to either side fails, and a row-key contract test in storage pinning pk_to_text's literal output (standard base64, "+/+/") so a change to hash_key_text cannot silently re-key B rows. The ddb-export fixture is constructed from the documented output format, not captured from the service. Coverage includes the streaming end-of-stream digest check, every expectation a DataFileEntry carries, the uncompressed and compressed size caps firing early, wrong item counts in both directions, try_finish failure poisoning, Interrupted on every sink and source call, invalid UTF-8 as Decode, an unterminated final line as an item, dropped encoders writing nothing, line numbers in data-line errors, range-key pairs that tell bytewise from length-first and from base64 order, B hash keys in base64 order, N range keys differing beyond 17 digits, overlap between later files, last-endpoint validation, the per-file key-range check's refusals (a repeated key and an invalid stored N key among them) and two non-overlapping files each holding every key, through ExtenddbManifest::reader as well, a first item that is not the range's first and a file with no items through reader, reader_prehashed reading a wrong-digest file that reader refuses, the tracker refusing a boundary that does not ascend at the next file's first item with nothing encoded, the summary's itemCount as the sum when source_table.item_count differs, a non-UTF-8 sidecar, each etag on its file, the three documents before the manifest and the manifest document, the size sum refused at 2^63, the prehashed decoder's uncompressed count one too high, AttributeValue's ten descriptors and no other, the writer's tracker refusing an out-of-order or repeated key and agreeing with KeyOrder::compare for S, N and B keys, the prehashed decoder keeping every check but the digest, BackupDocuments' four documents agreeing on one prefix, the pinned enum values, N exponents at both ends of i32, a forged position in a data-line error, un-normalized and empty endpoints, resolve's key-order rules, required index members, negative source counts, the version gate's range and the gate's message naming the range 1 to CURRENT, unknown members injected at every level of two fixtures, the key-schema shape rules on an unordered manifest, to_json_bytes refusing what from_json_bytes refuses (non-finite timestamps included), endpoints repeating a type descriptor, the pinned member list of every manifest type, the two manifest constructors agreeing with the encoder and the manifest entry, the summary manifest's documented values, the key-order doc's normalization examples, size-sum overflow, md5 and sidecar shapes, identity rules, explicit nulls, bounded messages at field lengths from 81 bytes to 2 MB, a bounded Debug of FormatError::Json, an escaped ChecksumMismatch, and nested deserializer error texts. 32 hand mutants of these rules, and 39 more of the round-5 fixes, were each killed by the suite; 20 of the round-6 fixes were killed and one was equivalent (a dead branch in number.rs, replaced by three others, killed); 13 of the fixes for the maintainer's review of 2026-10-28 were killed, none equivalent; 15 of the round-7 fixes were killed, none equivalent; 18 of the round-8 fixes were killed at the test each targets, among them the four that survived round 8 (an AttributeValue descriptor alias, reader_prehashed hashing anyway, the summary's itemCount taken from source_table, build's size sum wrapping), none equivalent. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 29, 2026
Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), eight review rounds and the maintainer's reviews (on 2026-10-28 the sixth round, the maintainer's review and the seventh round; on 2026-10-29 the eighth, a sentence-level verification of the revision against the companions' code and main) against the companion PRs #393, #394 and #395, and a reconciliation against main abf8082 (#384 in its final form, the Cassandra backend #338, the PostgreSQL GSI build #401). The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). Indexes are reported in whatever build state they had and only an index being deleted is dropped; a GSI without its data table is left out with the attribute definitions only it used. On PostgreSQL's shared layout the open is one connection and one REPEATABLE READ transaction; on the separate layout the catalog row is held FOR SHARE until the data snapshot pins, and an index that becomes ACTIVE between the definition read and the pin is reported CREATING. PostgreSQL bounds open snapshots at half the pool and refuses more with LimitExceeded, so open snapshots leave half the pool to requests; from PR 6 on max_concurrent_jobs (default 4) bounds backups and restores together under that permit, and #384's restore slots go. It streams a table whose key columns predate COLLATE "C" unordered rather than sort it, and on both layouts DeleteTable commits the catalog removal only after its drops succeed, base table first, under one 1 s budget, taking its asynchronous path with the row kept when its probe finds a reader or a drop fails in a lock wait (lock timeout, deadlock victim or statement timeout), so no DROP waits behind a reader with the row gone; on the separate layout a task the request awaits commits the catalog and then the data transaction, and a data commit that fails after the catalog commit is retried on a fresh connection for a bounded time before the request returns (an orphan survives only a process death or a data database down past that bound, and the troubleshooting page says how to drop it); the control-plane poller probes held tables and retries each about once a second, at once on a snapshot's or an in-catalog backup's release, a drop DeleteTable deferred is due at once, and DeleteTable of a DELETING table returns its description on PostgreSQL and SQLite. On sqlite-memory the snapshot copies the table on the one connection inside the request and keeps a backup slot until its stream ends, so CreateBackup keeps working there once [backup.store] is set. MongoDB reads without the explicit snapshot level fail, and the document says so with the server's error. - The snapshot is opened in the CreateBackup request (stricter than the service's two-minute window), and the backup id is minted from the instant the open returned, just after the pin, so the backup holds every write acknowledged before the request was received, and backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (#384's seed and key), the row itself on MongoDB (born through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read. A job deletes a prefix, or a restore target, only after its FAILED write matched a row. The reconciler's duties are listed once, in section 4, and include the prefixes of DELETED rows, orphan restore targets, stale spools and, on Cassandra, active_restores entries whose target is not a CREATING restore target of the backup. Every duty that reads or writes a backup row acts only on rows of the configured store (store_kind, store_root and the store id); a CREATING or DELETING row of another store is reported and left alone. The PostgreSQL staleness test also selects a NULL heartbeat. While a maintenance claim is held the reconciler only finishes DELETING rows, resolves rows whose owner is dead and runs its FAILED-target duty, which never removes a target left for reason target_not_empty; that one is the operator's, removed with extenddb backup remove-target. The PostgreSQL claim connection sets TCP keepalives, so the loss of the owner's host frees its lock in about a minute. While the configured store is unhealthy the reconciler acts on no row of the store; its legacy-row duty, which touches no store, still runs. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty. backup sync is scoped to the configured store; extenddb-store.json records the store id and the owning catalog's catalog_id, and a root another catalog owns, rows without the file, rows with two store ids or a missing newest manifest mark the store unhealthy (store operations refused, the reconciler idle on the store's rows) instead of stopping startup, so an unmounted volume, a recreated bucket or two catalogs on one root are caught; backup sync --adopt-store hands a root to another catalog, and sqlite-memory's startup sync takes ownership of a root a sqlite-memory catalog owned the same way, so one catalog owns a root at a time there too; instances that share a catalog must share one store root; it never adopts a CREATING, DELETING, DELETED or FAILED prefix, nor one whose listing lacks a data file the manifest lists at its recorded size, and marks a row whose manifest head is gone DELETING only with --prune-missing. --move-store resolves the old root's CREATING and DELETING rows, compares every data file, refuses to carry a tombstone over a verified manifest and offers --forget-backup; --forget-store deletes the rows of a root. purge, remove-target, --move-store, --forget-store, --prune-missing and --adopt-store take a catalog maintenance claim that every job checks; max_concurrent_jobs is per instance. DeleteBackup and Restore of another store's row return a ValidationException that names no root. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. The store-id file is created with put_if_absent (If-None-Match on S3, O_EXCL and link on the filesystem). A backup is in use while a restore target registered from it is CREATING, as on main: DeleteBackup and a second restore return BackupInUseException. Purge deletes each row's prefix, never a whole account prefix. Every backup row is read by account and backup id and must match the request's account, table and ARN, and no row stores its key prefix. DeleteBackup's in-use check and DELETING write run under the backup row's lock, and a restore's registration re-checks AVAILABLE and in-use under the same lock, so the two serialize as on main (on Cassandra through main's active_restores lightweight transactions, now for new rows too); main's two PostgreSQL race tests stay for the legacy path and PR 6 adds the same two orderings over an off-host backup. - Restore returns CREATING from the request (TableAlreadyExists or TableInUse on a name race, as #384 returns), spools and verifies every data file before writing any item, and records progress atomically with the data on PostgreSQL and SQLite and by idempotent re-application of the one unrecorded file or batch on MongoDB and Cassandra; on MongoDB the job writes DONE and the backfill flag in one transaction and the worker's ACTIVE flip is conditioned on the flag, not on the job's generation. A key-ordered restore runs #394's per-file key-range check and an unordered one ends with a count of the target, which is Cassandra's only duplicate check. A PostgreSQL restore is resumed, or its target deleted, only after the stale window; the manifest and sidecar are spooled for a resume; a legacy restore with a dead owner is failed, not resumed. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as #384 and the service refuse it, and the job and the reconciler remove a failed target through the backend-internal remove_restore_target; the restore record is the table's child. Multi-part-key tables are restored (#384 refuses them; bulk_load keys rows with the backend's item-path functions, so the table is laid out as a natively created one). On PostgreSQL and SQLite bulk_load writes GSI, LSI and vector-index rows in the batch's transaction with the base rows, and the ACTIVE flip sets the table and every index ACTIVE in one catalog transaction; on Cassandra the index rows are created ACTIVE and the lightweight transaction flips the table alone. Table class is not restored, as on the service and #384; every override but VectorIndexOverride is applied, as #384 applies them. A second restore of a backup in use is refused before the permit and re-checked at registration. A restore stops on shutdown leaving its row RUNNING, download transport errors are retried for 15 minutes, and a takeover rewrites spool_host. Each data file is read through ExtenddbManifest::reader_prehashed, which applies the key-range check without hashing the spooled bytes again. - The engine binds a BackupCatalog trait (PR 6) with a JobClaim for the per-backend claim, heartbeat and conditional status writes; new rows live in backup_index (a new table or collection; Cassandra reuses its tables with store_kind set and adds backup_index_by_status); MongoDB's catalog gets the versioned migration step it lacks; PR 6's migration backfills in-flight #384 restores with heartbeat_at at the epoch so the first reconciler pass fails them; every pre-PR-6 server stops before migrate. Vector tables are backed up and restored on the off-host path, while the legacy create keeps main's refusal (one #384 engine test is rewritten in PR 9a and the wire test follows the configured path). source_table carries stream_specification and time_to_live for DescribeBackup. Both requests take the job permit last, after every refusal that needs no store access. The format's evolution rule is a version bump for any byte change, with v{N} fixtures beside v1. The trait gains listings for ListBackups, the store-id check and the root-matched commands, and backup_in_use. ListBackups accepts a table ARN in TableName from PR 6, as the service does. PR 6 adds the [backup] struct with the job keys. Row and restore-target methods take account-scoped keys, and the trait gains catalog_id and DeleteBackup's lock-holding start_backup_delete. - The manifest self-checksum is replaced by a sidecar over the stored bytes, accepted in its sha256sum line form only; backup_arn is the backup's identity and source_table must agree with it; N key values are normalized before key text or key order is computed, and an un-normalized key_range endpoint is refused; the version gate accepts 1 to the current version. The line cap is 32 MiB, sized from the measured worst case, and the compressed size is enforced while reading; both are format rules in section 1. Section 7 cites the decode memory from #394's DataFileDecoder doc (76 and 128 bytes per line byte as of #394, measured without preserve_order; about 4 GiB at the cap). A reader refuses a manifest without stream_specification or time_to_live. The backup job validates every item with an item validator PR 8a adds to extenddb-core, which the restore binds, and fails the backup on a value #618 let in through any of the four write paths, a regression from main for tables that hold one; it validates the rest of the snapshot first and logs every refused item, and the FAILED reason gives the count and the first. - Store (#393): owner-only root and directories re-checked by every operation, an ancestor writable by group or others without the sticky bit refused, a root whose canonical path changed since open refused, the prefix resolved with the same no-follow walk as a key, listings that report only keys passing the key rule, multipart uploads aborted when a put is dropped and swept by delete_prefix (a lifecycle rule as backstop), validate by a one-key listing so a 403 keeps S3's code and then by probes of the grants, a GetObject of an absent key included, a 255-byte component cap and U+FFFE/U+FFFF refused by the key rule, settings under [backup.store]. Section 8 gains a transport rule (a non-loopback http:// endpoint, from the configuration or from the SDK's own endpoint sources, needs allow_insecure_endpoint; an http:// S3 endpoint needs force_path_style unless allow_insecure_endpoint is set; the credential clients' endpoints (STS, SSO, SSO OIDC and the general AWS_ENDPOINT_URL or profile endpoint_url) are held to the loopback rule, the IMDS endpoint setting may be http:// only to loopback or the two IMDS addresses, and a refusal names the client and the setting; S3 Express directory buckets and directory-bucket access points (--x-s3, --xa-s3) are refused; runtime shutdown sends no abort for in-flight puts, so the server's shutdown awaits its jobs' failure paths) and an IAM example with the list actions on the bucket, the KMS actions for an SSE-KMS bucket, the rule that a missing key reads as 404 only under s3:ListBucket, and a delete-marker lifecycle rule for versioned buckets. The endpoint is refused with user information, a query, a fragment or an IPv6 zone id, from any source; the S3 client for an http:// endpoint is built with aws_smithy_http_client's Builder and no proxy, and the credential chain's clients get the same client when a credential endpoint setting, the IMDS endpoint setting or the container credentials URI is http://, so the chain's own http:// requests do not use the proxy; without such a setting the chain uses the SDK's default client and NO_PROXY must list 169.254.169.254 (fd00:ec2::254 in IPv6 mode) and 169.254.170.2; a denied AbortMultipartUpload in delete_prefix's sweep is logged and the deletion succeeds; a put dropped during the temporary's open leaves no temporary. PR 7 adds put_if_absent to the store trait. - Configuration: the default store root is per backend (beside the database file on file-backed SQLite, where a defaulted root that fails validation is a warning, not a startup error); without [backup.store], CreateBackup on every backend keeps the legacy in-catalog create with a deprecation warning until PR 14 removes it (Cassandra always, having no SnapshotEngine), after which it is a ValidationException naming [backup.store], and kind = "disabled" is the opt-out; PR 14's migration refuses while legacy backup rows remain unless forced. A store transport failure at startup degrades the store component instead of stopping the server. The memory formula is in item text bytes, and PR 8a bounds each PostgreSQL and SQLite snapshot page by SNAPSHOT_FETCH_BYTES (64 MiB) in the page's query, PostgreSQL's cursor replaced by a keyset page under the same snapshot, so a page holds at most the budget plus one row; the PostgreSQL page returns its key columns, orders the outer query and sizes its row limit from the previous page's average row, starting at 16; the PostgreSQL pool rule is max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10), 18 at the default 4; the spool default is per backend and spool_host is host plus spool path. - Phase 1 is the PR series with acceptance criteria and GitHub numbers; migration numbers are reserved up front (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend and for DeleteBackup. The Phasing section refers to the companion PRs' heads on the revision date rather than naming commits. migrate-legacy re-reads and validates what it wrote and refuses, keeping the legacy rows, backups it cannot carry. Also adds the missing rows for designs 12 and 13 to docs/design/README.md and marks 15 as a draft. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 29, 2026
Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), eight review rounds and the maintainer's reviews (on 2026-10-28 the sixth round, the maintainer's review and the seventh round; on 2026-10-29 the eighth, a sentence-level verification of the revision against the companions' code and main) against the companion PRs #393, #394 and #395, and a reconciliation against main abf8082 (#384 in its final form, the Cassandra backend #338, the PostgreSQL GSI build #401). The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). Indexes are reported in whatever build state they had and only an index being deleted is dropped; a GSI without its data table is left out with the attribute definitions only it used. On PostgreSQL's shared layout the open is one connection and one REPEATABLE READ transaction; on the separate layout the catalog row is held FOR SHARE until the data snapshot pins, and an index that becomes ACTIVE between the definition read and the pin is reported CREATING. PostgreSQL bounds open snapshots at half the pool and refuses more with LimitExceeded, so open snapshots leave half the pool to requests; from PR 6 on max_concurrent_jobs (default 4) bounds backups and restores together under that permit, and #384's restore slots go. It streams a table whose key columns predate COLLATE "C" unordered rather than sort it, and on both layouts DeleteTable commits the catalog removal only after its drops succeed, base table first, under one 1 s budget, taking its asynchronous path with the row kept when its probe finds a reader or a drop fails in a lock wait (lock timeout, deadlock victim or statement timeout), so no DROP waits behind a reader with the row gone; on the separate layout a task the request awaits commits the catalog and then the data transaction, and a data commit that fails after the catalog commit is retried on a fresh connection for a bounded time before the request returns (an orphan survives only a process death or a data database down past that bound, and the troubleshooting page says how to drop it); the control-plane poller probes held tables and retries each about once a second, at once on a snapshot's or an in-catalog backup's release, a drop DeleteTable deferred is due at once, and DeleteTable of a DELETING table returns its description on PostgreSQL and SQLite. On sqlite-memory the snapshot copies the table on the one connection inside the request and keeps a backup slot until its stream ends, so CreateBackup keeps working there once [backup.store] is set. MongoDB reads without the explicit snapshot level fail, and the document says so with the server's error. - The snapshot is opened in the CreateBackup request (stricter than the service's two-minute window), and the backup id is minted from the instant the open returned, just after the pin, so the backup holds every write acknowledged before the request was received, and backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (#384's seed and key), the row itself on MongoDB (born through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read. A job deletes a prefix, or a restore target, only after its FAILED write matched a row. The reconciler's duties are listed once, in section 4, and include the prefixes of DELETED rows, orphan restore targets, stale spools and, on Cassandra, active_restores entries whose target is not a CREATING restore target of the backup. Every duty that reads or writes a backup row acts only on rows of the configured store (store_kind, store_root and the store id); a CREATING or DELETING row of another store is reported and left alone. The PostgreSQL staleness test also selects a NULL heartbeat. While a maintenance claim is held the reconciler only finishes DELETING rows, resolves rows whose owner is dead and runs its FAILED-target duty, which never removes a target left for reason target_not_empty; that one is the operator's, removed with extenddb backup remove-target. The PostgreSQL claim connection sets TCP keepalives, so the loss of the owner's host frees its lock in about a minute. While the configured store is unhealthy the reconciler acts on no row of the store; its legacy-row duty, which touches no store, still runs. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty. backup sync is scoped to the configured store; extenddb-store.json records the store id and the owning catalog's catalog_id, and a root another catalog owns, rows without the file, rows with two store ids or a missing newest manifest mark the store unhealthy (store operations refused, the reconciler idle on the store's rows) instead of stopping startup, so an unmounted volume, a recreated bucket or two catalogs on one root are caught; backup sync --adopt-store hands a root to another catalog, and sqlite-memory's startup sync takes ownership of a root a sqlite-memory catalog owned the same way, so one catalog owns a root at a time there too; instances that share a catalog must share one store root; it never adopts a CREATING, DELETING, DELETED or FAILED prefix, nor one whose listing lacks a data file the manifest lists at its recorded size, and marks a row whose manifest head is gone DELETING only with --prune-missing. --move-store resolves the old root's CREATING and DELETING rows, compares every data file, refuses to carry a tombstone over a verified manifest and offers --forget-backup; --forget-store deletes the rows of a root. purge, remove-target, --move-store, --forget-store, --prune-missing and --adopt-store take a catalog maintenance claim that every job checks; max_concurrent_jobs is per instance. DeleteBackup and Restore of another store's row return a ValidationException that names no root. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. The store-id file is created with put_if_absent (If-None-Match on S3, O_EXCL and link on the filesystem). A backup is in use while a restore target registered from it is CREATING, as on main: DeleteBackup and a second restore return BackupInUseException. Purge deletes each row's prefix, never a whole account prefix. Every backup row is read by account and backup id and must match the request's account, table and ARN, and no row stores its key prefix. DeleteBackup's in-use check and DELETING write run under the backup row's lock, and a restore's registration re-checks AVAILABLE and in-use under the same lock, so the two serialize as on main (on Cassandra through main's active_restores lightweight transactions, now for new rows too); main's two PostgreSQL race tests stay for the legacy path and PR 6 adds the same two orderings over an off-host backup. - Restore returns CREATING from the request (TableAlreadyExists or TableInUse on a name race, as #384 returns), spools and verifies every data file before writing any item, and records progress atomically with the data on PostgreSQL and SQLite and by idempotent re-application of the one unrecorded file or batch on MongoDB and Cassandra; on MongoDB the job writes DONE and the backfill flag in one transaction and the worker's ACTIVE flip is conditioned on the flag, not on the job's generation. A key-ordered restore runs #394's per-file key-range check and an unordered one ends with a count of the target, which is Cassandra's only duplicate check. A PostgreSQL restore is resumed, or its target deleted, only after the stale window; the manifest and sidecar are spooled for a resume; a legacy restore with a dead owner is failed, not resumed. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as #384 and the service refuse it, and the job and the reconciler remove a failed target through the backend-internal remove_restore_target; the restore record is the table's child. Multi-part-key tables are restored (#384 refuses them; bulk_load keys rows with the backend's item-path functions, so the table is laid out as a natively created one). On PostgreSQL and SQLite bulk_load writes GSI, LSI and vector-index rows in the batch's transaction with the base rows, and the ACTIVE flip sets the table and every index ACTIVE in one catalog transaction; on Cassandra the index rows are created ACTIVE and the lightweight transaction flips the table alone. Table class is not restored, as on the service and #384; every override but VectorIndexOverride is applied, as #384 applies them. A second restore of a backup in use is refused before the permit and re-checked at registration. A restore stops on shutdown leaving its row RUNNING, download transport errors are retried for 15 minutes, and a takeover rewrites spool_host. Each data file is read through ExtenddbManifest::reader_prehashed, which applies the key-range check without hashing the spooled bytes again. - The engine binds a BackupCatalog trait (PR 6) with a JobClaim for the per-backend claim, heartbeat and conditional status writes; new rows live in backup_index (a new table or collection; Cassandra reuses its tables with store_kind set and adds backup_index_by_status); MongoDB's catalog gets the versioned migration step it lacks; PR 6's migration backfills in-flight #384 restores with heartbeat_at at the epoch so the first reconciler pass fails them; every pre-PR-6 server stops before migrate. Vector tables are backed up and restored on the off-host path, while the legacy create keeps main's refusal (one #384 engine test is rewritten in PR 9a and the wire test follows the configured path). source_table carries stream_specification and time_to_live for DescribeBackup. Both requests take the job permit last, after every refusal that needs no store access. The format's evolution rule is a version bump for any byte change, with v{N} fixtures beside v1. The trait gains listings for ListBackups, the store-id check and the root-matched commands, and backup_in_use. ListBackups accepts a table ARN in TableName from PR 6, as the service does. PR 6 adds the [backup] struct with the job keys. Row and restore-target methods take account-scoped keys, and the trait gains catalog_id and DeleteBackup's lock-holding start_backup_delete. - The manifest self-checksum is replaced by a sidecar over the stored bytes, accepted in its sha256sum line form only; backup_arn is the backup's identity and source_table must agree with it; N key values are normalized before key text or key order is computed, and an un-normalized key_range endpoint is refused; the version gate accepts 1 to the current version. The line cap is 32 MiB, sized from the measured worst case, and the compressed size is enforced while reading; both are format rules in section 1. Section 7 cites the decode memory from #394's DataFileDecoder doc (76 and 128 bytes per line byte as of #394, measured without preserve_order; about 4 GiB at the cap). A reader refuses a manifest without stream_specification or time_to_live. The backup job validates every item with an item validator PR 8 adds to extenddb-core, which the restore binds, and fails the backup on a value #618 let in through any of the four write paths, a regression from main for tables that hold one; it validates the rest of the snapshot first and logs every refused item, and the FAILED reason gives the count and the first. - Store (#393): owner-only root and directories re-checked by every operation, an ancestor writable by group or others without the sticky bit refused, a root whose canonical path changed since open refused, the prefix resolved with the same no-follow walk as a key, listings that report only keys passing the key rule, multipart uploads aborted when a put is dropped and swept by delete_prefix (a lifecycle rule as backstop), validate by a one-key listing so a 403 keeps S3's code and then by probes of the grants, a GetObject of an absent key included, a 255-byte component cap and U+FFFE/U+FFFF refused by the key rule, settings under [backup.store]. Section 8 gains a transport rule (a non-loopback http:// endpoint, from the configuration or from the SDK's own endpoint sources, needs allow_insecure_endpoint; an http:// S3 endpoint needs force_path_style unless allow_insecure_endpoint is set; the credential clients' endpoints (STS, SSO, SSO OIDC and the general AWS_ENDPOINT_URL or profile endpoint_url) are held to the loopback rule, the IMDS endpoint setting may be http:// only to loopback or the two IMDS addresses, and a refusal names the client and the setting; S3 Express directory buckets and directory-bucket access points (--x-s3, --xa-s3) are refused; runtime shutdown sends no abort for in-flight puts, so the server's shutdown awaits its jobs' failure paths) and an IAM example with the list actions on the bucket, the KMS actions for an SSE-KMS bucket, the rule that a missing key reads as 404 only under s3:ListBucket, and a delete-marker lifecycle rule for versioned buckets. The endpoint is refused with user information, a query, a fragment or an IPv6 zone id, from any source; the S3 client for an http:// endpoint is built with aws_smithy_http_client's Builder and no proxy, and the credential chain's clients get the same client when a credential endpoint setting, the IMDS endpoint setting or the container credentials URI is http://, so the chain's own http:// requests do not use the proxy; without such a setting the chain uses the SDK's default client and NO_PROXY must list 169.254.169.254 (fd00:ec2::254 in IPv6 mode) and 169.254.170.2; a denied AbortMultipartUpload in delete_prefix's sweep is logged and the deletion succeeds; a put dropped during the temporary's open leaves no temporary. PR 7 adds put_if_absent to the store trait. - Configuration: the default store root is per backend (beside the database file on file-backed SQLite, where a defaulted root that fails validation is a warning, not a startup error); without [backup.store], CreateBackup on every backend keeps the legacy in-catalog create with a deprecation warning until PR 14 removes it (Cassandra always, having no SnapshotEngine), after which it is a ValidationException naming [backup.store], and kind = "disabled" is the opt-out; PR 14's migration refuses while legacy backup rows remain unless forced. A store transport failure at startup degrades the store component instead of stopping the server. The memory formula is in item text bytes, and PR 8 bounds each PostgreSQL and SQLite snapshot page by SNAPSHOT_FETCH_BYTES (64 MiB) in the page's query, PostgreSQL's cursor replaced by a keyset page under the same snapshot, so a page holds at most the budget plus one row; the PostgreSQL page returns its key columns, orders the outer query and sizes its row limit from the previous page's average row, starting at 16; the PostgreSQL pool rule is max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10), 18 at the default 4; the spool default is per backend and spool_host is host plus spool path. - Phase 1 is the PR series with acceptance criteria and GitHub numbers; migration numbers are reserved up front (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend and for DeleteBackup. The Phasing section refers to the companion PRs' heads on the revision date rather than naming commits. migrate-legacy re-reads and validates what it wrote and refuses, keeping the legacy rows, backups it cannot carry. Also adds the missing rows for designs 12 and 13 to docs/design/README.md and marks 15 as a draft. The series after the four open PRs is ten PRs: the reconciler ships with the CreateBackup job as PR 8 (a job without its reconciler leaves rows no one resolves), and the CI changes and the documentation ship together as PR 11, since both finalize after PR 9b. The revision log (item 24) records the renaming; its earlier items keep the old labels. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 29, 2026
…hash, typed backup ARN Adds crates/core/src/backup_format, the on-disk format for off-host backups, built from #337's draft with the changes the design review asked for before v1 freezes. The module doc says that the design overview (docs/design/15-backup-restore.md, #396) states the rules the format's writers and readers honour, and that this module's docs and the testdata README are the record of the format itself. Data files: - DataFileEncoder<W: Write> hashes compressed bytes as they leave the compressor, so a writer streaming to a file or an upload holds the gzip window and one line, not the whole file. encode_data_file() remains as a convenience over it. encode_item_line_into is the one encoding of the {"Item": ...} line, into a buffer the caller keeps; encode_item_line, which the engine's ExportTableToPointInTime writer calls, and write_item, which reuses one buffer per encoder, both call it. write_item refuses a line over MAX_LINE_BYTES (32 MiB) with the encoder still usable, so a backup cannot contain a line the decoder would refuse at restore. The worst case at the default 400 KB item limit is a list of number sets, 67.6x (a 27.7 MB line, a 1.21x margin); operator-raised item limits above about 496 KB can exceed the cap. The doc carries the measured table and a test pins the worst case. A write error poisons the encoder (every later call is Encode); try_finish keeps the sink so an upload can be aborted, finish returns it. Only a successful finish writes the gzip trailer: dropping an encoder in any other state (fresh, open, poisoned) takes the sink out first, so flate2's Drop cannot complete a valid file of the items so far in a sink the caller abandoned. A sink that accepts zero bytes is WriteZero, not a spin; Interrupted is retried on every sink write and flush, and on the decoder's source reads. is_finished and is_poisoned are distinct. GZIP_LEVEL = 6 is pinned, and a test pins the output length and sha256 of a fixed input, so two runs of one build produce the same bytes. The writer emits one gzip member; the reader accepts concatenated members as one stream (a line may span members), which the reader contract states together with UTF-8 without BOM. - DataFileDecoder<R: Read> hashes the compressed bytes it consumes. The recorded compressed size and uncompressed size are enforced while reading (a source longer than its recorded compressed size fails with CompressedSizeMismatch after one byte past it); digests and the item count are checked at end of stream. Lines are capped at MAX_LINE_BYTES and read into one buffer the decoder reuses across items, as the encoder reuses its own; it keeps the longest line's capacity, at most the cap. Decoding one line peaks at about 76 bytes of heap per line byte for a list of empty strings, and at most about 128 for any shape measured (115 for the worst item, lists of one-member map chains; 127 for nested objects that are refused once built), whatever the nesting depth: about 4 GiB at the cap (measured without serde_json's preserve_order, the default binary; a build with the MongoDB backend enables it and measures about two thirds). DataFileDecoder's doc states the bound once. Any error is sticky (later next_item calls return Decode naming the earlier failure, never Ok(None)); the Iterator impl is fused on top so error-dropping consumers terminate. Line-level errors name the 1-based line in the file, and serde's column is labelled as within the line when serde's line is 1; a type descriptor or name ending in ` at line 3 column 4` sets serde's position, and such a message is shown as serde rendered it, with no column of its own. Items are checked for JSON shape only; the decoder does not apply the target's item limits, which the restore applies to each item before writing it. verified() keeps the slice path that checks digests before the first item, and then decodes without hashing again. A line ends in \n, except that the last line's may be absent (that line is an item, counted in uncompressed_bytes without a newline), and holds one object with the one member Item (an unknown member or a repeated top-level attribute is refused; a repeated type descriptor at the top of an attribute value is refused on this path and stays last-wins on the request path; nested maps keep JSON's last-wins); JSON whitespace, including a \r before the newline, is accepted as JSON accepts it. CompressedHasher is the incremental hasher a restore runs while spooling a download or over a file in place under an owner-only store root; DataFileEntry::decoder_prehashed and ExtenddbManifest::reader_prehashed then read that source without hashing it again (about 0.43 µs per 318-byte item, 9% of decoding) and keep every size, count and line check; the digests are not compared again, so a file replaced by one of another length is refused by the compressed-size check and a same-length replacement is not detected. AttributeValue deserializer (crates/core/src/types/attribute_value.rs, main's wire deserializer): the L and M arms now consume the serde_json::Value they own instead of cloning and re-deserializing it at each level. That copy had made an item PutItem accepts, 30 L wrappers around a list of 409,470 empty strings (31 containers, the maximum), cost 8.6 GB and 12.7 s to decode, and the request path, which shares the deserializer, the same. Results and every API error text are unchanged: a nested error is rendered once per level through serde_json::Error::custom, as main's per-level from_value rendered it, so a message ending in ` at line 0 column 0` loses one such suffix per level as before (a reviewer-run differential over 600k inputs, not committed; nested_error_text_is_unchanged pins thirteen nested texts, seven of that class at one to three levels with main's output, one of them with a suffix more than the levels strip). data_file_nested_decode_memory, which is not ignored, bounds a 32-level item's decode at 120 bytes per line byte. validation/number.rs (main's code, which every format parser and the request path reach) does its exponent arithmetic in i64. An exponent at either end of i32 combined with the digits no longer overflows: a debug build panicked on 10E2147483647, 49E2147483647 and .00E-2147483647, and a release build reported overflow as underflow and the reverse. An invalid N key's InvalidKey message carries the number validation's reason. Manifest: - Integrity is a sidecar, extenddb-manifest.sha256, in sha256sum format over the stored bytes (written by BackupDocuments::build, checked by verify_manifest_checksum), replacing an in-document digest over a re-serialized struct that no other implementation could reproduce. verify_manifest_checksum takes the sidecar as the bytes the store returned and accepts one shape only, the line the writer stores, "<64 lowercase hex> extenddb-manifest.json", with or without its final \n. Bytes that are not UTF-8 (refused with their own reason, the bytes not shown), a bare digest, which sha256sum -c refuses too, and anything else is InvalidManifest. The sidecar is written before the manifest, so the manifest's presence still marks a complete backup. - from_json_bytes requires a JSON document whose format_version is a u32, gates on it (check_format_version: 1 to CURRENT_FORMAT_VERSION; 0 and later versions are refused with a message naming the range this build reads) before any other member is interpreted, parses the document, then runs ExtenddbManifest::validate: the version gate, validate_source_table (item_count and table_size_bytes not negative; table_creation_date_time finite, since JSON has no NaN or infinity and serde_json writes them as null; the key schema has at least one HASH, HASH before RANGE, each key attribute named once and defined exactly once, on every manifest; the count of HASH and RANGE parts is left to the restore's CreateTable validation, and a multi-part key is backed up unordered), validate_identity (backup_arn is in canonical form and parses; backup_creation_date_time is exactly the f64 its id's instant gives; backup_name is 3-255 of the table-name alphabet; source_table.table_name is the ARN's table; source_table.table_arn, read with core::arn::parse, is a table ARN in the same partition for the same account and table) and validate_data_files (keys valid and unique, sha256 64 lowercase hex, md5 the standard base64 of 16 bytes, every file has items and non-zero sizes (entry errors are InvalidDataFileEntry, key errors InvalidDataFileKey), backup_size_bytes equals the sum without overflow, key_ordered manifests carry a key_range per file whose endpoints are key-only items of the schema's shape (S and B non-empty, N written in normalized form; the parser reads N endpoints as written, so 5.0 and 6E0 are refused) with first <= last and ranges ascending without overlap against every earlier file; unordered manifests carry none; an empty table is a manifest with no files). BackupDocuments::build runs validate too, so a writer cannot produce bytes the reader refuses. The document is parsed twice (once for the version, under 10% of the cost, once for the members); the doc states the cost per data file: about 6 to 12 µs and 1.6 KB retained per file with short keys. A key_range endpoint naming an attribute twice, or whose value repeats its type descriptor ({"S":"0","S":"A"}) or names two, is refused at parse time, as a data file line of that shape is. Non-overlapping ranges do not by themselves rule out a key in two files. ExtenddbManifest::reader (and reader_prehashed) runs KeyRangeCheck over each file's items when the manifest is key-ordered, without the caller constructing it, and its finish after the decoder's end-of-stream checks. The check refuses a first or last item that is not the endpoint, an item not strictly after the one before, an item past last, an item whose key KeyOrder::endpoint_key refuses, and an empty file; with the ranges, that rules duplicates out. KeyRangeCheck is a KeyRangeTracker plus the recorded range. The writer runs one tracker over the items of every file it encodes, so the endpoint check and the strict-ascent rule exist once: next_file closes a file with its KeyRange and keeps its last key, so a boundary that does not ascend (the next file starting with that key or one before it) is refused at the next file's first item, naming the previous file's last key, with nothing of that file encoded; finish closes the last file. With one tracker per file the same boundary is found only at build, after every file is stored. The tracker checks each key in place and keeps the first and latest keys, with an N range key parsed once; the check parses the recorded endpoints once. Per 318-byte item the check went from 1.07 µs and 11.5 allocations to 0.66 µs and 5 (one partition: 1.64 µs and 19.3 to 0.69 µs and 5.3). - Error messages carry a bounded slice of any untrusted field: at most 80 bytes of its escaped rendering, otherwise a 48-byte prefix and the total length. FormatError::Json holds a JsonError that bounds serde's text (which quotes the input) to 120 bytes in Display and Debug alike, and does not expose the inner error, as source() or otherwise; ChecksumMismatch prints its expected value with {:?}. - key_order.rs is the one definition of base-table key order: hash key by hash_key_text (S as is, N as held, B standard padded base64) compared as UTF-8 bytes, range key S as UTF-8 bytes, B as raw bytes and N by exact decimal value; the module doc states the N normalization rules (the exponent a signed 32-bit integer), the base64 alphabet and the string encoding another implementation must reproduce, and a test pins its examples. It states which writers store an invalid N key as written (BatchWriteItem and TransactWriteItems, #618; UpdateItem on an absent item and ImportTable run no number check on the key) and the rule for a key-ordered writer that meets one: the backup is refused with an error naming the key, through KeyRangeTracker::push, the one call per item, which checks the key as KeyOrder::endpoint_key does. N endpoints must be valid DynamoDB numbers written in normalized form; the parser refuses any other spelling, and items built in code are checked the same way. SnapshotDescriptor carries the three KIND_* constants the backends report, so the manifest and the storage layer share the strings; kind is a free string the reader does not validate and a restore must not branch on. KeyOrder::resolve refuses multi-part keys, duplicate attribute definitions, a range key listed before the hash key, and a range key with the hash key's name. The storage layer's pk_to_text delegates to hash_key_text. The module doc gives the per-backend ORDER BY recipe, including COLLATE "C" on Postgres text keys. - Optional source_table members serialize as explicit null. global_secondary_indexes, local_secondary_indexes, vector_indexes, stream_specification and time_to_live are required members (the last three may be null); stream_specification and time_to_live record what DescribeTable and DescribeTimeToLive reported, for DescribeBackup's SourceTableFeatureDetails, and are not restored. OnDemandThroughput writes null for an absent member, as on the wire, and the module doc records it; vector_indexes is three-state (null / [] / list) and holds the creation shape; sse_description, item_order, per-file md5 and uncompressed_bytes; the per-file compressed size feeds the decoder's expectations. - Golden fixtures testdata/extenddb-manifest/v1.json (every optional member set, a stream specification and a TTL description included), v1-minimal.json (every optional member absent, every list empty) and v1-unordered.json ("key_range":null files and the absent state of every nested optional member v1.json sets: an OnDemandThroughput with both limits null, an SseDescription without SSEType or KMSMasterKeyArn, a GSI without ProvisionedThroughput, a disabled stream specification without StreamViewType, a disabled TTL without AttributeName; and a vector index with SearchSchema and Projection, an LSI with INCLUDE), each with its .sha256, compared byte for byte, and a non-ignored test binds the writer's three constructors to the checked-in bytes. They are permanent read tests: the #[ignore]d writer (EXTENDDB_WRITE_GOLDEN) writes v{CURRENT} fixtures with create_new and refuses to overwrite an existing file. Any change to the bytes written for the fixtures' inputs is a version bump; a new member gets serde(default) and v{N} fixtures beside the v1 ones. A new member omitted when absent leaves the bytes unchanged, so nested_types_have_the_pinned_members pins the members every manifest type reads and writes, and fails it until the list changes. A new or renamed enum value that no fixture uses changes no byte either, so enum_values_are_pinned pins the values of every enum in the manifest: the derived ones through serde's variant list, and KeyType, ScalarAttributeType and DistanceFunction, whose readers are hand-written, through tables filled from a match with no wildcard, and AttributeValue, the value type of every key_range endpoint and data-file line, through its ten type descriptors the same way, plus a check that the descriptors its reader accepts among every string of one to four upper-case letters are exactly those ten. Unknown members are tolerated at every level. DynamoDB export manifests: ExportFileEntry.etag and s3SseAlgorithm are Option<String>; dataFileS3Key is the full store key, <prefix>/data/<file> (the service writes <s3Prefix>/AWSDynamoDB/<export id>/data/<file> for a full export to a prefix, measured; the documentation's full-export example omits the prefix; the module doc states both and what ExtendDB writes). dataFileS3Key and manifestFilesS3Key are built by one function, so an empty prefix gives data/<file> and manifest-files.json, without a leading /. Every leading and trailing / of the prefix is dropped once, so a// and /a/ give a/data/<file> and a/manifest-files.json, as a does; the store drops a leading /, one trailing / and refuses an empty component, so the two rules agree on every prefix the store accepts. DataFileEntry::from_checksums builds each entry from the encoder's result. BackupDocuments::build(header, data_files, store_key_prefix, bucket), with each data file paired with the ETag its put returned (None on a filesystem store), assembles the manifest, its sidecar, manifest-files.json and manifest-summary.json from one input. It derives backup_size_bytes, backup_creation_date_time and the key prefix (the store's key prefix with leading and trailing / dropped, then BackupArn::store_prefix) once and runs validate once. It builds each manifest-files.json line from the manifest's DataFileEntry, so the two manifests carry one md5 and one item_count, and the summary from the manifest (exportArn = the backup ARN, the three instants = the snapshot instant in RFC 3339 UTC with milliseconds, s3SseAlgorithm null, itemCount the sum; the module doc has the table). Its doc gives the write order: documents_before_manifest() returns manifest-files.json, manifest-summary.json and the sidecar, the job re-checks its claim, and manifest_document() is the manifest, written last; documents() is the four in that order. for_backup, from_data_file, write_files_manifest, to_json_bytes and manifest_checksum are private to backup_format. Layout: the backup id and ARN implementation moves from extenddb_storage::util::backup_id into extenddb_core::backup_format::layout (format_backup_id, is_backup_id, backup_id_epoch_millis, backup_creation_date_time, check_backup_arn, parse_backup_arn, BackupArnParts, BackupArnError, MAX_BACKUP_ARN_LEN), unchanged in behaviour (verified by a reviewer's differential run over 5.9M inputs); the field split is core::arn::fields and the table-name alphabet is validation::validate_table_name_chars, so authorization, CreateTable and this parser share them. The engine's validate_name_member and validate_index_name still spell the alphabet themselves. format_backup_arn is the one backup-ARN formatter (backup_arn() in storage, BackupArn, and the server's canonical resource ARN call it) and trim_backup_arn the one whitespace rule (check_backup_arn and the two authorization sites call it). It trims both ends as main did; the doc notes that the service resolves a trailing newline but refuses a leading space or tab. extenddb_storage::util re-exports the rest so the engine and the four backends keep their import path, and keeps new_backup_id (needs rand), backup_arn, and is_backup_resource_arn (the authorization predicate). BackupArn is the owned form the format uses: check_backup_arn plus the two store-prefix rules (table name at most 255 bytes and not only dots), with the ARN string and the store prefix; a test enumerates both divergences and the id-shape grid. The 255-byte bound is narrower than the engine's max_table_name_length when an operator raises it; the layout doc says so. Other copies of ARN facts in the server and storage crates (is_backup_resource_arn's own split, a partition constant) predate this PR and are not changed here. Tests: 175 in the module (174 run, 1 ignored fixture writer), three integration binaries in crates/core/tests (data_file_line_bomb: a 256 MiB single-line bomb stops within twice the cap's memory; data_file_nested_decode_memory: a 32-level item decodes within 120 bytes of heap per line byte; data_file_decode_peak: the ignored RSS measurement), a storage-sqlite test binding encode_orderable_number's byte order to KeyOrder::compare over a fixed pseudo-random corpus of DynamoDB numbers, so a change to either side fails, and a row-key contract test in storage pinning pk_to_text's literal output (standard base64, "+/+/") so a change to hash_key_text cannot silently re-key B rows. The ddb-export fixture is constructed from the documented output format, not captured from the service. Coverage includes the streaming end-of-stream digest check, every expectation a DataFileEntry carries, the uncompressed and compressed size caps firing early, wrong item counts in both directions, try_finish failure poisoning, Interrupted on every sink and source call, invalid UTF-8 as Decode, an unterminated final line as an item, dropped encoders writing nothing, line numbers in data-line errors, range-key pairs that tell bytewise from length-first and from base64 order, B hash keys in base64 order, N range keys differing beyond 17 digits, overlap between later files, last-endpoint validation, the per-file key-range check's refusals (a repeated key and an invalid stored N key among them) and two non-overlapping files each holding every key, through ExtenddbManifest::reader as well, a first item that is not the range's first and a file with no items through reader, reader_prehashed reading a wrong-digest file that reader refuses, the tracker refusing a boundary that does not ascend at the next file's first item with nothing encoded, the summary's itemCount as the sum when source_table.item_count differs, a non-UTF-8 sidecar, each etag on its file, the three documents before the manifest and the manifest document, the size sum refused at 2^63, the prehashed decoder's uncompressed count one too high, AttributeValue's ten descriptors and no other, the writer's tracker refusing an out-of-order or repeated key and agreeing with KeyOrder::compare for S, N and B keys, the prehashed decoder keeping every check but the digest, BackupDocuments' four documents agreeing on one prefix, the pinned enum values, N exponents at both ends of i32, a forged position in a data-line error, un-normalized and empty endpoints, resolve's key-order rules, required index members, negative source counts, the version gate's range and the gate's message naming the range 1 to CURRENT, unknown members injected at every level of two fixtures, the key-schema shape rules on an unordered manifest, to_json_bytes refusing what from_json_bytes refuses (non-finite timestamps included), endpoints repeating a type descriptor, the pinned member list of every manifest type, the two manifest constructors agreeing with the encoder and the manifest entry, the summary manifest's documented values, the key-order doc's normalization examples, size-sum overflow, md5 and sidecar shapes, identity rules, explicit nulls, bounded messages at field lengths from 81 bytes to 2 MB, a bounded Debug of FormatError::Json, an escaped ChecksumMismatch, and nested deserializer error texts. 32 hand mutants of these rules, and 39 more of the round-5 fixes, were each killed by the suite; 20 of the round-6 fixes were killed and one was equivalent (a dead branch in number.rs, replaced by three others, killed); 13 of the fixes for the maintainer's review of 2026-10-28 were killed, none equivalent; 15 of the round-7 fixes were killed, none equivalent; 18 of the round-8 fixes were killed at the test each targets, among them the four that survived round 8 (an AttributeValue descriptor alias, reader_prehashed hashing anyway, the summary's itemCount taken from source_table, build's size sum wrapping), none equivalent. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
robinnsc
added a commit
that referenced
this pull request
Oct 29, 2026
Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), eight review rounds and the maintainer's reviews against the companion PRs #393, #394 and #395, and a reconciliation against main abf8082 (#384 in its final form, the Cassandra backend #338, the PostgreSQL GSI build #401); then split so that the document is the overview of the series and the companions' own documentation is the record of what their code does (see the last paragraph). The revision log at the end lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). Indexes are reported in whatever build state they had and only an index being deleted is dropped; a GSI without its data table is left out with the attribute definitions only it used. On PostgreSQL's shared layout the open is one connection and one REPEATABLE READ transaction; on the separate layout the catalog row is held FOR SHARE until the data snapshot pins, and an index that becomes ACTIVE between the definition read and the pin is reported CREATING. PostgreSQL bounds open snapshots at half the pool and refuses more with LimitExceeded, so open snapshots leave half the pool to requests; from PR 6 on max_concurrent_jobs (default 4) bounds backups and restores together under that permit, and #384's restore slots go. It streams a table whose key columns predate COLLATE "C" unordered rather than sort it, and on both layouts DeleteTable commits the catalog removal only after its drops succeed, base table first, under one 1 s budget, taking its asynchronous path with the row kept when its probe finds a reader or a drop fails in a lock wait (lock timeout, deadlock victim or statement timeout), so no DROP waits behind a reader with the row gone; on the separate layout a task the request awaits commits the catalog and then the data transaction, and a data commit that fails after the catalog commit is retried on a fresh connection for a bounded time before the request returns (an orphan survives only a process death or a data database down past that bound, and the troubleshooting page says how to drop it); the control-plane poller probes held tables and retries each about once a second, at once on a snapshot's or an in-catalog backup's release, a drop DeleteTable deferred is due at once, and DeleteTable of a DELETING table returns its description on PostgreSQL and SQLite. On sqlite-memory the snapshot copies the table on the one connection inside the request and keeps a backup slot until its stream ends, so CreateBackup keeps working there once [backup.store] is set. MongoDB reads without the explicit snapshot level fail (#395's MongoDB module doc gives the server's error). - The snapshot is opened in the CreateBackup request (stricter than the service's two-minute window), and the backup id is minted from the instant the open returned, just after the pin, so the backup holds every write acknowledged before the request was received, and backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (#384's seed and key), the row itself on MongoDB (born through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read. A job deletes a prefix, or a restore target, only after its FAILED write matched a row. The reconciler's duties are listed once, in PR 8's section, and include the prefixes of DELETED rows, orphan restore targets, stale spools and, on Cassandra, active_restores entries whose target is not a CREATING restore target of the backup. Every duty that reads or writes a backup row acts only on rows of the configured store (store_kind, store_root and the store id); a CREATING or DELETING row of another store is reported and left alone. The PostgreSQL staleness test also selects a NULL heartbeat. While a maintenance claim is held the reconciler only finishes DELETING rows, resolves rows whose owner is dead and runs its FAILED-target duty, which never removes a target left for reason target_not_empty; that one is the operator's, removed with extenddb backup remove-target. The PostgreSQL claim connection sets TCP keepalives, so the loss of the owner's host frees its lock in about a minute. While the configured store is unhealthy the reconciler acts on no row of the store; its legacy-row duty, which touches no store, still runs. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty. backup sync is scoped to the configured store; extenddb-store.json records the store id and the owning catalog's catalog_id, and a root another catalog owns, rows without the file, rows with two store ids or a missing newest manifest mark the store unhealthy (store operations refused, the reconciler idle on the store's rows) instead of stopping startup, so an unmounted volume, a recreated bucket or two catalogs on one root are caught; backup sync --adopt-store hands a root to another catalog, and sqlite-memory's startup sync takes ownership of a root a sqlite-memory catalog owned the same way, so one catalog owns a root at a time there too; instances that share a catalog must share one store root; it never adopts a CREATING, DELETING, DELETED or FAILED prefix, nor one whose listing lacks a data file the manifest lists at its recorded size, and marks a row whose manifest head is gone DELETING only with --prune-missing. --move-store resolves the old root's CREATING and DELETING rows, compares every data file, refuses to carry a tombstone over a verified manifest and offers --forget-backup; --forget-store deletes the rows of a root. purge, remove-target, --move-store, --forget-store, --prune-missing and --adopt-store take a catalog maintenance claim that every job checks; max_concurrent_jobs is per instance. DeleteBackup and Restore of another store's row return a ValidationException that names no root. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. The store-id file is created with put_if_absent (If-None-Match on S3, O_EXCL and link on the filesystem). A backup is in use while a restore target registered from it is CREATING, as on main: DeleteBackup and a second restore return BackupInUseException. Purge deletes each row's prefix, never a whole account prefix. Every backup row is read by account and backup id and must match the request's account, table and ARN, and no row stores its key prefix. DeleteBackup's in-use check and DELETING write run under the backup row's lock, and a restore's registration re-checks AVAILABLE and in-use under the same lock, so the two serialize as on main (on Cassandra through main's active_restores lightweight transactions, now for new rows too); main's two PostgreSQL race tests stay for the legacy path and PR 6 adds the same two orderings over an off-host backup. - Restore returns CREATING from the request (TableAlreadyExists or TableInUse on a name race, as #384 returns), spools and verifies every data file before writing any item, and records progress atomically with the data on PostgreSQL and SQLite and by idempotent re-application of the one unrecorded file or batch on MongoDB and Cassandra; on MongoDB the job writes DONE and the backfill flag in one transaction and the worker's ACTIVE flip is conditioned on the flag, not on the job's generation. A key-ordered restore runs #394's per-file key-range check and an unordered one ends with a count of the target, which is Cassandra's only duplicate check. A PostgreSQL restore is resumed, or its target deleted, only after the stale window; the manifest and sidecar are spooled for a resume; a legacy restore with a dead owner is failed, not resumed. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as #384 and the service refuse it, and the job and the reconciler remove a failed target through the backend-internal remove_restore_target; the restore record is the table's child. Multi-part-key tables are restored (#384 refuses them; bulk_load keys rows with the backend's item-path functions, so the table is laid out as a natively created one). On PostgreSQL and SQLite bulk_load writes GSI, LSI and vector-index rows in the batch's transaction with the base rows, and the ACTIVE flip sets the table and every index ACTIVE in one catalog transaction; on Cassandra the index rows are created ACTIVE and the lightweight transaction flips the table alone. Table class is not restored, as on the service and #384; every override but VectorIndexOverride is applied, as #384 applies them. A second restore of a backup in use is refused before the permit and re-checked at registration. A restore stops on shutdown leaving its row RUNNING, download transport errors are retried for 15 minutes, and a takeover rewrites spool_host. Each data file is read through ExtenddbManifest::reader_prehashed, which applies the key-range check without hashing the spooled bytes again. - The engine binds a BackupCatalog trait (PR 6) with a JobClaim for the per-backend claim, heartbeat and conditional status writes; new rows live in backup_index (a new table or collection; Cassandra reuses its tables with store_kind set and adds backup_index_by_status); MongoDB's catalog gets the versioned migration step it lacks; PR 6's migration backfills in-flight #384 restores with heartbeat_at at the epoch so the first reconciler pass fails them; every pre-PR-6 server stops before migrate. Vector tables are backed up and restored on the off-host path, while the legacy create keeps main's refusal (one #384 engine test is rewritten in PR 9a and the wire test follows the configured path). source_table carries stream_specification and time_to_live for DescribeBackup. Both requests take the job permit last, after every refusal that needs no store access. The format's evolution rule is a version bump for any byte change, with v{N} fixtures beside v1. The trait gains listings for ListBackups, the store-id check and the root-matched commands, and backup_in_use. ListBackups accepts a table ARN in TableName from PR 6, as the service does. PR 6 adds the [backup] struct with the job keys. Row and restore-target methods take account-scoped keys, and the trait gains catalog_id and DeleteBackup's lock-holding start_backup_delete. - The manifest self-checksum is replaced by a sidecar over the stored bytes, accepted in its sha256sum line form only; backup_arn is the backup's identity and source_table must agree with it; N key values are normalized before key text or key order is computed, and an un-normalized key_range endpoint is refused; the version gate accepts 1 to the current version. The line cap is 32 MiB and is the floor of data_file_target_bytes; the format's other rules, the decoder's memory included, are #394's module docs, which section 1 points at. A reader refuses a manifest without stream_specification or time_to_live. The backup job validates every item with an item validator PR 8 adds to extenddb-core, which the restore binds, and fails the backup on a value #618 let in through any of the four write paths, a regression from main for tables that hold one; it validates the rest of the snapshot first and logs every refused item, and the FAILED reason gives the count and the first. - Store (#393): section 2 points at the crate's README and module docs and keeps the three store facts the design's rules rest on (put and delete_prefix are unordered, so a caller never puts under a prefix it is deleting; runtime shutdown sends no abort, so the server awaits each job's failure path; listing has no delimiter form) and names put_if_absent as PR 7's addition. Section 8 points at the README's IAM example and config.rs's transport and proxy rules. The revision log (items 21 to 23) records the store facts the companion rounds settled: the endpoint is refused with user information, a query, a fragment or an IPv6 zone id, from any source; the S3 client for an http:// endpoint is built with aws_smithy_http_client's Builder and no proxy, and the credential chain's clients get the same client when a credential endpoint setting, the IMDS endpoint setting or the container credentials URI is http://, so the chain's own http:// requests do not use the proxy; without such a setting the chain uses the SDK's default client and NO_PROXY must list 169.254.169.254 (fd00:ec2::254 in IPv6 mode) and 169.254.170.2; a denied AbortMultipartUpload in delete_prefix's sweep is logged and the deletion succeeds; a put dropped during the temporary's open leaves no temporary. - Configuration: the default store root is per backend (beside the database file on file-backed SQLite, where a defaulted root that fails validation is a warning, not a startup error); without [backup.store], CreateBackup on every backend keeps the legacy in-catalog create with a deprecation warning until PR 14 removes it (Cassandra always, having no SnapshotEngine), after which it is a ValidationException naming [backup.store], and kind = "disabled" is the opt-out; PR 14's migration refuses while legacy backup rows remain unless forced. A store transport failure at startup degrades the store component instead of stopping the server. The memory formula is in item text bytes, and PR 8 bounds each PostgreSQL and SQLite snapshot page by SNAPSHOT_FETCH_BYTES (64 MiB) in the page's query, PostgreSQL's cursor replaced by a keyset page under the same snapshot, so a page holds at most the budget plus one row; the PostgreSQL page returns its key columns, orders the outer query and sizes its row limit from the previous page's average row, starting at 16; the PostgreSQL pool rule is max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10), 18 at the default 4; the spool default is per backend and spool_host is host plus spool path. - Phase 1 is the PR series with acceptance criteria and GitHub numbers; migration numbers are reserved up front (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend and for DeleteBackup. The status line names the companion PRs' heads (aef17dd, 3f7107d, 069c4b1); the Phasing section refers to them by PR number. migrate-legacy re-reads and validates what it wrote and refuses, keeping the legacy rows, backups it cannot carry. Also adds the missing rows for designs 12 and 13 to docs/design/README.md; the row for 15 describes the document as the overview of the series and names the crates whose documentation holds the format, store and snapshot details. The series after the four open PRs is ten PRs: the reconciler ships with the CreateBackup job as PR 8 (a job without its reconciler leaves rows no one resolves), and the CI changes and the documentation ship together as PR 11, since both finalize after PR 9b. The revision log (item 24) records the renaming; its earlier items keep the old labels. The document is then split (revision log item 25): sections 1, 2 and 3 point at the companions' documentation (crates/core/src/backup_format and its testdata README, #394; crates/backup-store/README.md and its module docs, #393; crates/storage/src/snapshot.rs and the four snapshot_engine.rs, #395) and keep only the rules later PRs build on; sections 4 and 5 keep the invariants, the state machines and the per-operation contracts; the schema, the per-backend mechanics, the reconciler's duty list and the maintenance commands move into one section per later PR under Phasing (PR 1, 6, 7, 8, 9a, 9b, 10, 11, 13, 14), each with its goal, the invariants it honours, the interfaces it adds with their row keys, the transitions it owns and its acceptance list kept whole; section 10 is a table of which document each PR changes. The three companions gain the facts only this document stated (the filesystem list's whole-root Vec and the empty-prefix IAM example in #393's README; the fetch-size reason and the pre-#361 sort cost in #395's module docs) and a sentence each saying this overview states the rules their callers honour. 279,584 bytes become 187,722. Co-authored-by: Lee Hannigan <lhnng@amazon.com> Signed-off-by: Scott Robinson <robinnsc@amazon.com>
LeeroyHannigan
added a commit
to yesyayen/extenddb
that referenced
this pull request
Oct 30, 2026
…de (ExtendDB#396) Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs ExtendDB#384, ExtendDB#335, ExtendDB#336, ExtendDB#337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture (localhost; ratios only), eight review rounds and the maintainer's reviews against the companion PRs ExtendDB#393, ExtendDB#394 and ExtendDB#395, and a reconciliation against main abf8082 (ExtendDB#384 in its final form, the Cassandra backend ExtendDB#338, the PostgreSQL GSI build ExtendDB#401); then split so that the document is the overview of the series and the companions' own documentation is the record of what their code does (see the last paragraph). The revision log at the end lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). Indexes are reported in whatever build state they had and only an index being deleted is dropped; a GSI without its data table is left out with the attribute definitions only it used. On PostgreSQL's shared layout the open is one connection and one REPEATABLE READ transaction; on the separate layout the catalog row is held FOR SHARE until the data snapshot pins, and an index that becomes ACTIVE between the definition read and the pin is reported CREATING. PostgreSQL bounds open snapshots at half the pool and refuses more with LimitExceeded, so open snapshots leave half the pool to requests; from PR 6 on max_concurrent_jobs (default 4) bounds backups and restores together under that permit, and ExtendDB#384's restore slots go. It streams a table whose key columns predate COLLATE "C" unordered rather than sort it, and on both layouts DeleteTable commits the catalog removal only after its drops succeed, base table first, under one 1 s budget, taking its asynchronous path with the row kept when its probe finds a reader or a drop fails in a lock wait (lock timeout, deadlock victim or statement timeout), so no DROP waits behind a reader with the row gone; on the separate layout a task the request awaits commits the catalog and then the data transaction, and a data commit that fails after the catalog commit is retried on a fresh connection for a bounded time before the request returns (an orphan survives only a process death or a data database down past that bound, and the troubleshooting page says how to drop it); the control-plane poller probes held tables and retries each about once a second, at once on a snapshot's or an in-catalog backup's release, a drop DeleteTable deferred is due at once, and DeleteTable of a DELETING table returns its description on PostgreSQL and SQLite. On sqlite-memory the snapshot copies the table on the one connection inside the request and keeps a backup slot until its stream ends, so CreateBackup keeps working there once [backup.store] is set. MongoDB reads without the explicit snapshot level fail (ExtendDB#395's MongoDB module doc gives the server's error). - The snapshot is opened in the CreateBackup request (stricter than the service's two-minute window), and the backup id is minted from the instant the open returned, just after the pin, so the backup holds every write acknowledged before the request was received, and backup_creation_date_time, the catalog's created_at and the ARN carry one time and the ListBackups cursor is a function of the ARN. - Per-job claims with fencing replace the single-instance lease the draft assumed: an advisory lock on a dedicated catalog connection on PostgreSQL (ExtendDB#384's seed and key), the row itself on MongoDB (born through an upsert pipeline with $literal), an in-process slot on SQLite, a lightweight-transaction row on Cassandra (restore targets). Heartbeats are a timer on the database's clock with a deadline; any heartbeat that does not confirm the claim aborts the job. Every job status write is conditioned on the generation and on the row's state, and an ambiguous AVAILABLE update is re-read. A job deletes a prefix, or a restore target, only after its FAILED write matched a row. The reconciler's duties are listed once, in PR 8's section, and include the prefixes of DELETED rows, orphan restore targets, stale spools and, on Cassandra, active_restores entries whose target is not a CREATING restore target of the backup. Every duty that reads or writes a backup row acts only on rows of the configured store (store_kind, store_root and the store id); a CREATING or DELETING row of another store is reported and left alone. The PostgreSQL staleness test also selects a NULL heartbeat. While a maintenance claim is held the reconciler only finishes DELETING rows, resolves rows whose owner is dead and runs its FAILED-target duty, which never removes a target left for reason target_not_empty; that one is the operator's, removed with extenddb backup remove-target. The PostgreSQL claim connection sets TCP keepalives, so the loss of the owner's host frees its lock in about a minute. While the configured store is unhealthy the reconciler acts on no row of the store; its legacy-row duty, which touches no store, still runs. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty. backup sync is scoped to the configured store; extenddb-store.json records the store id and the owning catalog's catalog_id, and a root another catalog owns, rows without the file, rows with two store ids or a missing newest manifest mark the store unhealthy (store operations refused, the reconciler idle on the store's rows) instead of stopping startup, so an unmounted volume, a recreated bucket or two catalogs on one root are caught; backup sync --adopt-store hands a root to another catalog, and sqlite-memory's startup sync takes ownership of a root a sqlite-memory catalog owned the same way, so one catalog owns a root at a time there too; instances that share a catalog must share one store root; it never adopts a CREATING, DELETING, DELETED or FAILED prefix, nor one whose listing lacks a data file the manifest lists at its recorded size, and marks a row whose manifest head is gone DELETING only with --prune-missing. --move-store resolves the old root's CREATING and DELETING rows, compares every data file, refuses to carry a tombstone over a verified manifest and offers --forget-backup; --forget-store deletes the rows of a root. purge, remove-target, --move-store, --forget-store, --prune-missing and --adopt-store take a catalog maintenance claim that every job checks; max_concurrent_jobs is per instance. DeleteBackup and Restore of another store's row return a ValidationException that names no root. Restore and Describe never read the store without a catalog row; an ARN with no row goes to the legacy reader. The store-id file is created with put_if_absent (If-None-Match on S3, O_EXCL and link on the filesystem). A backup is in use while a restore target registered from it is CREATING, as on main: DeleteBackup and a second restore return BackupInUseException. Purge deletes each row's prefix, never a whole account prefix. Every backup row is read by account and backup id and must match the request's account, table and ARN, and no row stores its key prefix. DeleteBackup's in-use check and DELETING write run under the backup row's lock, and a restore's registration re-checks AVAILABLE and in-use under the same lock, so the two serialize as on main (on Cassandra through main's active_restores lightweight transactions, now for new rows too); main's two PostgreSQL race tests stay for the legacy path and PR 6 adds the same two orderings over an off-host backup. - Restore returns CREATING from the request (TableAlreadyExists or TableInUse on a name race, as ExtendDB#384 returns), spools and verifies every data file before writing any item, and records progress atomically with the data on PostgreSQL and SQLite and by idempotent re-application of the one unrecorded file or batch on MongoDB and Cassandra; on MongoDB the job writes DONE and the backfill flag in one transaction and the worker's ACTIVE flip is conditioned on the flag, not on the job's generation. A key-ordered restore runs ExtendDB#394's per-file key-range check and an unordered one ends with a count of the target, which is Cassandra's only duplicate check. A PostgreSQL restore is resumed, or its target deleted, only after the stale window; the manifest and sidecar are spooled for a resume; a legacy restore with a dead owner is failed, not resumed. The ACTIVE flip precedes the deletion of the PostgreSQL progress rows. DeleteTable on a restore target stays refused, as ExtendDB#384 and the service refuse it, and the job and the reconciler remove a failed target through the backend-internal remove_restore_target; the restore record is the table's child. Multi-part-key tables are restored (ExtendDB#384 refuses them; bulk_load keys rows with the backend's item-path functions, so the table is laid out as a natively created one). On PostgreSQL and SQLite bulk_load writes GSI, LSI and vector-index rows in the batch's transaction with the base rows, and the ACTIVE flip sets the table and every index ACTIVE in one catalog transaction; on Cassandra the index rows are created ACTIVE and the lightweight transaction flips the table alone. Table class is not restored, as on the service and ExtendDB#384; every override but VectorIndexOverride is applied, as ExtendDB#384 applies them. A second restore of a backup in use is refused before the permit and re-checked at registration. A restore stops on shutdown leaving its row RUNNING, download transport errors are retried for 15 minutes, and a takeover rewrites spool_host. Each data file is read through ExtenddbManifest::reader_prehashed, which applies the key-range check without hashing the spooled bytes again. - The engine binds a BackupCatalog trait (PR 6) with a JobClaim for the per-backend claim, heartbeat and conditional status writes; new rows live in backup_index (a new table or collection; Cassandra reuses its tables with store_kind set and adds backup_index_by_status); MongoDB's catalog gets the versioned migration step it lacks; PR 6's migration backfills in-flight ExtendDB#384 restores with heartbeat_at at the epoch so the first reconciler pass fails them; every pre-PR-6 server stops before migrate. Vector tables are backed up and restored on the off-host path, while the legacy create keeps main's refusal (one ExtendDB#384 engine test is rewritten in PR 9a and the wire test follows the configured path). source_table carries stream_specification and time_to_live for DescribeBackup. Both requests take the job permit last, after every refusal that needs no store access. The format's evolution rule is a version bump for any byte change, with v{N} fixtures beside v1. The trait gains listings for ListBackups, the store-id check and the root-matched commands, and backup_in_use. ListBackups accepts a table ARN in TableName from PR 6, as the service does. PR 6 adds the [backup] struct with the job keys. Row and restore-target methods take account-scoped keys, and the trait gains catalog_id and DeleteBackup's lock-holding start_backup_delete. - The manifest self-checksum is replaced by a sidecar over the stored bytes, accepted in its sha256sum line form only; backup_arn is the backup's identity and source_table must agree with it; N key values are normalized before key text or key order is computed, and an un-normalized key_range endpoint is refused; the version gate accepts 1 to the current version. The line cap is 32 MiB and is the floor of data_file_target_bytes; the format's other rules, the decoder's memory included, are ExtendDB#394's module docs, which section 1 points at. A reader refuses a manifest without stream_specification or time_to_live. The backup job validates every item with an item validator PR 8 adds to extenddb-core, which the restore binds, and fails the backup on a value ExtendDB#618 let in through any of the four write paths, a regression from main for tables that hold one; it validates the rest of the snapshot first and logs every refused item, and the FAILED reason gives the count and the first. - Store (ExtendDB#393): section 2 points at the crate's README and module docs and keeps the three store facts the design's rules rest on (put and delete_prefix are unordered, so a caller never puts under a prefix it is deleting; runtime shutdown sends no abort, so the server awaits each job's failure path; listing has no delimiter form) and names put_if_absent as PR 7's addition. Section 8 points at the README's IAM example and config.rs's transport and proxy rules. The revision log (items 21 to 23) records the store facts the companion rounds settled: the endpoint is refused with user information, a query, a fragment or an IPv6 zone id, from any source; the S3 client for an http:// endpoint is built with aws_smithy_http_client's Builder and no proxy, and the credential chain's clients get the same client when a credential endpoint setting, the IMDS endpoint setting or the container credentials URI is http://, so the chain's own http:// requests do not use the proxy; without such a setting the chain uses the SDK's default client and NO_PROXY must list 169.254.169.254 (fd00:ec2::254 in IPv6 mode) and 169.254.170.2; a denied AbortMultipartUpload in delete_prefix's sweep is logged and the deletion succeeds; a put dropped during the temporary's open leaves no temporary. - Configuration: the default store root is per backend (beside the database file on file-backed SQLite, where a defaulted root that fails validation is a warning, not a startup error); without [backup.store], CreateBackup on every backend keeps the legacy in-catalog create with a deprecation warning until PR 14 removes it (Cassandra always, having no SnapshotEngine), after which it is a ValidationException naming [backup.store], and kind = "disabled" is the opt-out; PR 14's migration refuses while legacy backup rows remain unless forced. A store transport failure at startup degrades the store component instead of stopping the server. The memory formula is in item text bytes, and PR 8 bounds each PostgreSQL and SQLite snapshot page by SNAPSHOT_FETCH_BYTES (64 MiB) in the page's query, PostgreSQL's cursor replaced by a keyset page under the same snapshot, so a page holds at most the budget plus one row; the PostgreSQL page returns its key columns, orders the outer query and sizes its row limit from the previous page's average row, starting at 16; the PostgreSQL pool rule is max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10), 18 at the default 4; the spool default is per backend and spool_host is host plus spool path. - Phase 1 is the PR series with acceptance criteria and GitHub numbers; migration numbers are reserved up front (Cassandra V006 to PR 1, V007 to PR 6); PR 5 is stacked on PR 4 and PR 6 depends on both. Kill points for the crash tests are enumerated per backend and for DeleteBackup. The status line names the companion PRs' heads (aef17dd, 3f7107d, 069c4b1); the Phasing section refers to them by PR number. migrate-legacy re-reads and validates what it wrote and refuses, keeping the legacy rows, backups it cannot carry. Also adds the missing rows for designs 12 and 13 to docs/design/README.md; the row for 15 describes the document as the overview of the series and names the crates whose documentation holds the format, store and snapshot details. The series after the four open PRs is ten PRs: the reconciler ships with the CreateBackup job as PR 8 (a job without its reconciler leaves rows no one resolves), and the CI changes and the documentation ship together as PR 11, since both finalize after PR 9b. The revision log (item 24) records the renaming; its earlier items keep the old labels. The document is then split (revision log item 25): sections 1, 2 and 3 point at the companions' documentation (crates/core/src/backup_format and its testdata README, ExtendDB#394; crates/backup-store/README.md and its module docs, ExtendDB#393; crates/storage/src/snapshot.rs and the four snapshot_engine.rs, ExtendDB#395) and keep only the rules later PRs build on; sections 4 and 5 keep the invariants, the state machines and the per-operation contracts; the schema, the per-backend mechanics, the reconciler's duty list and the maintenance commands move into one section per later PR under Phasing (PR 1, 6, 7, 8, 9a, 9b, 10, 11, 13, 14), each with its goal, the invariants it honours, the interfaces it adds with their row keys, the transitions it owns and its acceptance list kept whole; section 10 is a table of which document each PR changes. The three companions gain the facts only this document stated (the filesystem list's whole-root Vec and the empty-prefix IAM example in ExtendDB#393's README; the fetch-size reason and the pre-ExtendDB#361 sort cost in ExtendDB#395's module docs) and a sentence each saying this overview states the rules their callers honour. 279,584 bytes become 187,722. Signed-off-by: Scott Robinson <robinnsc@amazon.com> Co-authored-by: Lee Hannigan <lhnng@amazon.com>
LeeroyHannigan
added a commit
to yesyayen/extenddb
that referenced
this pull request
Oct 30, 2026
…hash, typed backup ARN (ExtendDB#394) Adds crates/core/src/backup_format, the on-disk format for off-host backups, built from ExtendDB#337's draft with the changes the design review asked for before v1 freezes. The module doc says that the design overview (docs/design/15-backup-restore.md, ExtendDB#396) states the rules the format's writers and readers honour, and that this module's docs and the testdata README are the record of the format itself. Data files: - DataFileEncoder<W: Write> hashes compressed bytes as they leave the compressor, so a writer streaming to a file or an upload holds the gzip window and one line, not the whole file. encode_data_file() remains as a convenience over it. encode_item_line_into is the one encoding of the {"Item": ...} line, into a buffer the caller keeps; encode_item_line, which the engine's ExportTableToPointInTime writer calls, and write_item, which reuses one buffer per encoder, both call it. write_item refuses a line over MAX_LINE_BYTES (32 MiB) with the encoder still usable, so a backup cannot contain a line the decoder would refuse at restore. The worst case at the default 400 KB item limit is a list of number sets, 67.6x (a 27.7 MB line, a 1.21x margin); operator-raised item limits above about 496 KB can exceed the cap. The doc carries the measured table and a test pins the worst case. A write error poisons the encoder (every later call is Encode); try_finish keeps the sink so an upload can be aborted, finish returns it. Only a successful finish writes the gzip trailer: dropping an encoder in any other state (fresh, open, poisoned) takes the sink out first, so flate2's Drop cannot complete a valid file of the items so far in a sink the caller abandoned. A sink that accepts zero bytes is WriteZero, not a spin; Interrupted is retried on every sink write and flush, and on the decoder's source reads. is_finished and is_poisoned are distinct. GZIP_LEVEL = 6 is pinned, and a test pins the output length and sha256 of a fixed input, so two runs of one build produce the same bytes. The writer emits one gzip member; the reader accepts concatenated members as one stream (a line may span members), which the reader contract states together with UTF-8 without BOM. - DataFileDecoder<R: Read> hashes the compressed bytes it consumes. The recorded compressed size and uncompressed size are enforced while reading (a source longer than its recorded compressed size fails with CompressedSizeMismatch after one byte past it); digests and the item count are checked at end of stream. Lines are capped at MAX_LINE_BYTES and read into one buffer the decoder reuses across items, as the encoder reuses its own; it keeps the longest line's capacity, at most the cap. Decoding one line peaks at about 76 bytes of heap per line byte for a list of empty strings, and at most about 128 for any shape measured (115 for the worst item, lists of one-member map chains; 127 for nested objects that are refused once built), whatever the nesting depth: about 4 GiB at the cap (measured without serde_json's preserve_order, the default binary; a build with the MongoDB backend enables it and measures about two thirds). DataFileDecoder's doc states the bound once. Any error is sticky (later next_item calls return Decode naming the earlier failure, never Ok(None)); the Iterator impl is fused on top so error-dropping consumers terminate. Line-level errors name the 1-based line in the file, and serde's column is labelled as within the line when serde's line is 1; a type descriptor or name ending in ` at line 3 column 4` sets serde's position, and such a message is shown as serde rendered it, with no column of its own. Items are checked for JSON shape only; the decoder does not apply the target's item limits, which the restore applies to each item before writing it. verified() keeps the slice path that checks digests before the first item, and then decodes without hashing again. A line ends in \n, except that the last line's may be absent (that line is an item, counted in uncompressed_bytes without a newline), and holds one object with the one member Item (an unknown member or a repeated top-level attribute is refused; a repeated type descriptor at the top of an attribute value is refused on this path and stays last-wins on the request path; nested maps keep JSON's last-wins); JSON whitespace, including a \r before the newline, is accepted as JSON accepts it. CompressedHasher is the incremental hasher a restore runs while spooling a download or over a file in place under an owner-only store root; DataFileEntry::decoder_prehashed and ExtenddbManifest::reader_prehashed then read that source without hashing it again (about 0.43 µs per 318-byte item, 9% of decoding) and keep every size, count and line check; the digests are not compared again, so a file replaced by one of another length is refused by the compressed-size check and a same-length replacement is not detected. AttributeValue deserializer (crates/core/src/types/attribute_value.rs, main's wire deserializer): the L and M arms now consume the serde_json::Value they own instead of cloning and re-deserializing it at each level. That copy had made an item PutItem accepts, 30 L wrappers around a list of 409,470 empty strings (31 containers, the maximum), cost 8.6 GB and 12.7 s to decode, and the request path, which shares the deserializer, the same. Results and every API error text are unchanged: a nested error is rendered once per level through serde_json::Error::custom, as main's per-level from_value rendered it, so a message ending in ` at line 0 column 0` loses one such suffix per level as before (a reviewer-run differential over 600k inputs, not committed; nested_error_text_is_unchanged pins thirteen nested texts, seven of that class at one to three levels with main's output, one of them with a suffix more than the levels strip). data_file_nested_decode_memory, which is not ignored, bounds a 32-level item's decode at 120 bytes per line byte. validation/number.rs (main's code, which every format parser and the request path reach) does its exponent arithmetic in i64. An exponent at either end of i32 combined with the digits no longer overflows: a debug build panicked on 10E2147483647, 49E2147483647 and .00E-2147483647, and a release build reported overflow as underflow and the reverse. An invalid N key's InvalidKey message carries the number validation's reason. Manifest: - Integrity is a sidecar, extenddb-manifest.sha256, in sha256sum format over the stored bytes (written by BackupDocuments::build, checked by verify_manifest_checksum), replacing an in-document digest over a re-serialized struct that no other implementation could reproduce. verify_manifest_checksum takes the sidecar as the bytes the store returned and accepts one shape only, the line the writer stores, "<64 lowercase hex> extenddb-manifest.json", with or without its final \n. Bytes that are not UTF-8 (refused with their own reason, the bytes not shown), a bare digest, which sha256sum -c refuses too, and anything else is InvalidManifest. The sidecar is written before the manifest, so the manifest's presence still marks a complete backup. - from_json_bytes requires a JSON document whose format_version is a u32, gates on it (check_format_version: 1 to CURRENT_FORMAT_VERSION; 0 and later versions are refused with a message naming the range this build reads) before any other member is interpreted, parses the document, then runs ExtenddbManifest::validate: the version gate, validate_source_table (item_count and table_size_bytes not negative; table_creation_date_time finite, since JSON has no NaN or infinity and serde_json writes them as null; the key schema has at least one HASH, HASH before RANGE, each key attribute named once and defined exactly once, on every manifest; the count of HASH and RANGE parts is left to the restore's CreateTable validation, and a multi-part key is backed up unordered), validate_identity (backup_arn is in canonical form and parses; backup_creation_date_time is exactly the f64 its id's instant gives; backup_name is 3-255 of the table-name alphabet; source_table.table_name is the ARN's table; source_table.table_arn, read with core::arn::parse, is a table ARN in the same partition for the same account and table) and validate_data_files (keys valid and unique, sha256 64 lowercase hex, md5 the standard base64 of 16 bytes, every file has items and non-zero sizes (entry errors are InvalidDataFileEntry, key errors InvalidDataFileKey), backup_size_bytes equals the sum without overflow, key_ordered manifests carry a key_range per file whose endpoints are key-only items of the schema's shape (S and B non-empty, N written in normalized form; the parser reads N endpoints as written, so 5.0 and 6E0 are refused) with first <= last and ranges ascending without overlap against every earlier file; unordered manifests carry none; an empty table is a manifest with no files). BackupDocuments::build runs validate too, so a writer cannot produce bytes the reader refuses. The document is parsed twice (once for the version, under 10% of the cost, once for the members); the doc states the cost per data file: about 6 to 12 µs and 1.6 KB retained per file with short keys. A key_range endpoint naming an attribute twice, or whose value repeats its type descriptor ({"S":"0","S":"A"}) or names two, is refused at parse time, as a data file line of that shape is. Non-overlapping ranges do not by themselves rule out a key in two files. ExtenddbManifest::reader (and reader_prehashed) runs KeyRangeCheck over each file's items when the manifest is key-ordered, without the caller constructing it, and its finish after the decoder's end-of-stream checks. The check refuses a first or last item that is not the endpoint, an item not strictly after the one before, an item past last, an item whose key KeyOrder::endpoint_key refuses, and an empty file; with the ranges, that rules duplicates out. KeyRangeCheck is a KeyRangeTracker plus the recorded range. The writer runs one tracker over the items of every file it encodes, so the endpoint check and the strict-ascent rule exist once: next_file closes a file with its KeyRange and keeps its last key, so a boundary that does not ascend (the next file starting with that key or one before it) is refused at the next file's first item, naming the previous file's last key, with nothing of that file encoded; finish closes the last file. With one tracker per file the same boundary is found only at build, after every file is stored. The tracker checks each key in place and keeps the first and latest keys, with an N range key parsed once; the check parses the recorded endpoints once. Per 318-byte item the check went from 1.07 µs and 11.5 allocations to 0.66 µs and 5 (one partition: 1.64 µs and 19.3 to 0.69 µs and 5.3). - Error messages carry a bounded slice of any untrusted field: at most 80 bytes of its escaped rendering, otherwise a 48-byte prefix and the total length. FormatError::Json holds a JsonError that bounds serde's text (which quotes the input) to 120 bytes in Display and Debug alike, and does not expose the inner error, as source() or otherwise; ChecksumMismatch prints its expected value with {:?}. - key_order.rs is the one definition of base-table key order: hash key by hash_key_text (S as is, N as held, B standard padded base64) compared as UTF-8 bytes, range key S as UTF-8 bytes, B as raw bytes and N by exact decimal value; the module doc states the N normalization rules (the exponent a signed 32-bit integer), the base64 alphabet and the string encoding another implementation must reproduce, and a test pins its examples. It states which writers store an invalid N key as written (BatchWriteItem and TransactWriteItems, ExtendDB#618; UpdateItem on an absent item and ImportTable run no number check on the key) and the rule for a key-ordered writer that meets one: the backup is refused with an error naming the key, through KeyRangeTracker::push, the one call per item, which checks the key as KeyOrder::endpoint_key does. N endpoints must be valid DynamoDB numbers written in normalized form; the parser refuses any other spelling, and items built in code are checked the same way. SnapshotDescriptor carries the three KIND_* constants the backends report, so the manifest and the storage layer share the strings; kind is a free string the reader does not validate and a restore must not branch on. KeyOrder::resolve refuses multi-part keys, duplicate attribute definitions, a range key listed before the hash key, and a range key with the hash key's name. The storage layer's pk_to_text delegates to hash_key_text. The module doc gives the per-backend ORDER BY recipe, including COLLATE "C" on Postgres text keys. - Optional source_table members serialize as explicit null. global_secondary_indexes, local_secondary_indexes, vector_indexes, stream_specification and time_to_live are required members (the last three may be null); stream_specification and time_to_live record what DescribeTable and DescribeTimeToLive reported, for DescribeBackup's SourceTableFeatureDetails, and are not restored. OnDemandThroughput writes null for an absent member, as on the wire, and the module doc records it; vector_indexes is three-state (null / [] / list) and holds the creation shape; sse_description, item_order, per-file md5 and uncompressed_bytes; the per-file compressed size feeds the decoder's expectations. - Golden fixtures testdata/extenddb-manifest/v1.json (every optional member set, a stream specification and a TTL description included), v1-minimal.json (every optional member absent, every list empty) and v1-unordered.json ("key_range":null files and the absent state of every nested optional member v1.json sets: an OnDemandThroughput with both limits null, an SseDescription without SSEType or KMSMasterKeyArn, a GSI without ProvisionedThroughput, a disabled stream specification without StreamViewType, a disabled TTL without AttributeName; and a vector index with SearchSchema and Projection, an LSI with INCLUDE), each with its .sha256, compared byte for byte, and a non-ignored test binds the writer's three constructors to the checked-in bytes. They are permanent read tests: the #[ignore]d writer (EXTENDDB_WRITE_GOLDEN) writes v{CURRENT} fixtures with create_new and refuses to overwrite an existing file. Any change to the bytes written for the fixtures' inputs is a version bump; a new member gets serde(default) and v{N} fixtures beside the v1 ones. A new member omitted when absent leaves the bytes unchanged, so nested_types_have_the_pinned_members pins the members every manifest type reads and writes, and fails it until the list changes. A new or renamed enum value that no fixture uses changes no byte either, so enum_values_are_pinned pins the values of every enum in the manifest: the derived ones through serde's variant list, and KeyType, ScalarAttributeType and DistanceFunction, whose readers are hand-written, through tables filled from a match with no wildcard, and AttributeValue, the value type of every key_range endpoint and data-file line, through its ten type descriptors the same way, plus a check that the descriptors its reader accepts among every string of one to four upper-case letters are exactly those ten. Unknown members are tolerated at every level. DynamoDB export manifests: ExportFileEntry.etag and s3SseAlgorithm are Option<String>; dataFileS3Key is the full store key, <prefix>/data/<file> (the service writes <s3Prefix>/AWSDynamoDB/<export id>/data/<file> for a full export to a prefix, measured; the documentation's full-export example omits the prefix; the module doc states both and what ExtendDB writes). dataFileS3Key and manifestFilesS3Key are built by one function, so an empty prefix gives data/<file> and manifest-files.json, without a leading /. Every leading and trailing / of the prefix is dropped once, so a// and /a/ give a/data/<file> and a/manifest-files.json, as a does; the store drops a leading /, one trailing / and refuses an empty component, so the two rules agree on every prefix the store accepts. DataFileEntry::from_checksums builds each entry from the encoder's result. BackupDocuments::build(header, data_files, store_key_prefix, bucket), with each data file paired with the ETag its put returned (None on a filesystem store), assembles the manifest, its sidecar, manifest-files.json and manifest-summary.json from one input. It derives backup_size_bytes, backup_creation_date_time and the key prefix (the store's key prefix with leading and trailing / dropped, then BackupArn::store_prefix) once and runs validate once. It builds each manifest-files.json line from the manifest's DataFileEntry, so the two manifests carry one md5 and one item_count, and the summary from the manifest (exportArn = the backup ARN, the three instants = the snapshot instant in RFC 3339 UTC with milliseconds, s3SseAlgorithm null, itemCount the sum; the module doc has the table). Its doc gives the write order: documents_before_manifest() returns manifest-files.json, manifest-summary.json and the sidecar, the job re-checks its claim, and manifest_document() is the manifest, written last; documents() is the four in that order. for_backup, from_data_file, write_files_manifest, to_json_bytes and manifest_checksum are private to backup_format. Layout: the backup id and ARN implementation moves from extenddb_storage::util::backup_id into extenddb_core::backup_format::layout (format_backup_id, is_backup_id, backup_id_epoch_millis, backup_creation_date_time, check_backup_arn, parse_backup_arn, BackupArnParts, BackupArnError, MAX_BACKUP_ARN_LEN), unchanged in behaviour (verified by a reviewer's differential run over 5.9M inputs); the field split is core::arn::fields and the table-name alphabet is validation::validate_table_name_chars, so authorization, CreateTable and this parser share them. The engine's validate_name_member and validate_index_name still spell the alphabet themselves. format_backup_arn is the one backup-ARN formatter (backup_arn() in storage, BackupArn, and the server's canonical resource ARN call it) and trim_backup_arn the one whitespace rule (check_backup_arn and the two authorization sites call it). It trims both ends as main did; the doc notes that the service resolves a trailing newline but refuses a leading space or tab. extenddb_storage::util re-exports the rest so the engine and the four backends keep their import path, and keeps new_backup_id (needs rand), backup_arn, and is_backup_resource_arn (the authorization predicate). BackupArn is the owned form the format uses: check_backup_arn plus the two store-prefix rules (table name at most 255 bytes and not only dots), with the ARN string and the store prefix; a test enumerates both divergences and the id-shape grid. The 255-byte bound is narrower than the engine's max_table_name_length when an operator raises it; the layout doc says so. Other copies of ARN facts in the server and storage crates (is_backup_resource_arn's own split, a partition constant) predate this PR and are not changed here. Tests: 175 in the module (174 run, 1 ignored fixture writer), three integration binaries in crates/core/tests (data_file_line_bomb: a 256 MiB single-line bomb stops within twice the cap's memory; data_file_nested_decode_memory: a 32-level item decodes within 120 bytes of heap per line byte; data_file_decode_peak: the ignored RSS measurement), a storage-sqlite test binding encode_orderable_number's byte order to KeyOrder::compare over a fixed pseudo-random corpus of DynamoDB numbers, so a change to either side fails, and a row-key contract test in storage pinning pk_to_text's literal output (standard base64, "+/+/") so a change to hash_key_text cannot silently re-key B rows. The ddb-export fixture is constructed from the documented output format, not captured from the service. Coverage includes the streaming end-of-stream digest check, every expectation a DataFileEntry carries, the uncompressed and compressed size caps firing early, wrong item counts in both directions, try_finish failure poisoning, Interrupted on every sink and source call, invalid UTF-8 as Decode, an unterminated final line as an item, dropped encoders writing nothing, line numbers in data-line errors, range-key pairs that tell bytewise from length-first and from base64 order, B hash keys in base64 order, N range keys differing beyond 17 digits, overlap between later files, last-endpoint validation, the per-file key-range check's refusals (a repeated key and an invalid stored N key among them) and two non-overlapping files each holding every key, through ExtenddbManifest::reader as well, a first item that is not the range's first and a file with no items through reader, reader_prehashed reading a wrong-digest file that reader refuses, the tracker refusing a boundary that does not ascend at the next file's first item with nothing encoded, the summary's itemCount as the sum when source_table.item_count differs, a non-UTF-8 sidecar, each etag on its file, the three documents before the manifest and the manifest document, the size sum refused at 2^63, the prehashed decoder's uncompressed count one too high, AttributeValue's ten descriptors and no other, the writer's tracker refusing an out-of-order or repeated key and agreeing with KeyOrder::compare for S, N and B keys, the prehashed decoder keeping every check but the digest, BackupDocuments' four documents agreeing on one prefix, the pinned enum values, N exponents at both ends of i32, a forged position in a data-line error, un-normalized and empty endpoints, resolve's key-order rules, required index members, negative source counts, the version gate's range and the gate's message naming the range 1 to CURRENT, unknown members injected at every level of two fixtures, the key-schema shape rules on an unordered manifest, to_json_bytes refusing what from_json_bytes refuses (non-finite timestamps included), endpoints repeating a type descriptor, the pinned member list of every manifest type, the two manifest constructors agreeing with the encoder and the manifest entry, the summary manifest's documented values, the key-order doc's normalization examples, size-sum overflow, md5 and sidecar shapes, identity rules, explicit nulls, bounded messages at field lengths from 81 bytes to 2 MB, a bounded Debug of FormatError::Json, an escaped ChecksumMismatch, and nested deserializer error texts. 32 hand mutants of these rules, and 39 more of the round-5 fixes, were each killed by the suite; 20 of the round-6 fixes were killed and one was equivalent (a dead branch in number.rs, replaced by three others, killed); 13 of the fixes for the maintainer's review of 2026-10-28 were killed, none equivalent; 15 of the round-7 fixes were killed, none equivalent; 18 of the round-8 fixes were killed at the test each targets, among them the four that survived round 8 (an AttributeValue descriptor alias, reader_prehashed hashing anyway, the summary's itemCount taken from source_table, build's size sum wrapping), none equivalent. Signed-off-by: Scott Robinson <robinnsc@amazon.com> Co-authored-by: Lee <leeroyhannigan@yahoo.ie>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
crates/core/src/backup_format/, the pure-type layer of the portable backup format: types, serde, and checksum helpers with no I/O and no storage dependency.ExtenddbManifest(extenddb-manifest.json): the backup commit record.format_version(current 1) gated by an explicitcheck_supported(); backup identity withbackup_creation_date_timeas f64 epoch seconds, the representationBackupDetailsuses; a snapshot descriptor (kind,marker); asource_tablesection built from the existing wire types (KeySchemaElement,AttributeDefinition,BillingMode,ProvisionedThroughput,OnDemandThroughput,SseDescription,GsiInput,LsiInput,VectorIndexDescription); per-data-file entries (key,item_count,sha256,bytes); andmanifest_sha256, computed over the manifest's compact JSON with the checksum field set to the empty string, with compute, seal, and verify helpers. Unknown fields are tolerated on read for forward compatibility; the version gate is the refusal point.ExportSummaryManifestandExportFileEntrymodelmanifest-summary.jsonandmanifest-files.jsonexactly as Amazon DynamoDB writes them forExportTableToPointInTimewithDYNAMODB_JSONoutput, per the documented export output format (cited in the module doc). That includes the explicitnullfors3SseKmsKeyId,exportTypeappearing only on incremental exports, and JSON-lines serialization for the files manifest. Serializer and parser both provided; the parser also accepts the documented incremental-export members it does not model.encode_data_filetakes an iterator ofItemand produces gzip bytes plus running item count, sha256 (hex) and md5 (base64) over the compressed bytes, and the uncompressed byte count.DataFileDecoderstreams items back and verifies whichever expectations it is given. The line encoding is the samejson!({"Item": item})wrapper the engine's export handler writes, so the two agree byte for byte.backup_prefix(account_id, table_name, backup_id),data_file_key(data/000001.json.gz), the fixed object names, andparse_backup_arnfor the ARN shape the engine's backup handlers already use, unchanged.Dependencies added to the workspace:
flate21.1.9,md-50.10.6,sha20.10.9, the versions already resolved inCargo.lock.Why
The backup and restore design replaces per-backend backup storage with one engine-owned portable format: the DynamoDB S3 export layout plus one ExtendDB manifest, so a backup restores into any backend, imports into Amazon DynamoDB with
ImportTable, and a DynamoDB export restores into ExtendDB. This PR is the format layer that the store abstraction and the engine jobs build on.Testing done
cargo fmt --all -- --checkclean.cargo clippy --all-targets -- -D warningsclean.cargo clippy -p extenddb-core -- -W clippy::pedantic: the newbackup_formatmodule is clean; the remaining warnings are pre-existing elsewhere in the crate.cargo test -p extenddb-core: 546 passed, 54 of them inbackup_format.cargo test --workspace: 1212 passed, 0 failed.cargo +1.88.0 check --workspace --locked: clean.manifest_sha256compute and verify with tamper detection;check_supportedrejecting format_version 2; encoder and decoder round trips over items covering every attribute type (S, N at 38 significant digits plus negative and exponent forms, B, SS, NS, BS, M nested to depth 32, L, NULL, BOOL) with checksum and count tamper detection; a decompression-bound test that feeds a 64 MiB single-line gzip bomb and asserts the typed cap error fires with peak resident memory bounded (measured from/proc/self/statuson Linux); multi-member gzip round trips; data-file key layout validation rejecting traversal, absolute, wrong-prefix, and extra-directory shapes; ARN parse and prefix round trips with malformed-ARN rejection; and golden-file tests against a checked-in export undercrates/core/testdata/ddb-export/asserting the parser reads it and the serializer reproduces themanifest-files.jsonlines byte for byte. The fixture is constructed to the documented output format (provenance in its README).Checklist
cargo test --workspace)cargo fmt --check)cargo clippy --all-targets -- -D warnings); the newbackup_formatmodule is additionallyclippy::pedantic-cleanStoragetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC: docs/design/15-backup-restore.md (landing in PR 1)
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.