Repository navigation
feat(storage): a StorageError::Service variant the engine passes through - #529
Open
LeeroyHannigan wants to merge 2 commits into
Open
LeeroyHannigan wants to merge 2 commits into
LeeroyHannigan wants to merge 2 commits into
Conversation
A backend that can classify its own failure into the service's error
vocabulary (a hosted database reporting a documented unavailability or
capacity condition) had no way to return that error with its own message:
every unrecognised StorageError variant reaches the client as the fixed
"Internal server error" string after an error-level log line.
StorageError::Service(DynamoDbError) carries a fully formed client error.
Nine engine functions map a StorageError to a client error. Six own a
catch-all and gain a Service arm that returns the error with its variant
and message unchanged and without the fault log line: the data path mapping
in create_table.rs, the backup mapping in backup.rs, the UpdateTable
mapping, the streams and TTL mappings, and sanitize_storage_error for the
metadata and tagging paths. The other three (the condition-check variant of
the data path mapping, the create-backup and restore mappings) delegate
their catch-all to one of the six. The server's authorization read
delegates to the data path mapping.
The engine keeps one rule for itself. A ConditionalCheckFailedException
carries its item to the client only when the request set
ReturnValuesOnConditionCheckFailure to ALL_OLD, which the condition-check
mapping applies to a backend's error as it does to the engine's own
ConditionFailed; the generic data path mapping, which has no such value to
read, discards the item as it does for ConditionFailed. The message is kept.
Tests: a Service pass-through test at every one of the nine mappings; at
each of the six with a catch-all, a control that Internal("driver text")
still becomes the fixed string, and a subscriber-based test that counts
error-level log events (none for a Service error, one for an Internal
error). tracing-subscriber becomes a dev-dependency of the engine crate for
that test. The condition-check item rule has a test per
ReturnValuesOnConditionCheckFailure value and one for the generic path.
The doc comment states the backend's obligation: the message crosses the
wire verbatim, so it must be service-shaped text with no driver output, host
names, connection strings or credentials; an unclassified failure stays
Internal, and Transient remains the queue workers' retry signal.
…ls, the RFCs and an ADR The storage design lists the variant beside the Internal redaction rule and states the backend's obligation for the message. The security model's error message section names it as the one path that carries backend-chosen text to a client, under the same wording rule as the engine's own messages. The backend author manual's DataEngine section states when to return it and when a failure stays Internal. RFC-0003 section 9.2 adds the obligation to the error fidelity criteria. RFC-0002 states that variants may be added to StorageError in a minor release, so a backend crate matching on it exhaustively carries a catch-all arm. ADR-0023 records the decision, the options not taken and the consequences.
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
October 24, 2026 22:05
8 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
StorageErrorgains aService(DynamoDbError)variant, marked#[error(transparent)]. It carries a client error the backend has already classified. Nine engine functions map aStorageErrorto a client error. Six own a catch-all and gain aServicearm that returns the error with its variant and message unchanged and without the fault log line: the data path mapping increate_table.rs, the backup mapping inbackup.rs, theUpdateTablemapping, the streams and TTL mappings, andsanitize_storage_errorfor the metadata and tagging paths. The other three (the condition-check variant of the data path mapping, the create-backup and restore mappings) delegate their catch-all to one of the six. The server's authorization read delegates to the data path mapping.The engine keeps one rule for itself. A
ConditionalCheckFailedExceptioncarries its item to the client only when the request setReturnValuesOnConditionCheckFailuretoALL_OLD: the condition-check mapping applies that to a backend's error as it does to the engine's ownConditionFailed, and the generic data path mapping, which has no such value to read, discards the item as it does forConditionFailed. The message is kept.The redaction rule for every other variant is unchanged: an
Internalerror is logged at error level and reaches the client as the fixedInternalServerErrortext.The storage design lists the variant beside the redaction rule, the security model's Error Message Security section names it as the one path that carries backend-chosen text to a client, the backend author manual's
DataEnginesection says when to return it, RFC-0003 section 9.2 adds the obligation to the error fidelity criteria, RFC-0002 states that variants may be added toStorageErrorin a minor release, and ADR-0023 records the decision and the options not taken.Why
A backend that can classify its own failure into the service's error vocabulary (a hosted database reporting a documented capacity or unavailability condition) had no way to return that error with its own message. Every unrecognised
StorageErrorreaches the client as the fixedInternal server errorstring after an error-level log line, so the client learns nothing about what to do and the operator's log counts a documented condition among storage faults.Transientis also redacted, and the in-tree queue workers read it as the retry signal, so it is not a substitute.The obligation that comes with the variant is stated in the doc comment, the design, both manuals and RFC-0003: the message inside crosses the wire verbatim, so it must be service-shaped text with no driver output, host names, connection strings or credentials, and a failure the backend cannot classify stays
Internal. The status code and the body shape still come from theDynamoDbErrorvariant through the one serializer, so a backend cannot produce a status or a shape outside the service's catalog through this variant. No in-tree backend returnsServiceyet.Testing done
A
Servicepass-through test at every one of the nine mappings (InternalServerError,ServiceUnavailable, a capacity error,LimitExceededException,ExpiredIteratorException,BackupNotFoundExceptionthrough the restore mapping). At each of the six with a catch-all, a control checksInternal("driver text")still becomes the fixed string. The item rule has a test perReturnValuesOnConditionCheckFailurevalue and one for the generic path, each with a one-attribute item. A subscriber-based test (crates/engine/src/service_error_log_tests.rs,tracing-subscriberas a dev-dependency) counts error-level log events at every mapping site that owns a catch-all: zero for aServiceerror, one for anInternalerror.Negative controls, each on a clean committed tree and restored after: removing the
Servicearm at each of the six sites fails that site's tests (increate_table.rsthe removal does not compile, since the match is exhaustive); removing either conditional check arm fails its test with the item reported as reaching the client; inserting the fault log line before theServicereturn insanitize_storage_error, or inside theServicearm inttl.rs, fails the log test naming that site.The branch adds 32 tests and removes or renames none (test name lists compared against
main).Checklist
cargo test --workspace --lib; the integration suites run in CI)cargo fmt --check)cargo clippy --all-targets -- -D warnings)Storagetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC:
docs/adr/0023-backend-classified-client-error.md; RFC-0003 section 9.2 and RFC-0002 (release model) updated in this PR.Breaking changes
None for the wire protocol or any in-tree backend.
StorageErrorgains a variant: a backend crate outside this workspace that matches on it exhaustively needs a catch-all arm, which RFC-0002 now states as the policy for minor releases.By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.