Skip to content

feat(storage): a StorageError::Service variant the engine passes through - #529

Open
LeeroyHannigan wants to merge 2 commits into
mainfrom
feat/storage-service-error
Open

LeeroyHannigan wants to merge 2 commits into
mainfrom
feat/storage-service-error

Conversation

@LeeroyHannigan

Copy link
Copy Markdown
Collaborator

What

StorageError gains a Service(DynamoDbError) variant, marked #[error(transparent)]. It carries a client error the backend has already classified. Nine engine functions map a StorageError to a client error. Six own a catch-all and gain a Service arm that returns the error with its variant and message unchanged and without the fault log line: the data path mapping in create_table.rs, the backup mapping in backup.rs, the UpdateTable mapping, the streams and TTL mappings, and sanitize_storage_error for the metadata and tagging paths. The other three (the condition-check variant of the data path mapping, the create-backup and restore mappings) delegate their catch-all to one of the six. The server's authorization read delegates to the data path mapping.

The engine keeps one rule for itself. A ConditionalCheckFailedException carries its item to the client only when the request set ReturnValuesOnConditionCheckFailure to ALL_OLD: the condition-check mapping applies that to a backend's error as it does to the engine's own ConditionFailed, and the generic data path mapping, which has no such value to read, discards the item as it does for ConditionFailed. The message is kept.

The redaction rule for every other variant is unchanged: an Internal error is logged at error level and reaches the client as the fixed InternalServerError text.

The storage design lists the variant beside the redaction rule, the security model's Error Message Security section names it as the one path that carries backend-chosen text to a client, the backend author manual's DataEngine section says when to return it, RFC-0003 section 9.2 adds the obligation to the error fidelity criteria, RFC-0002 states that variants may be added to StorageError in a minor release, and ADR-0023 records the decision and the options not taken.

Why

A backend that can classify its own failure into the service's error vocabulary (a hosted database reporting a documented capacity or unavailability condition) had no way to return that error with its own message. Every unrecognised StorageError reaches the client as the fixed Internal server error string after an error-level log line, so the client learns nothing about what to do and the operator's log counts a documented condition among storage faults. Transient is also redacted, and the in-tree queue workers read it as the retry signal, so it is not a substitute.

The obligation that comes with the variant is stated in the doc comment, the design, both manuals and RFC-0003: the message inside crosses the wire verbatim, so it must be service-shaped text with no driver output, host names, connection strings or credentials, and a failure the backend cannot classify stays Internal. The status code and the body shape still come from the DynamoDbError variant through the one serializer, so a backend cannot produce a status or a shape outside the service's catalog through this variant. No in-tree backend returns Service yet.

Testing done

A Service pass-through test at every one of the nine mappings (InternalServerError, ServiceUnavailable, a capacity error, LimitExceededException, ExpiredIteratorException, BackupNotFoundException through the restore mapping). At each of the six with a catch-all, a control checks Internal("driver text") still becomes the fixed string. The item rule has a test per ReturnValuesOnConditionCheckFailure value and one for the generic path, each with a one-attribute item. A subscriber-based test (crates/engine/src/service_error_log_tests.rs, tracing-subscriber as a dev-dependency) counts error-level log events at every mapping site that owns a catch-all: zero for a Service error, one for an Internal error.

Negative controls, each on a clean committed tree and restored after: removing the Service arm at each of the six sites fails that site's tests (in create_table.rs the removal does not compile, since the match is exhaustive); removing either conditional check arm fails its test with the item reported as reaching the client; inserting the fault log line before the Service return in sanitize_storage_error, or inside the Service arm in ttl.rs, fails the log test naming that site.

cargo +1.97.0 fmt --all -- --check
cargo +1.97.0 clippy --all-targets -- -D warnings
cargo +1.97.0 test -p extenddb-engine -p extenddb-storage --lib   # 210 + 125 passed
cargo +1.97.0 test --workspace --lib                               # 1,712 passed, 0 failed

The branch adds 32 tests and removes or renames none (test name lists compared against main).

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace --lib; the integration suites run in CI)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy --all-targets -- -D warnings)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: docs/adr/0023-backend-classified-client-error.md; RFC-0003 section 9.2 and RFC-0002 (release model) updated in this PR.

Breaking changes

None for the wire protocol or any in-tree backend. StorageError gains a variant: a backend crate outside this workspace that matches on it exhaustively needs a catch-all arm, which RFC-0002 now states as the policy for minor releases.


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

A backend that can classify its own failure into the service's error
vocabulary (a hosted database reporting a documented unavailability or
capacity condition) had no way to return that error with its own message:
every unrecognised StorageError variant reaches the client as the fixed
"Internal server error" string after an error-level log line.

StorageError::Service(DynamoDbError) carries a fully formed client error.
Nine engine functions map a StorageError to a client error. Six own a
catch-all and gain a Service arm that returns the error with its variant
and message unchanged and without the fault log line: the data path mapping
in create_table.rs, the backup mapping in backup.rs, the UpdateTable
mapping, the streams and TTL mappings, and sanitize_storage_error for the
metadata and tagging paths. The other three (the condition-check variant of
the data path mapping, the create-backup and restore mappings) delegate
their catch-all to one of the six. The server's authorization read
delegates to the data path mapping.

The engine keeps one rule for itself. A ConditionalCheckFailedException
carries its item to the client only when the request set
ReturnValuesOnConditionCheckFailure to ALL_OLD, which the condition-check
mapping applies to a backend's error as it does to the engine's own
ConditionFailed; the generic data path mapping, which has no such value to
read, discards the item as it does for ConditionFailed. The message is kept.

Tests: a Service pass-through test at every one of the nine mappings; at
each of the six with a catch-all, a control that Internal("driver text")
still becomes the fixed string, and a subscriber-based test that counts
error-level log events (none for a Service error, one for an Internal
error). tracing-subscriber becomes a dev-dependency of the engine crate for
that test. The condition-check item rule has a test per
ReturnValuesOnConditionCheckFailure value and one for the generic path.

The doc comment states the backend's obligation: the message crosses the
wire verbatim, so it must be service-shaped text with no driver output, host
names, connection strings or credentials; an unclassified failure stays
Internal, and Transient remains the queue workers' retry signal.
…ls, the RFCs and an ADR

The storage design lists the variant beside the Internal redaction rule and
states the backend's obligation for the message. The security model's error
message section names it as the one path that carries backend-chosen text
to a client, under the same wording rule as the engine's own messages. The
backend author manual's DataEngine section states when to return it and
when a failure stays Internal. RFC-0003 section 9.2 adds the obligation to
the error fidelity criteria. RFC-0002 states that variants may be added to
StorageError in a minor release, so a backend crate matching on it
exhaustively carries a catch-all arm. ADR-0023 records the decision, the
options not taken and the consequences.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant