Skip to content

feat(core): throttle errors carry the service's ThrottlingReasons - #532

Open
LeeroyHannigan wants to merge 2 commits into
mainfrom
feat/throttling-reasons
Open

LeeroyHannigan wants to merge 2 commits into
mainfrom
feat/throttling-reasons

Conversation

@LeeroyHannigan

Copy link
Copy Markdown
Collaborator

What

Amazon DynamoDB names the throttled resource and the limit on a throttle error: a reasons list with one {reason, resource} entry, where reason is ResourceType + OperationType + LimitType and resource is the table or index ARN. ExtendDB's three throttle errors carried a message string only, and the server's own throttle refusal sent one fixed provisioned-capacity body for every bucket and billing mode.

The three throttle variants of DynamoDbError (ProvisionedThroughputExceededException, ThrottlingException, RequestLimitExceeded) become struct variants with a message and a reasons list, built through provisioned_throughput_exceeded, throttling and request_limit_exceeded and read through throttling_reasons and is_throttle. The error serializer renders the list as ThrottlingReasons on the first and third and as throttlingReasons on ThrottlingException, and omits it when empty. ThrottlingException moves to the com.amazon.coral.availability# namespace. Message constants carry the measured texts for a table's own capacity, a hot key and the OnDemandThroughput caps, and the documented text for the account quota.

The throttle manager reports which bucket refused (the table, the partition key or the account aggregate) with the table's billing mode, and an on-demand table whose OnDemandThroughput caps are set is sized by the caps, per direction: a table capped on writes only keeps the default read ceiling and reports the caps limit on a write refusal alone. As with every bucket, it starts with one second of its rate and accumulates unused capacity up to 300 seconds of it. The server maps that cause onto the body the service sends:

Bucket Billing Error class Reasons field Reason Source
table provisioned ProvisionedThroughputExceededException ThrottlingReasons Table{Read|Write}ProvisionedThroughputExceeded measured, every read and write operation
partition key on-demand, write ThrottlingException throttlingReasons TableWriteKeyRangeThroughputExceeded measured, 1,455
partition key on-demand, read ThrottlingException none none measured, 5,615 of 5,615
table at its caps on-demand ThrottlingException throttlingReasons Table{Read|Write}MaxOnDemandThroughputExceeded measured, 100
partition key provisioned ProvisionedThroughputExceededException ThrottlingReasons Table{Read|Write}KeyRangeThroughputExceeded write form captured once on 2026-10-20 with this class and message; read form follows the documented grammar
table at the default ceiling on-demand ThrottlingException throttlingReasons Table{Read|Write}AccountLimitExceeded API Reference, not produced by the probes
account aggregate any RequestLimitExceeded ThrottlingReasons Table{Read|Write}AccountLimitExceeded API Reference, not produced by the probes

The resource is the table ARN DescribeTable returns. The account bucket can refuse before the engine has seen the name, so an ARN-form TableName naming a table in this account is reduced to the bare name first, and a name no table can have gets the class and message with no reasons list. The buckets are per table and per partition key; there is no per-index capacity, so the Index reason forms the service sends for an under-provisioned global secondary index are not modelled, and the differences doc says so.

Why

A client that reads ThrottlingReasons (the SDKs expose it on the parsed exception) learns which resource and which limit to act on; against ExtendDB that member was absent on every throttle, and the ThrottlingException namespace differed from the service's. The server also sent the provisioned-capacity message for an on-demand table's hot key and for the account aggregate, which name different limits on the service.

Measured against the service on 2026-10-20 (435 bodies, one Region) and 2026-10-24 (9,622 bodies in us-east-1 and eu-west-1, boto3 with retries off, raw bodies captured from the response hook): every read and write operation on a provisioned table (1,538 table reads, 648 index reads, 162 second-Region writes), hot-key writes (1,455) and reads (5,615) on an on-demand table, a table at its OnDemandThroughput caps (52 reads, 48 writes), and the index back-pressure case (104). No response carried x-amz-retry-after. Every reasons list present carried exactly one entry.

Testing done

In the core crate, the three classes keep their status and type name, carry their reasons through the accessor and answer is_throttle; only ThrottlingException moves namespace; the reason entry serializes with the two lower-case field names. The serializer tests cover the measured document for each class, the empty list omitted under both spellings, the caps body, the documented RequestLimitExceeded casing, the transaction cancel shape untouched. The throttle manager tests cover the cause and billing mode for the table, partition and account buckets; a capped table sized by its caps with absent and negative caps keeping the default ceiling; a half-capped table reporting its cap for its direction only; re-registration moving the mode and scaling the balance to the new rate. The server tests cover every arm of the cause mapping in both directions, with and without a resource; the throughput extraction for provisioned, on-demand and capped descriptions on both registration paths (the response body and the typed description the lazy path reads after a restart); seven in-process requests through handle_request against an in-memory SQLite engine (a provisioned write and read, an on-demand hot-key write and read, a capped table's Query and PutItem, a half-capped table's write and read, an account refusal with an ARN-form and an invalid name, and a served request). The SQLite storage crate becomes a dev-dependency of the server crate for those.

Negative controls, each on a clean committed tree and restored after: swapping the field casing fails the two ThrottlingException body tests; rendering the empty list fails the omission test; moving ThrottlingException back to the dynamodb namespace fails the namespace test; sending a reason on an on-demand hot-key read fails the mapping table and the wire test; ignoring the caps on either registration path fails that path's extraction test (and, for the response-body path, the capped wire test); dropping the billing mode from the partition cause fails the manager test; collapsing the per-direction caps to one flag fails the half-capped manager and wire tests; passing the raw TableName to the resource fails the account-arm wire test; keeping the balance unscaled on re-registration fails the scaling test.

cargo +1.97.0 fmt --all -- --check
cargo +1.97.0 clippy --all-targets -- -D warnings
cargo +1.97.0 test -p extenddb-core -p extenddb-server --lib   # 528 + 222 passed

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace --lib; the integration suites run in CI)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy --all-targets -- -D warnings)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a. The wire change reproduces measured service behaviour and adds no option; the Rust shape change is stated under Breaking changes and follows RFC-0002's release model (a 0.x minor).

Breaking changes

On the wire, throttle bodies gain the reasons member and ThrottlingException changes its __type prefix; both match the service, and botocore, the Java v2, Go v2, Rust and JavaScript v3 SDKs classify a throttle by the name after #, so no retry decision changes. For Rust code outside this workspace, the three throttle variants change from tuple to struct variants on the #[non_exhaustive] DynamoDbError (construct through provisioned_throughput_exceeded, throttling, request_limit_exceeded; read through throttling_reasons, is_throttle, message); ThrottleResult::ThrottledRead and ThrottledWrite carry a ThrottleCause (read through cause()); TableThroughput gains read_capped and write_capped (build through provisioned, on_demand, on_demand_capped). No in-tree crate outside the changed files names any of them. The change lands in the next 0.x minor under RFC-0002's release model.


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

Amazon DynamoDB names the throttled resource and the limit on a throttle
error: a reasons list with one {reason, resource} entry, where reason is
ResourceType + OperationType + LimitType and resource is the table or index
ARN. ExtendDB's three throttle errors carried a message string only, and
the server's own throttle refusal sent one fixed provisioned-capacity body
for every bucket and billing mode.

The three throttle variants of DynamoDbError (ProvisionedThroughputExceededException,
ThrottlingException, RequestLimitExceeded) become struct variants with a
message and a reasons list, built through provisioned_throughput_exceeded,
throttling and request_limit_exceeded, read through throttling_reasons and
is_throttle. The error serializer renders the list as ThrottlingReasons on
the first and third and as throttlingReasons on ThrottlingException, and
omits it when empty. ThrottlingException moves to the
com.amazon.coral.availability# namespace. The message constants carry the
measured texts for a table's own capacity, a hot key and the
OnDemandThroughput caps, and the documented text for the account quota.

The throttle manager reports which bucket refused (the table, the partition
key or the account aggregate) with the table's billing mode, and an
on-demand table whose OnDemandThroughput caps are set is sized by the caps,
per direction: a table capped on writes only keeps the default read ceiling
and reports the caps limit on a write refusal alone. The server maps the
cause onto the body the service sends: the provisioned class and
Table{Read|Write}ProvisionedThroughputExceeded for a provisioned table's own
capacity; ThrottlingException with TableWriteKeyRangeThroughputExceeded for
a hot-key write on an on-demand table and no reasons list for a hot-key
read; ThrottlingException with the caps message and
Table{Read|Write}MaxOnDemandThroughputExceeded at a cap. The resource is the
table ARN DescribeTable returns; an ARN-form TableName naming a table in
this account is reduced to the bare name first, and a name no table can
have gets no reasons list, since the account bucket can refuse before the
engine has seen the name. A hot key on a provisioned table was captured
once (the provisioned class and message with
TableWriteKeyRangeThroughputExceeded); its read form, the on-demand default
ceiling and the account aggregate follow the API Reference's reason grammar
and were not produced by the measurements. The code says which is which.

Measured against the service on 2026-10-20 (435 bodies, one Region) and
2026-10-24 (9,622 bodies in us-east-1 and eu-west-1): every read and write
operation on a provisioned table, hot-key writes and reads on an on-demand
table, a table at its OnDemandThroughput caps, and index back-pressure,
which the server does not model. No response carried x-amz-retry-after.

Compatibility. The wire change matches the service; the SDKs classify a
throttle by the name after the # in __type, so the namespace move changes
no retry. The Rust change is breaking for code outside this workspace that
constructs or destructures the three throttle variants, ThrottleResult's
throttled arms or a TableThroughput literal: the variants become struct
variants, the arms carry a ThrottleCause, and TableThroughput gains two
fields. Such code moves to the constructors (provisioned_throughput_exceeded,
throttling, request_limit_exceeded; TableThroughput::provisioned, on_demand,
on_demand_capped) and the accessors (throttling_reasons, is_throttle,
ThrottleResult::cause). No in-tree crate outside the changed files names
any of them. The change lands in the next 0.x minor.

Tests: the three classes keep their status and type name and carry their
reasons; the serializer produces the measured documents for each class and
omits the empty list; the throttle manager reports the cause and the
billing mode for the table, partition and account buckets, sizes a capped
table by its caps on both registration paths, reports a half-capped table's
cap for its direction only, and scales the balance when a table is
re-registered under a cap; every arm of the cause mapping in both
directions, with and without a resource; and seven in-process requests
through the handler against an in-memory engine (a provisioned write and
read, an on-demand hot-key write and read, a capped table's Query and
PutItem, a half-capped table's write and read, an account refusal with an
ARN-form and an invalid name, and a served request). The SQLite storage
crate becomes a dev-dependency of the server crate for those.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants