Repository navigation
feat(core): throttle errors carry the service's ThrottlingReasons - #532
Open
LeeroyHannigan wants to merge 2 commits into
Open
LeeroyHannigan wants to merge 2 commits into
LeeroyHannigan wants to merge 2 commits into
Conversation
Amazon DynamoDB names the throttled resource and the limit on a throttle
error: a reasons list with one {reason, resource} entry, where reason is
ResourceType + OperationType + LimitType and resource is the table or index
ARN. ExtendDB's three throttle errors carried a message string only, and
the server's own throttle refusal sent one fixed provisioned-capacity body
for every bucket and billing mode.
The three throttle variants of DynamoDbError (ProvisionedThroughputExceededException,
ThrottlingException, RequestLimitExceeded) become struct variants with a
message and a reasons list, built through provisioned_throughput_exceeded,
throttling and request_limit_exceeded, read through throttling_reasons and
is_throttle. The error serializer renders the list as ThrottlingReasons on
the first and third and as throttlingReasons on ThrottlingException, and
omits it when empty. ThrottlingException moves to the
com.amazon.coral.availability# namespace. The message constants carry the
measured texts for a table's own capacity, a hot key and the
OnDemandThroughput caps, and the documented text for the account quota.
The throttle manager reports which bucket refused (the table, the partition
key or the account aggregate) with the table's billing mode, and an
on-demand table whose OnDemandThroughput caps are set is sized by the caps,
per direction: a table capped on writes only keeps the default read ceiling
and reports the caps limit on a write refusal alone. The server maps the
cause onto the body the service sends: the provisioned class and
Table{Read|Write}ProvisionedThroughputExceeded for a provisioned table's own
capacity; ThrottlingException with TableWriteKeyRangeThroughputExceeded for
a hot-key write on an on-demand table and no reasons list for a hot-key
read; ThrottlingException with the caps message and
Table{Read|Write}MaxOnDemandThroughputExceeded at a cap. The resource is the
table ARN DescribeTable returns; an ARN-form TableName naming a table in
this account is reduced to the bare name first, and a name no table can
have gets no reasons list, since the account bucket can refuse before the
engine has seen the name. A hot key on a provisioned table was captured
once (the provisioned class and message with
TableWriteKeyRangeThroughputExceeded); its read form, the on-demand default
ceiling and the account aggregate follow the API Reference's reason grammar
and were not produced by the measurements. The code says which is which.
Measured against the service on 2026-10-20 (435 bodies, one Region) and
2026-10-24 (9,622 bodies in us-east-1 and eu-west-1): every read and write
operation on a provisioned table, hot-key writes and reads on an on-demand
table, a table at its OnDemandThroughput caps, and index back-pressure,
which the server does not model. No response carried x-amz-retry-after.
Compatibility. The wire change matches the service; the SDKs classify a
throttle by the name after the # in __type, so the namespace move changes
no retry. The Rust change is breaking for code outside this workspace that
constructs or destructures the three throttle variants, ThrottleResult's
throttled arms or a TableThroughput literal: the variants become struct
variants, the arms carry a ThrottleCause, and TableThroughput gains two
fields. Such code moves to the constructors (provisioned_throughput_exceeded,
throttling, request_limit_exceeded; TableThroughput::provisioned, on_demand,
on_demand_capped) and the accessors (throttling_reasons, is_throttle,
ThrottleResult::cause). No in-tree crate outside the changed files names
any of them. The change lands in the next 0.x minor.
Tests: the three classes keep their status and type name and carry their
reasons; the serializer produces the measured documents for each class and
omits the empty list; the throttle manager reports the cause and the
billing mode for the table, partition and account buckets, sizes a capped
table by its caps on both registration paths, reports a half-capped table's
cap for its direction only, and scales the balance when a table is
re-registered under a cap; every arm of the cause mapping in both
directions, with and without a resource; and seven in-process requests
through the handler against an in-memory engine (a provisioned write and
read, an on-demand hot-key write and read, a capped table's Query and
PutItem, a half-capped table's write and read, an account refusal with an
ARN-form and an invalid name, and a served request). The SQLite storage
crate becomes a dev-dependency of the server crate for those.
…index back-pressure gap
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
October 24, 2026 23:13
This was referenced Oct 25, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Amazon DynamoDB names the throttled resource and the limit on a throttle error: a reasons list with one
{reason, resource}entry, wherereasonisResourceType + OperationType + LimitTypeandresourceis the table or index ARN. ExtendDB's three throttle errors carried a message string only, and the server's own throttle refusal sent one fixed provisioned-capacity body for every bucket and billing mode.The three throttle variants of
DynamoDbError(ProvisionedThroughputExceededException,ThrottlingException,RequestLimitExceeded) become struct variants with amessageand areasonslist, built throughprovisioned_throughput_exceeded,throttlingandrequest_limit_exceededand read throughthrottling_reasonsandis_throttle. The error serializer renders the list asThrottlingReasonson the first and third and asthrottlingReasonsonThrottlingException, and omits it when empty.ThrottlingExceptionmoves to thecom.amazon.coral.availability#namespace. Message constants carry the measured texts for a table's own capacity, a hot key and theOnDemandThroughputcaps, and the documented text for the account quota.The throttle manager reports which bucket refused (the table, the partition key or the account aggregate) with the table's billing mode, and an on-demand table whose
OnDemandThroughputcaps are set is sized by the caps, per direction: a table capped on writes only keeps the default read ceiling and reports the caps limit on a write refusal alone. As with every bucket, it starts with one second of its rate and accumulates unused capacity up to 300 seconds of it. The server maps that cause onto the body the service sends:ProvisionedThroughputExceededExceptionThrottlingReasonsTable{Read|Write}ProvisionedThroughputExceededThrottlingExceptionthrottlingReasonsTableWriteKeyRangeThroughputExceededThrottlingExceptionThrottlingExceptionthrottlingReasonsTable{Read|Write}MaxOnDemandThroughputExceededProvisionedThroughputExceededExceptionThrottlingReasonsTable{Read|Write}KeyRangeThroughputExceededThrottlingExceptionthrottlingReasonsTable{Read|Write}AccountLimitExceededRequestLimitExceededThrottlingReasonsTable{Read|Write}AccountLimitExceededThe resource is the table ARN
DescribeTablereturns. The account bucket can refuse before the engine has seen the name, so an ARN-formTableNamenaming a table in this account is reduced to the bare name first, and a name no table can have gets the class and message with no reasons list. The buckets are per table and per partition key; there is no per-index capacity, so theIndexreason forms the service sends for an under-provisioned global secondary index are not modelled, and the differences doc says so.Why
A client that reads
ThrottlingReasons(the SDKs expose it on the parsed exception) learns which resource and which limit to act on; against ExtendDB that member was absent on every throttle, and theThrottlingExceptionnamespace differed from the service's. The server also sent the provisioned-capacity message for an on-demand table's hot key and for the account aggregate, which name different limits on the service.Measured against the service on 2026-10-20 (435 bodies, one Region) and 2026-10-24 (9,622 bodies in us-east-1 and eu-west-1, boto3 with retries off, raw bodies captured from the response hook): every read and write operation on a provisioned table (1,538 table reads, 648 index reads, 162 second-Region writes), hot-key writes (1,455) and reads (5,615) on an on-demand table, a table at its
OnDemandThroughputcaps (52 reads, 48 writes), and the index back-pressure case (104). No response carriedx-amz-retry-after. Every reasons list present carried exactly one entry.Testing done
In the core crate, the three classes keep their status and type name, carry their reasons through the accessor and answer
is_throttle; onlyThrottlingExceptionmoves namespace; the reason entry serializes with the two lower-case field names. The serializer tests cover the measured document for each class, the empty list omitted under both spellings, the caps body, the documentedRequestLimitExceededcasing, the transaction cancel shape untouched. The throttle manager tests cover the cause and billing mode for the table, partition and account buckets; a capped table sized by its caps with absent and negative caps keeping the default ceiling; a half-capped table reporting its cap for its direction only; re-registration moving the mode and scaling the balance to the new rate. The server tests cover every arm of the cause mapping in both directions, with and without a resource; the throughput extraction for provisioned, on-demand and capped descriptions on both registration paths (the response body and the typed description the lazy path reads after a restart); seven in-process requests throughhandle_requestagainst an in-memory SQLite engine (a provisioned write and read, an on-demand hot-key write and read, a capped table'sQueryandPutItem, a half-capped table's write and read, an account refusal with an ARN-form and an invalid name, and a served request). The SQLite storage crate becomes a dev-dependency of the server crate for those.Negative controls, each on a clean committed tree and restored after: swapping the field casing fails the two
ThrottlingExceptionbody tests; rendering the empty list fails the omission test; movingThrottlingExceptionback to the dynamodb namespace fails the namespace test; sending a reason on an on-demand hot-key read fails the mapping table and the wire test; ignoring the caps on either registration path fails that path's extraction test (and, for the response-body path, the capped wire test); dropping the billing mode from the partition cause fails the manager test; collapsing the per-direction caps to one flag fails the half-capped manager and wire tests; passing the rawTableNameto the resource fails the account-arm wire test; keeping the balance unscaled on re-registration fails the scaling test.Checklist
cargo test --workspace --lib; the integration suites run in CI)cargo fmt --check)cargo clippy --all-targets -- -D warnings)Storagetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC: n/a. The wire change reproduces measured service behaviour and adds no option; the Rust shape change is stated under Breaking changes and follows RFC-0002's release model (a 0.x minor).
Breaking changes
On the wire, throttle bodies gain the reasons member and
ThrottlingExceptionchanges its__typeprefix; both match the service, and botocore, the Java v2, Go v2, Rust and JavaScript v3 SDKs classify a throttle by the name after#, so no retry decision changes. For Rust code outside this workspace, the three throttle variants change from tuple to struct variants on the#[non_exhaustive]DynamoDbError(construct throughprovisioned_throughput_exceeded,throttling,request_limit_exceeded; read throughthrottling_reasons,is_throttle,message);ThrottleResult::ThrottledReadandThrottledWritecarry aThrottleCause(read throughcause());TableThroughputgainsread_cappedandwrite_capped(build throughprovisioned,on_demand,on_demand_capped). No in-tree crate outside the changed files names any of them. The change lands in the next 0.x minor under RFC-0002's release model.By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.