Skip to content

fix(limits): attribute-name size limit on every write path - #433

Open
LeeroyHannigan wants to merge 2 commits into
mainfrom
fix/update-attribute-name-limit
Open

LeeroyHannigan wants to merge 2 commits into
mainfrom
fix/update-attribute-name-limit

Conversation

@LeeroyHannigan

Copy link
Copy Markdown
Collaborator

Summary

Fixes #365. An attribute name is at most 65,535 bytes of UTF-8, at every depth. ExtendDB enforced the limit only on the put paths, after the table lookup, with its own wording; UpdateItem and the Update member of TransactWriteItems stored a 65,536-byte name with HTTP 200, and a long name that arrived only through a condition's ExpressionAttributeNames was accepted on PutItem, DeleteItem and every transaction member.

What the service does

Measured on Amazon DynamoDB (us-east-1) on 2026-10-21 at 65,535 and 65,536 bytes: 35 wire cases at 23:40 UTC and a 91-row probe at 23:52 UTC.

  • A 65,536-byte name is refused with a ValidationException on PutItem, UpdateItem, DeleteItem, TransactWriteItems and BatchWriteItem; 65,535 bytes is accepted. The limit counts bytes: 32,768 two-byte characters are refused, 32,767 of them plus one ASCII character are accepted.
  • The wording follows the request form. PutItem (name in the item), UpdateItem (map key in a value, AttributeUpdates), and PutItem or DeleteItem (map key in a condition value): 1 validation error detected: Attribute name is too large, must be less than 65536 bytes. PutItem, UpdateItem and DeleteItem for an ExpressionAttributeNames value: the same message followed by for key #n. Any TransactWriteItems member for an ExpressionAttributeNames value: ExpressionAttributeNames contains invalid value: Attribute name is too large, must be less than 65536 bytes for key #n; for a map key inside an ExpressionAttributeValues value: ExpressionAttributeValues contains invalid value: ... for key :v. A TransactWriteItems Put or a BatchWriteItem PutRequest for a name in the item: Attribute name is too large, must be less than 65536 bytes. A transaction fails as a whole with no CancellationReasons.
  • Every ExpressionAttributeNames value is checked, including one no expression uses.
  • Position. The name check runs ahead of the table lookup on every request. In a transaction it runs for every member before any member's table lookup: [Put on a missing table, Update with a long #n] answers the Update member's name refusal. In a batch it runs before the table lookup and the key check: a PutRequest item with a long name and no key answers the name refusal. UpdateItem and a transacted Update report the name ahead of an UpdateExpression syntax error, a failing condition and a secondary-index key type mismatch.
  • Order with two long names at once. UpdateItem reports the ExpressionAttributeNames placeholder before a map key in a value; PutItem reports the item before the ExpressionAttributeNames placeholder. A transaction member reports a map key in a value first, then the ExpressionAttributeNames placeholder, then (Put) the item.

Changes

  • crates/core/src/validation/mod.rs: AttributeNameRequestForm { SingleItem, TransactionMember, BatchRequest } selects the wording and the order of the two expression maps. validate_attribute_name_sizes takes it; new validate_expression_attribute_name_sizes (names half and values half, ordered by form) and validate_attribute_updates_name_sizes. validate_put_item no longer checks names (the handler does, earlier). Unit tests for every form and both orders.
  • crates/engine/src/put_item.rs: item names, then the expression maps, right after deserialization and before the table lookup.
  • crates/engine/src/update_item.rs: expression maps and AttributeUpdates checked after validate_table_name, before the table lookup.
  • crates/engine/src/delete_item.rs: expression maps checked after validate_table_name, before the table lookup.
  • crates/engine/src/transact_write_items.rs: validate_member_attribute_name_sizes runs for every member in request order before prepare_write_op (expression maps, then a Put member's item); the per-arm checks are removed.
  • crates/engine/src/batch_write_item.rs: every PutRequest item checked in a pass before the per-table loop; the per-request check is removed.
  • crates/engine/src/import_export.rs: passes SingleItem; the error is only logged.
  • docs/dynamodb-limits.md: Attribute name size row Partial to Enforced, with the wording, position and order per form; summary counts updated.
  • tests/test_attribute_name_limit_on_update.py: 49 dual-target cases.

No storage crate changes: every path reaches the checks through the engine, so Cassandra receives them too.

Verification

Check Service SQLite PostgreSQL MongoDB Main (ec81e9d)
tests/test_attribute_name_limit_on_update.py, 49 cases 49 passed (01:05 UTC on 2026-10-22) 49 passed 49 passed 49 passed 32 of the first 35 fail on the base of the earlier rounds; 11 of the 14 later cases fail on the branch before the fold
365-probe.py, 91 rows, compared on status, code, message, stored item reference 0 differ 1 differs: O05, the control row for an index-key type mismatch alone on UpdateItem, which PostgreSQL on main does not refuse (#416 covers it; not touched here) 0 differ
16 related wire files (item operations, conditional writes, transactions, batch, expression validation, precedence, wording parity) 353 passed 353 passed 353 passed
cargo test -p extenddb-core 524 passed
cargo test -p extenddb-engine 103 passed

Negative controls

Each mutation applied to the committed tree with an anchor occurring once, run, then restored with git checkout; the tree was clean after each.

Mutation Result
core: SingleItem arm of validate_expression_attribute_name_sizes reverted to names-then-values for every form unit test transaction_expression_values_are_checked_before_names fails; wire TestTransactUpdate::test_values_map_reported_before_the_names_map fails
core: message text reverted to the old wording; `TransactionMember BatchRequest` arm given the prefix; map name dropped from the transaction form; values branch forced to the bare sentence
put_item.rs: expression-map check removed TestPutItem::test_placeholder_name_only_in_the_condition fails (DID NOT RAISE)
put_item.rs: item check moved after the table lookup TestPutItem::test_reported_ahead_of_a_missing_table fails
delete_item.rs: expression-map check removed TestDeleteItem::test_placeholder_name_only_in_the_condition fails (DID NOT RAISE)
update_item.rs: expression-map check removed; AttributeUpdates check removed 11 and 2 cases fail (DID NOT RAISE)
transact_write_items.rs: expression-map check removed from the member pass 19 cases fail: every Update, Delete, ConditionCheck and Put-condition refusal
transact_write_items.rs: member pass moved after the prepare_write_op loop 3 cases fail: both missing-table orderings and the Put missing-table case (ResourceNotFoundException)
transact_write_items.rs: Put item check removed from the member pass TestTransactPut::test_name_over_limit[2] and test_reported_ahead_of_a_missing_table fail
transact_write_items.rs: Put item check swapped ahead of the expression maps TestTransactPut::test_names_map_reported_before_the_item fails (bare sentence instead of ... for key #n)
transact_write_items.rs: TransactionMember changed to SingleItem in the Put arm; in the Update arm 2 and 11 cases fail on the message
batch_write_item.rs: pre-pass removed; pre-pass moved back into the per-request loop 4 cases fail (2 DID NOT RAISE, missing table, missing key); 2 cases fail (missing table, missing key)
batch_write_item.rs: BatchRequest changed to SingleItem 2 cases fail on the message

Gates

  • cargo fmt --all -- --check: clean.
  • cargo +1.97.0 clippy -p extenddb-core -p extenddb-engine --all-targets -- -D warnings: clean.
  • cargo +1.88.0 build -p extenddb-core -p extenddb-engine: clean.
  • cargo test -p extenddb-core: 524 passed. cargo test -p extenddb-engine: 103 passed.
  • 16 related wire files on SQLite, PostgreSQL and MongoDB: 353 passed on each.
  • No dependency added or removed; notices files unchanged.

Not in this change

…-form wording

An attribute name is at most 65,535 bytes of UTF-8, at every depth: a
top-level name, a map key inside any value, and a name supplied through
ExpressionAttributeNames. Measured against Amazon DynamoDB (us-east-1,
2026-10-21) at 65,535 and 65,536 bytes, the service refuses a longer name on
PutItem, UpdateItem, DeleteItem, TransactWriteItems and BatchWriteItem with a
ValidationException whose wording follows the request form:

- PutItem for a name in the item, and UpdateItem for a map key inside an
  ExpressionAttributeValues value or an AttributeUpdates name or value:
  "1 validation error detected: Attribute name is too large, must be less than
  65536 bytes". PutItem and DeleteItem use the same message for a map key
  inside a condition value.
- PutItem, UpdateItem and DeleteItem for an ExpressionAttributeNames value:
  the same message followed by " for key #n", naming the placeholder.
- Any member of TransactWriteItems (Put, Update, Delete, ConditionCheck) for
  an ExpressionAttributeNames value: "ExpressionAttributeNames contains
  invalid value: Attribute name is too large, must be less than 65536 bytes
  for key #n"; for a map key inside an ExpressionAttributeValues value:
  "ExpressionAttributeValues contains invalid value: Attribute name is too
  large, must be less than 65536 bytes for key :v". The whole request fails
  with no CancellationReasons.
- The Put member of TransactWriteItems and a BatchWriteItem PutRequest, for a
  name in the item: "Attribute name is too large, must be less than 65536
  bytes".

The limit counts bytes: 32,768 two-byte characters are refused, and 32,767 of
them plus one ASCII character are accepted. The service checks every
ExpressionAttributeNames value, including one no expression uses. The name
check runs ahead of the table lookup on every one of these requests. In a
transaction it runs for every member before any member's table lookup, so a
request whose first member names a missing table answers with a later member's
name refusal; in a batch it runs for every PutRequest before the table lookup
and the key check, so an item that is also missing its key answers with the
name refusal. UpdateItem and a transacted Update also report the name ahead of
an UpdateExpression syntax error, a failing condition and a secondary-index key
type mismatch. With a long name in two places at once, a standalone UpdateItem
reports the ExpressionAttributeNames placeholder before a map key in a value,
and a standalone PutItem reports the item before the ExpressionAttributeNames
placeholder; a transaction member reports a map key in a value first, then the
ExpressionAttributeNames placeholder, then (for Put) the item.

ExtendDB checked names only on the put paths (PutItem, BatchWriteItem
PutRequest, TransactWriteItems Put and ImportTable), after the table lookup,
with its own wording: "One or more parameter values were invalid: Size of
attribute name '...' has exceeded the maximum size limit of 65535 bytes".
UpdateItem and the Update member of TransactWriteItems stored a 65,536-byte
name with HTTP 200, and a long name that arrived only through a condition's
ExpressionAttributeNames was accepted on PutItem, DeleteItem and every
transaction member.

validate_attribute_name_sizes and the new
validate_expression_attribute_name_sizes take an AttributeNameRequestForm
(SingleItem, TransactionMember or BatchRequest), which selects the wording and
the order of the two expression maps. validate_attribute_updates_name_sizes
covers the legacy AttributeUpdates map. Each map is visited in sorted
placeholder order, and the figure in the message is max_attribute_name_bytes
plus one. PutItem, UpdateItem and DeleteItem run the checks before the table
lookup. TransactWriteItems checks every member in a pass before
prepare_write_op; BatchWriteItem checks every PutRequest in a pass before the
per-table loop. The checks run in the engine, which all four storage backends
share, so no backend code changes. The attribute name size row in
docs/dynamodb-limits.md moves from Partial to Enforced and gives the wording
and order for each form.

tests/test_attribute_name_limit_on_update.py has 49 dual-target cases. They
pass on servers built from this change on SQLite, PostgreSQL and MongoDB. The
first 35 cases pass on the service; the 14 added for the condition maps, the
missing-table and missing-key positions and the two-map orders have not yet
been run on the service, and each one encodes a probe row the service
answered. A 91-row probe of the same shapes gives the same status, code,
message and stored item on the service and on each of the three servers, with
one exception outside this change: the control row for an index-key type
mismatch alone on UpdateItem, which PostgreSQL does not refuse on main.
Removing, switching, swapping or moving each engine call site, and each
wording and ordering branch in core, makes the cases that cover it fail. Core
unit tests pin every message form, both map orders, the byte count of
multi-byte names and the sorted placeholder order.

Fixes #365.
@LeeroyHannigan LeeroyHannigan changed the title fix(limits): enforce attribute-name size on every write path with per-form wording fix(limits): attribute-name size limit on every write path Oct 22, 2026
Conflicted file and resolution:
- docs/dynamodb-limits.md, Enforcement Coverage summary table: this branch
  moved the attribute name size row from Partial to Enforced (Items row
  6/0/0/0); main recounted Secondary Indexes (5/0/1/0) and
  Import/Export/Backup (2/2/0/7). Kept both category rows and recomputed
  the Total row from the merged category rows: 32 enforced, 2 partial,
  16 not enforced, 13 not applicable.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] UpdateItem and TransactWriteItems Update do not enforce the attribute-name size limit; message differs from the service

3 participants