Skip to content

fix(engine): validate numbers on every key and item path - #620

Open
LeeroyHannigan wants to merge 2 commits into
fix/616-number-exponent-overflowfrom
fix/618-number-validation-all-ops
Open

LeeroyHannigan wants to merge 2 commits into
fix/616-number-exponent-overflowfrom
fix/618-number-validation-all-ops

Conversation

@LeeroyHannigan

Copy link
Copy Markdown
Collaborator

Part of #618. Stacked on #619 (target branch fix/616-number-exponent-overflow), whose exponent rules these checks use.

What changed

Only the PutItem item and UpdateItem's update values ran number validation. AttributeValue keeps an N value it cannot read as written (crates/core/src/types/attribute_value.rs:105-108), so every other operation used values such as abc and 1E126 as written: BatchWriteItem and TransactWriteItems stored them, GetItem and DeleteItem looked them up, and an N partition key written that way became a partition of its own.

This PR checks the N and NS values on these paths, with the text DynamoDB gives on each:

Path Message on DynamoDB and here
GetItem, BatchGetItem and TransactGetItems keys the number message, no prefix
DeleteItem and UpdateItem keys, PutItem and DeleteItem ExpressionAttributeValues 1 validation error detected: and the number message
BatchWriteItem puts (keys, attributes, NS members, nested L and M) and delete keys the number message, no prefix
TransactWriteItems Put items, Update, Delete and ConditionCheck keys the number message, no prefix
TransactWriteItems ExpressionAttributeValues ExpressionAttributeValues contains invalid value: <message> for key :v

The new checks are in crates/core/src/validation/number/operations.rs, with one or two call sites in each of eight engine handlers. validation/mod.rs is not changed.

BatchWriteItem, TransactGetItems and TransactWriteItems read a number with a 32-bit exponent and scale on DynamoDB, and the other operations with a 64-bit exponent. So 10E2147483647 is Number underflow on the first three and Number overflow on the rest, and 1E2147483648 cannot be converted on the first three. validate_number_i32_scale reproduces the 32-bit reading, and a unit test runs both readers over 4,000 generated strings and fails if they answer differently outside the four documented differences.

Order against a second fault

Each check is placed where DynamoDB reports it, unless that position would change the answer to a two-fault request that main already answers as DynamoDB does:

  • GetItem, DeleteItem, UpdateItem, BatchGetItem and BatchWriteItem report a malformed number ahead of the table lookup; PutItem reports item numbers ahead of the mixed-parameter check and the lookup, and value numbers after the mixed-parameter check.
  • TransactGetItems checks each item's key right after its projection; TransactWriteItems checks each action right after its expressions are parsed. The ClientRequestToken check now runs before the actions, as on DynamoDB.
  • Where DynamoDB reports the number first and main reports the other fault, this branch keeps main's answer: a duplicate item or a missing table in TransactGetItems, a missing table in an earlier TransactWriteItems action, and an expression fault in the same action.
  • BatchWriteItem and BatchGetItem visit tables in name order. DynamoDB's order for two tables was neither body nor name order on four sends, and not stable for BatchGetItem.
  • In a PutItem item, numbers are checked before nesting depth, as on main. DynamoDB follows the attribute order of the request body, which Item does not keep.

Measured

DynamoDB in us-east-1, 2026-10-27, every request sent twice with the same answer both times: 20 paths with abc, 1E126, 0E126 and a control; 31 paths with 24 values; three probes of two-fault order; token, exponent marker and fraction rows. A refused batch or transaction writes nothing on DynamoDB and on this branch; main stored the whole request, invalid values included.

Against main, on the same probes: no request that main answers as DynamoDB does is answered differently here, by exact text or with the 1 validation error detected: prefix ignored, apart from Unicode digits below.

Not changed here

Tests

  • Unit: 18 tests in operations/tests.rs. In number.rs the two zero tests gain the measured rows and the guard test is renamed.
  • Wire: number_validation_all_ops.rs (27 tests) and number_validation_order.rs (11). Against SQLite dev-mode servers, the number_validation filter passes 50 of 50 here and fails 31 on fix(core): range check number exponents as DynamoDB does #619's head.
  • Full wire suite, one thread: 4 tests fail here and 35 on fix(core): range check number exponents as DynamoDB does #619's head; none fails here that passes there. The 4 fail on both (capacity throttling and a vector index test).
  • cargo fmt --all -- --check and cargo +1.97.0 clippy --all-targets -- -D warnings are clean. cargo test --workspace --exclude extenddb-storage-cassandra: 1,817 passed. cargo test -p extenddb-storage-cassandra --lib: 61 passed. cargo +1.88.0 test -p extenddb-core: 542 passed.

Merging with open PRs

13 of the 17 open PRs that touch these files merge cleanly. #433 conflicts in put_item.rs and transact_write_items.rs; both sides can be kept. #399, #507 and #597 conflict in delete_item.rs and put_item.rs on comment lines, where both sides can be kept, and in transact_write_items.rs, where #399 splits the action loop into phases. There the per-action check goes at the head of phase 1's per-item step; with it, three rows where this branch answers the number ahead of a nested value too deep move to the nesting error, because #399 checks nesting in a later phase.

Only the PutItem item and UpdateItem's update values were checked.
AttributeValue keeps a number it cannot read as written, so GetItem,
DeleteItem and UpdateItem keys, BatchGetItem and TransactGetItems keys,
BatchWriteItem writes, every TransactWriteItems action, and PutItem and
DeleteItem condition values used values such as abc and 1E126 as
written, where DynamoDB refuses them with a ValidationException.

Each of those paths now checks its N and NS values with the text that
operation answers. PutItem, UpdateItem and DeleteItem start the message
with "1 validation error detected: ", the other operations do not, and
a TransactWriteItems value names its placeholder. BatchWriteItem,
TransactGetItems and TransactWriteItems read a number with a 32-bit
exponent and scale, as DynamoDB does on those operations, so
10E2147483647 is a Number underflow there and a Number overflow on the
others.

Each check is placed where DynamoDB reports it, unless that position
would change the answer to a request with a second fault that main
already answers as DynamoDB does. The ClientRequestToken check now runs
before the actions, as on DynamoDB.

A zero whose exponent fits in an i32, such as 0E126, stays accepted as
0 until Query, Scan and ImportTable validate numbers. A zero whose
exponent does not fit is range checked on its magnitude, which makes
0E9223372036854775807 a Number underflow, as on DynamoDB.

Part of #618
number_validation_all_ops covers each validated path with DynamoDB's
message for abc, 1E126 and the 32-bit edge values, a refused batch or
transaction writing nothing, and valid zeros stored as 0.
number_validation_order pins the position of each check against a
second fault in the same request: an absent table, a mixed-parameter
error, an unused name, a projection or condition syntax error in another
item, a duplicate key and a long ClientRequestToken. Each expectation
is a DynamoDB answer measured twice on 2026-10-27.

Part of #618

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants