Skip to content

fix(expression): range check numbers before bigdecimal arithmetic and comparison - #803

Open
LeeroyHannigan wants to merge 2 commits into
fix/618-number-validation-all-opsfrom
fix/number-arith-bound
Open

LeeroyHannigan wants to merge 2 commits into
fix/618-number-validation-all-opsfrom
fix/number-arith-bound

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

What

Stacked on #620, so it merges after #620 and #619.

This PR changes how UpdateItem and TransactWriteItems do number arithmetic, and how condition and filter expressions compare numbers.

  • Update arithmetic: SET n = n + :v, SET n = n - :v and a number ADD now range check each operand with validate_and_normalize_number, and parse its normalized text. A sum of two numbers that the service accepts has fewer than 300 digits.
  • A stored operand out of range answers the check's message: Number overflow, Number underflow or the 38-digit message. A stored operand that is not a number answers The parameter cannot be converted to a numeric value. That answer does not include the stored text, because the caller may not be allowed to read the item.
  • Results use the service's spelling. Before, ExtendDB stored 2.25 - 0.25 as 2.00 and 1.5 + 1.5 as 3.0, where the service stores 2 and 3. A result outside the range keeps the Number overflow answer, which the service also gives for a result below the range.
  • Comparisons: the server parses numbers in condition and filter expressions the same way. When the server builds the expression maps, it records a placeholder that the check refuses. A filter then does not read that placeholder again for each item.
  • A number that the service cannot store compares as a value of another type. Only <> is true for it, so a <> b agrees with NOT (a = b). Before, such a value compared by its bigdecimal reading, when bigdecimal could parse it. This applies to an item stored without the number checks, and to a Query or Scan placeholder that the service refuses.
  • MongoDB: the native ADD pipeline now takes two shapes of delta, in normalized text. One is an integer of at most 33 digits. The other has at most 20 digits before the decimal point and 13 after it. Its filter admits a stored value without an exponent, within limits that keep the sum within 34 digits. A Decimal128 holds 34 digits exactly. Every other ADD takes the session path, where the update evaluator range checks both operands and the sum.

Not changed here:

Why

The update evaluator parsed both operands with bigdecimal and range checked only the result. bigdecimal aligns the operands to the larger scale before it adds them, so the cost of a sum grows with the gap between their exponents. UpdateItem and TransactWriteItems check ExpressionAttributeValues before an update runs. An operand can also come from the stored item. An item that an older release stored can hold a number that was never checked (#618). A stored 1E999999999 plus 1 builds an integer of about 10^9 digits, over 400 MB. A stored zero written 0E-999999999 does the same, because bigdecimal scales the other operand to the zero's scale.

bigdecimal's parse takes time quadratic in the digit count. A million digits took about a second in a release build, and a request body can carry 16 MiB. UpdateItem and TransactWriteItems parse their ExpressionAttributeValues before they check numbers. Query and Scan do not check them yet (#544, #562).

The MongoDB native ADD pipeline did not go through the update evaluator, so the first commit left it unchecked. On MongoDB 7, GetItem returned 1.000000000000000000000000000000000E+126 after 5E125 + 5E125, where the service answers Number overflow. It returned 99999999999999999999999999999999990000 after 38 nines + 1, where the service stores 100000000000000000000000000000000000000.

Testing done

Measured on DynamoDB in us-east-1 on 2026-11-03, two times for each request, on UpdateItem and on a TransactWriteItems Update:

  • 2.25 - 0.25 is 2, 1.5 - 1.5 is 0, and 1E125 + 1E125 is 2 followed by 125 zeros. 1.5 + 1.5 is 3, measured on UpdateItem only.
  • 5E125 + 5E125, 1E125 + 1 and 1.1E-130 - 1E-130 answer Number overflow.
  • 38 nines + 1 is 100000000000000000000000000000000000000, and the sum of two 33-digit integers is exact.

Tests:

  • update_evaluator_tests.rs: stored operands out of range, not a number, and zeros with large exponents. Each test runs on a thread with a 10 second limit. With the operand check removed, those tests time out.
  • evaluator_tests.rs: a 2,000,000-digit value in a placeholder and in the item, and one such placeholder compared against 100 items. With the comparison parse unbounded, or with a refused placeholder read again for each item, these tests time out. Another test pins the comparisons of a stored number out of range.
  • tests/rust/src/update_arithmetic_result.rs: the spellings and overflow answers on UpdateItem and TransactWriteItems. It passes against DynamoDB, and against ExtendDB on PostgreSQL, SQLite and MongoDB.
  • crates/storage-mongodb unit tests: which deltas the native pipeline takes, and the filter for each kind of delta. With the MongoDB change reverted, both tests fail. On MongoDB 7, each filter pattern matched every stored text that it should match, and refused every other stored text in the probe.
  • On MongoDB 7, 50 threads each sent 20 ADD calls to one item, with SDK retries off. Deltas of 1, 0.5 and 0.001 gave no errors and the exact total. So did 1 added to a stored 0.5 and to a stored 21-digit integer, and a 25-digit delta.
  • The MongoDB build also passed the four RFC-0003 tests, the import, export and arithmetic pytest cases, and the Rust update_item, update_expressions, number_validation and update_item_number_validation groups.
  • cargo +1.97.0 clippy --all-targets -- -D warnings and cargo fmt --check are clean.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a. The answers change to match the service, and no design document describes the old answers.


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

… comparison

SET a = b + c, SET a = b - c and a number ADD parsed both operands with
bigdecimal and range checked only the result. bigdecimal aligns operands
to the larger scale before adding them, so the cost of a sum grows with
the gap between their exponents, and an operand read from an item stored
without the number checks (#618) could build an integer of about 10^9
digits.

Each operand is now range checked and parsed from its normalized text. A
stored operand that is not a number answers the service's text without
the stored value. The result is stored as the service spells it (2.25 -
0.25 is 2, not 2.00), and a result out of range keeps the Number overflow
answer, as measured on DynamoDB on 2026-11-03.

Numbers in condition and filter comparisons are parsed the same way,
which bounds bigdecimal's quadratic parse for values that are not checked
before the expression maps are built. A placeholder the check refuses is
recorded once per request, and a number the service cannot store compares
as a value of another type, so only <> is true for it.
The MongoDB fast path for a number ADD adds a Decimal128 in an update
pipeline and does not read the stored value first. A Decimal128 keeps 34
digits and holds magnitudes DynamoDB refuses. After 5E125 + 5E125,
GetItem returned 1.000000000000000000000000000000000E+126, where the
update evaluator answers Number overflow. After 38 nines + 1, it
returned 99999999999999999999999999999999990000, where the exact sum is
1E38.

The fast path now takes a delta whose normalized text is an integer of
at most 33 digits, or has at most 20 digits before the decimal point and
13 after it. Its filter admits a stored value written without an
exponent, within limits that keep the sum within 34 digits: an integer
of up to 33 digits when both operands are integers, and up to 20 and 13
digits when either has a fraction. Any other ADD takes the session path,
where the update evaluator range checks both operands and the sum. The
delta is the Decimal128 of its normalized text, so 1E3 is added as 1000.

On MongoDB 7, fifty threads sending 20 ADD calls each to one item, with
SDK retries off, gave no errors and the exact total for deltas of 1, 0.5
and 0.001, for 1 added to a stored 0.5 and to a stored 21-digit integer,
and for a 25-digit delta.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-blocker Must ship for its milestone's release security v1 Needed for the 1.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants