Repository navigation
fix(expression): range check numbers before bigdecimal arithmetic and comparison - #803
Open
LeeroyHannigan wants to merge 2 commits into
Open
LeeroyHannigan wants to merge 2 commits into
LeeroyHannigan wants to merge 2 commits into
Conversation
… comparison SET a = b + c, SET a = b - c and a number ADD parsed both operands with bigdecimal and range checked only the result. bigdecimal aligns operands to the larger scale before adding them, so the cost of a sum grows with the gap between their exponents, and an operand read from an item stored without the number checks (#618) could build an integer of about 10^9 digits. Each operand is now range checked and parsed from its normalized text. A stored operand that is not a number answers the service's text without the stored value. The result is stored as the service spells it (2.25 - 0.25 is 2, not 2.00), and a result out of range keeps the Number overflow answer, as measured on DynamoDB on 2026-11-03. Numbers in condition and filter comparisons are parsed the same way, which bounds bigdecimal's quadratic parse for values that are not checked before the expression maps are built. A placeholder the check refuses is recorded once per request, and a number the service cannot store compares as a value of another type, so only <> is true for it.
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
November 3, 2026 17:29
The MongoDB fast path for a number ADD adds a Decimal128 in an update pipeline and does not read the stored value first. A Decimal128 keeps 34 digits and holds magnitudes DynamoDB refuses. After 5E125 + 5E125, GetItem returned 1.000000000000000000000000000000000E+126, where the update evaluator answers Number overflow. After 38 nines + 1, it returned 99999999999999999999999999999999990000, where the exact sum is 1E38. The fast path now takes a delta whose normalized text is an integer of at most 33 digits, or has at most 20 digits before the decimal point and 13 after it. Its filter admits a stored value written without an exponent, within limits that keep the sum within 34 digits: an integer of up to 33 digits when both operands are integers, and up to 20 and 13 digits when either has a fraction. Any other ADD takes the session path, where the update evaluator range checks both operands and the sum. The delta is the Decimal128 of its normalized text, so 1E3 is added as 1000. On MongoDB 7, fifty threads sending 20 ADD calls each to one item, with SDK retries off, gave no errors and the exact total for deltas of 1, 0.5 and 0.001, for 1 added to a stored 0.5 and to a stored 21-digit integer, and for a 25-digit delta.
This was referenced Nov 3, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Stacked on #620, so it merges after #620 and #619.
This PR changes how
UpdateItemandTransactWriteItemsdo number arithmetic, and how condition and filter expressions compare numbers.SET n = n + :v,SET n = n - :vand a numberADDnow range check each operand withvalidate_and_normalize_number, and parse its normalized text. A sum of two numbers that the service accepts has fewer than 300 digits.Number overflow,Number underflowor the 38-digit message. A stored operand that is not a number answersThe parameter cannot be converted to a numeric value. That answer does not include the stored text, because the caller may not be allowed to read the item.2.25 - 0.25as2.00and1.5 + 1.5as3.0, where the service stores2and3. A result outside the range keeps theNumber overflowanswer, which the service also gives for a result below the range.<>is true for it, soa <> bagrees withNOT (a = b). Before, such a value compared by its bigdecimal reading, when bigdecimal could parse it. This applies to an item stored without the number checks, and to aQueryorScanplaceholder that the service refuses.ADDpipeline now takes two shapes of delta, in normalized text. One is an integer of at most 33 digits. The other has at most 20 digits before the decimal point and 13 after it. Its filter admits a stored value without an exponent, within limits that keep the sum within 34 digits. ADecimal128holds 34 digits exactly. Every otherADDtakes the session path, where the update evaluator range checks both operands and the sum.Not changed here:
Expectedvalues onPutItem,UpdateItemandDeleteItemstill skip the number check, so a refusedExpectedvalue compares as above.BETWEENwhose bound the check refuses now evaluates false. Before, it answered the inverted-bound error.QueryandScanplaceholders get the request check with fix(query): check Query values against the key schema and the number rules #544 and fix(scan): check the numbers in ExpressionAttributeValues as the service does #562. Until fix(query): check Query values against the key schema and the number rules #544,Queryparses the sort-key value as written.TransactionConflictExceptionunder contention (MongoDB: concurrent UpdateItem calls that take the session path answer TransactionConflictException (RFC-0003 4.4) #809). The TTL sweep reads the stored text, which the native pipeline can write as1700000000.0(MongoDB: the TTL sweep parses the stored number text as an integer, so a TTL value the native ADD pipeline writes as 1700000000.0 never expires #810).mainhas both gaps.Why
The update evaluator parsed both operands with bigdecimal and range checked only the result. bigdecimal aligns the operands to the larger scale before it adds them, so the cost of a sum grows with the gap between their exponents.
UpdateItemandTransactWriteItemscheckExpressionAttributeValuesbefore an update runs. An operand can also come from the stored item. An item that an older release stored can hold a number that was never checked (#618). A stored1E999999999plus1builds an integer of about 10^9 digits, over 400 MB. A stored zero written0E-999999999does the same, because bigdecimal scales the other operand to the zero's scale.bigdecimal's parse takes time quadratic in the digit count. A million digits took about a second in a release build, and a request body can carry 16 MiB.
UpdateItemandTransactWriteItemsparse theirExpressionAttributeValuesbefore they check numbers.QueryandScando not check them yet (#544, #562).The MongoDB native
ADDpipeline did not go through the update evaluator, so the first commit left it unchecked. On MongoDB 7,GetItemreturned1.000000000000000000000000000000000E+126after5E125 + 5E125, where the service answersNumber overflow. It returned99999999999999999999999999999999990000after 38 nines+ 1, where the service stores100000000000000000000000000000000000000.Testing done
Measured on DynamoDB in us-east-1 on 2026-11-03, two times for each request, on
UpdateItemand on aTransactWriteItemsUpdate:2.25 - 0.25is2,1.5 - 1.5is0, and1E125 + 1E125is2followed by 125 zeros.1.5 + 1.5is3, measured onUpdateItemonly.5E125 + 5E125,1E125 + 1and1.1E-130 - 1E-130answerNumber overflow.+ 1is100000000000000000000000000000000000000, and the sum of two 33-digit integers is exact.Tests:
update_evaluator_tests.rs: stored operands out of range, not a number, and zeros with large exponents. Each test runs on a thread with a 10 second limit. With the operand check removed, those tests time out.evaluator_tests.rs: a 2,000,000-digit value in a placeholder and in the item, and one such placeholder compared against 100 items. With the comparison parse unbounded, or with a refused placeholder read again for each item, these tests time out. Another test pins the comparisons of a stored number out of range.tests/rust/src/update_arithmetic_result.rs: the spellings and overflow answers onUpdateItemandTransactWriteItems. It passes against DynamoDB, and against ExtendDB on PostgreSQL, SQLite and MongoDB.crates/storage-mongodbunit tests: which deltas the native pipeline takes, and the filter for each kind of delta. With the MongoDB change reverted, both tests fail. On MongoDB 7, each filter pattern matched every stored text that it should match, and refused every other stored text in the probe.ADDcalls to one item, with SDK retries off. Deltas of 1, 0.5 and 0.001 gave no errors and the exact total. So did 1 added to a stored 0.5 and to a stored 21-digit integer, and a 25-digit delta.update_item,update_expressions,number_validationandupdate_item_number_validationgroups.cargo +1.97.0 clippy --all-targets -- -D warningsandcargo fmt --checkare clean.Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC: n/a. The answers change to match the service, and no design document describes the old answers.
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.