Repository navigation
fix(core): range check number exponents as DynamoDB does - #619
Open
LeeroyHannigan wants to merge 1 commit into
Open
LeeroyHannigan wants to merge 1 commit into
LeeroyHannigan wants to merge 1 commit into
Conversation
LeeroyHannigan
requested review from
amrith,
c33howard,
jcshepherd,
pdf-amzn,
robinnsc and
yesyayen
as code owners
October 27, 2026 10:08
validate_and_normalize_number parsed the exponent as an i32. #394 moved its exponent sums to saturating i64 arithmetic, which fixed the overflow panics of #616, but an exponent outside the i32 range was still refused as malformed. The exponent is now parsed as an i64 and the sums are done in i128 instead of saturating i64, so none of them can overflow. DynamoDB reads the exponent the same way: 1E2147483648 is a Number overflow and 1E9223372036854775808 cannot be converted. A zero whose exponent does not fit in an i32 is range checked too; main refused those as malformed. A zero whose exponent fits in an i32 stays accepted as 0, although DynamoDB refuses 0E126 and 0.0E-130. A refused zero keeps its raw spelling in AttributeValue, and only the PutItem item and UpdateItem's update values are validated afterwards, so UpdateItem's key and the other operations would store 0E126 as a key separate from 0. #618 covers both. Closes #616
LeeroyHannigan
force-pushed
the
fix/616-number-exponent-overflow
branch
from
October 30, 2026 09:07
95997b6 to
bfde5fd
Compare
This was referenced Nov 3, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #616.
#394 (34aa2df, backup format v1) moved the three exponent sums in
validate_and_normalize_numberto saturatingi64arithmetic, which fixed the overflow #616 reports. Main 817b240 answers the first three rows of the table below as DynamoDB does; on abf8082 a debug build panicked on the first and third. Main still parses the exponent as ani32, so an exponent between thei32andi64limits is refused ascannot be converted to a numeric value, where DynamoDB answersNumber overfloworNumber underflow. This PR now carries thei64exponent and the zero check that goes with it.What changed
In
crates/core/src/validation/number.rs:i64, as DynamoDB reads it, and the sums are done ini128, where none of them can overflow. This replaces feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394's saturatingi64sums and itslen_i64helper withdigit_countandcheck_magnitude.1E2147483648is nowNumber overflowand1E-2147483649Number underflow; main answerscannot be convertedfor both.1E9223372036854775808is stillcannot be converted, as on DynamoDB.i32is range checked (0E2147483648isNumber overflow). Main refuses those zeros ascannot be converted, so the set of refused inputs does not change.i32is still accepted as0. DynamoDB refuses0E126and0.0E-130, but a zero refused here keeps its raw spelling inAttributeValue, and only the PutItem item and UpdateItem's ExpressionAttributeValues and AttributeUpdates are validated afterwards. BatchWriteItem, TransactWriteItems, UpdateItem's key and the other operations would then store and look up0E126as a key separate from0. BatchWriteItem, TransactWriteItems and other operations store N values that fail number validation as written #618 covers that validation gap and the zero check together; the unit testzero_with_i32_exponent_stays_accepted_until_618marks the place to change.The PR keeps the test #394 added in
number.rs,exponents_at_the_ends_of_i32_classify_the_magnitude.Measured
DynamoDB in us-east-1, PutItem with a non-key
Nattribute, 2026-10-27, every value sent twice with the same answer both times. The two ExtendDB columns are PutItem with the same attribute on 2026-10-30, against dev-mode SQLite servers (debug builds) built from main 817b240 and from this branch.10E2147483647,12E2147483647,100E2147483647,-10E2147483647Number overflowNumber overflowNumber overflow1E+2147483647,0.1E2147483647Number overflowNumber overflowNumber overflow1.5E-2147483648,-1.5E-2147483648,0.01E-2147483648Number underflowNumber underflowNumber underflow1E2147483648,1E4294967296,1E9223372036854775807Number overflowcannot be convertedNumber overflow1E-2147483649,1E-9223372036854775808Number underflowcannot be convertedNumber underflow1E9223372036854775808,1E-9223372036854775809cannot be convertedcannot be convertedcannot be converted0E2147483648,0E9223372036854775806Number overflowcannot be convertedNumber overflow0E-9223372036854775807,0E-9223372036854775808Number underflowcannot be convertedNumber underflow0E126,0.0E-130Number overflow,Number underflow00(#618)Not matched on purpose: DynamoDB answers
0E9223372036854775807withNumber underflow, and0E9223372036854775806withNumber overflow. This branch answersNumber overflowfor both, and main answerscannot be convertedfor both. Each answer is aValidationExceptionthat refuses the item.DynamoDB starts each of these messages with
1 validation error detected:and ExtendDB does not. That difference is not changed here.Tests
number.rs: five new tests, 26 tests in the module. Run against main 817b240'snumber.rs, two of the new tests fail oncannot be converted:exponent_is_read_as_i64at1E2147483648andzero_with_exponent_outside_i32_is_range_checkedat0E2147483648.exponent_near_i32_limits_is_an_overflow_or_underflow, which failed on abf8082 with the overflow panic, passes on main since feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394.tests/rust/src/number_validation.rs: three new tests. Against a dev-mode SQLite server built from this branch, thenumber_validationfilter passes 12 of 12. Against main 817b240,put_item_exponent_near_i32_limits_is_overflow_or_underflowfails at1E2147483648andput_item_zero_with_exponent_outside_i32_is_rejectedat0E2147483648;put_item_exponent_outside_i64_is_malformedpasses on main too.cargo fmt --all --checkandcargo +1.97.0 clippy --locked -p extenddb-core --all-targets -- -D warningsare clean.cargo +1.97.0 test --locked -p extenddb-core --lib: 700 passed, 1 ignored, among them feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394'sexponents_at_the_ends_of_i32_classify_the_magnitude,doc_examples_normalize_as_statedandexponent_at_the_end_of_i32_is_refused_as_an_overflow.0E126,-0E126and0.0E-130as keys and attributes gave the same output on the branch as on main abf8082. For those zeros this branch and main 817b240 both return0; the only check this PR adds on that path applies to an exponent outsidei32.