Skip to content

fix(core): range check number exponents as DynamoDB does - #619

Open
LeeroyHannigan wants to merge 1 commit into
mainfrom
fix/616-number-exponent-overflow
Open

LeeroyHannigan wants to merge 1 commit into
mainfrom
fix/616-number-exponent-overflow

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #616.

#394 (34aa2df, backup format v1) moved the three exponent sums in validate_and_normalize_number to saturating i64 arithmetic, which fixed the overflow #616 reports. Main 817b240 answers the first three rows of the table below as DynamoDB does; on abf8082 a debug build panicked on the first and third. Main still parses the exponent as an i32, so an exponent between the i32 and i64 limits is refused as cannot be converted to a numeric value, where DynamoDB answers Number overflow or Number underflow. This PR now carries the i64 exponent and the zero check that goes with it.

What changed

In crates/core/src/validation/number.rs:

  • The exponent is parsed as an i64, as DynamoDB reads it, and the sums are done in i128, where none of them can overflow. This replaces feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394's saturating i64 sums and its len_i64 helper with digit_count and check_magnitude. 1E2147483648 is now Number overflow and 1E-2147483649 Number underflow; main answers cannot be converted for both. 1E9223372036854775808 is still cannot be converted, as on DynamoDB.
  • A zero whose exponent does not fit in an i32 is range checked (0E2147483648 is Number overflow). Main refuses those zeros as cannot be converted, so the set of refused inputs does not change.
  • A zero whose exponent fits in an i32 is still accepted as 0. DynamoDB refuses 0E126 and 0.0E-130, but a zero refused here keeps its raw spelling in AttributeValue, and only the PutItem item and UpdateItem's ExpressionAttributeValues and AttributeUpdates are validated afterwards. BatchWriteItem, TransactWriteItems, UpdateItem's key and the other operations would then store and look up 0E126 as a key separate from 0. BatchWriteItem, TransactWriteItems and other operations store N values that fail number validation as written #618 covers that validation gap and the zero check together; the unit test zero_with_i32_exponent_stays_accepted_until_618 marks the place to change.

The PR keeps the test #394 added in number.rs, exponents_at_the_ends_of_i32_classify_the_magnitude.

Measured

DynamoDB in us-east-1, PutItem with a non-key N attribute, 2026-10-27, every value sent twice with the same answer both times. The two ExtendDB columns are PutItem with the same attribute on 2026-10-30, against dev-mode SQLite servers (debug builds) built from main 817b240 and from this branch.

Value DynamoDB main 817b240 this branch
10E2147483647, 12E2147483647, 100E2147483647, -10E2147483647 Number overflow Number overflow Number overflow
1E+2147483647, 0.1E2147483647 Number overflow Number overflow Number overflow
1.5E-2147483648, -1.5E-2147483648, 0.01E-2147483648 Number underflow Number underflow Number underflow
1E2147483648, 1E4294967296, 1E9223372036854775807 Number overflow cannot be converted Number overflow
1E-2147483649, 1E-9223372036854775808 Number underflow cannot be converted Number underflow
1E9223372036854775808, 1E-9223372036854775809 cannot be converted cannot be converted cannot be converted
0E2147483648, 0E9223372036854775806 Number overflow cannot be converted Number overflow
0E-9223372036854775807, 0E-9223372036854775808 Number underflow cannot be converted Number underflow
0E126, 0.0E-130 Number overflow, Number underflow 0 0 (#618)

Not matched on purpose: DynamoDB answers 0E9223372036854775807 with Number underflow, and 0E9223372036854775806 with Number overflow. This branch answers Number overflow for both, and main answers cannot be converted for both. Each answer is a ValidationException that refuses the item.

DynamoDB starts each of these messages with 1 validation error detected: and ExtendDB does not. That difference is not changed here.

Tests

  • Unit tests in number.rs: five new tests, 26 tests in the module. Run against main 817b240's number.rs, two of the new tests fail on cannot be converted: exponent_is_read_as_i64 at 1E2147483648 and zero_with_exponent_outside_i32_is_range_checked at 0E2147483648. exponent_near_i32_limits_is_an_overflow_or_underflow, which failed on abf8082 with the overflow panic, passes on main since feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394.
  • Wire tests in tests/rust/src/number_validation.rs: three new tests. Against a dev-mode SQLite server built from this branch, the number_validation filter passes 12 of 12. Against main 817b240, put_item_exponent_near_i32_limits_is_overflow_or_underflow fails at 1E2147483648 and put_item_zero_with_exponent_outside_i32_is_rejected at 0E2147483648; put_item_exponent_outside_i64_is_malformed passes on main too.
  • cargo fmt --all --check and cargo +1.97.0 clippy --locked -p extenddb-core --all-targets -- -D warnings are clean. cargo +1.97.0 test --locked -p extenddb-core --lib: 700 passed, 1 ignored, among them feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394's exponents_at_the_ends_of_i32_classify_the_magnitude, doc_examples_normalize_as_stated and exponent_at_the_end_of_i32_is_refused_as_an_overflow.
  • At the first revision, a probe of GetItem, Query, BatchGetItem, BatchWriteItem, TransactWriteItems, UpdateItem and DeleteItem with zeros such as 0E126, -0E126 and 0.0E-130 as keys and attributes gave the same output on the branch as on main abf8082. For those zeros this branch and main 817b240 both return 0; the only check this PR adds on that path applies to an exponent outside i32.

validate_and_normalize_number parsed the exponent as an i32. #394 moved
its exponent sums to saturating i64 arithmetic, which fixed the overflow
panics of #616, but an exponent outside the i32 range was still refused
as malformed.

The exponent is now parsed as an i64 and the sums are done in i128
instead of saturating i64, so none of them can overflow. DynamoDB reads
the exponent the same way: 1E2147483648 is a Number overflow and
1E9223372036854775808 cannot be converted. A zero whose exponent does
not fit in an i32 is range checked too; main refused those as malformed.

A zero whose exponent fits in an i32 stays accepted as 0, although
DynamoDB refuses 0E126 and 0.0E-130. A refused zero keeps its raw
spelling in AttributeValue, and only the PutItem item and UpdateItem's
update values are validated afterwards, so UpdateItem's key and the
other operations would store 0E126 as a key separate from 0. #618
covers both.

Closes #616

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

v1 Needed for the 1.0 release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An N value with an exponent near the i32 limits panics the number validator in debug builds and gets the wrong error in release builds

1 participant