Skip to content

Refuse an update whose result nests beyond 32 levels - #590

Open
LeeroyHannigan wants to merge 4 commits into
mainfrom
fix/update-result-nesting-depth
Open

LeeroyHannigan wants to merge 4 commits into
mainfrom
fix/update-result-nesting-depth

Conversation

@LeeroyHannigan

Copy link
Copy Markdown
Collaborator

Fixes #574.

An UpdateItem or a TransactWriteItems Update whose result nests an attribute more than 32 levels deep is now refused, as DynamoDB refuses it. Before, it was applied on every backend.

What happened on main

crates/engine/src/update_item.rs checks the nesting depth of ExpressionAttributeValues, and each value in such an update is within the limit. Nothing checked the item the update produced, so SET #m.#k...#k = :v with a value 2 wrappers deep at a path 30 wrappers deep stored an attribute nested past the limit.

What the service answers

Measured on 2026-10-25 at 19:53 UTC and 2026-10-26 at 00:55 UTC in us-east-1, each case twice. Depth here is counted in M or L wrappers around a scalar; 31 wrappers is 32 levels, the PutItem limit.

  • An update whose result has at most 31 wrappers in every attribute is applied. That covers a SET at the top level, into a map, into a list and through if_not_exists that keeps the existing value, and a SET of a scalar at the deepest path of an item already at the limit.
  • One more wrapper is refused with ValidationException Nesting Levels have exceeded supported limits, with no 1 validation error detected: prefix and no suffix, and nothing is written. That covers a SET into a map or list, into a new map member, through list_append, and through the default of if_not_exists.
  • A TransactWriteItems Update past the limit is cancelled with one ValidationError reason carrying the same text.
  • A condition that fails on the same request answers ConditionalCheckFailedException.
  • The depth error comes before the 400 KB item size error, for UpdateItem with the SET actions in either order and for a TransactWriteItems Update (measured 2026-10-26 at 01:26 and 01:44 UTC).

The change

apply_update_validated in crates/core/src/expression/update_evaluator.rs is the function every backend's UpdateItem and TransactWriteItems Update path calls after its condition check when it evaluates the expression in Rust. The MongoDB native update path takes only one-element paths with a checked value, so it cannot produce an over-limit value. apply_update_validated now calls validate_written_nesting_depth (new file crates/core/src/expression/update_nesting.rs), which checks each top-level attribute a SET or an ADD wrote, once however many actions write into it. ADD is checked because ExtendDB creates the missing maps on an ADD path. REMOVE and DELETE cannot deepen a value. Attributes the update does not set are not checked, so an item stored deeper than the limit before this change can still be updated through its other attributes.

Tests

  • crates/core/src/expression/update_nesting.rs: the limit for maps and lists, every branch of a list checked, ADD targets checked, two actions into one attribute, and attributes outside the SET and ADD left alone. A limit of 32 instead of 31 fails four of them, checking only the first element of each list fails one, and leaving ADD out fails one.
  • tests/test_update_result_nesting_depth.py: 22 dual-target wire tests from the measurements. The ADD test compares only the error code and the stored item: the service refuses that request for its 33-element path, with The document path has too many nesting levels, and ExtendDB has no path length check (a gap on main, not changed here).

Run locally on 2026-10-26 at the head, with every file under tests/ whose test names match update, nest, item_size or transact:

Backend Head Base (main)
SQLite 312 passed, 6 skipped; the new file 22 passed the new file: 10 failed, 12 passed
PostgreSQL 16 312 passed, 6 skipped not run
MongoDB 7 308 passed, 6 skipped, 4 failed not run

The 4 MongoDB failures are in tests/test_transact_absent_item_isolation.py, the write skew on main that the file marks XFAIL_UNTIL_MONGODB_FIX; the marker applies only when EXTENDDB_TEST_MONGODB_CONTAINER is set, and these runs did not set it. The 10 that fail on main are the refusals, the ADD one included; the 12 that pass are the updates that stay within the limit, the condition test and the update of another attribute.

Also run: cargo +1.97.0 fmt --all -- --check, cargo +1.97.0 clippy --all-targets -- -D warnings, cargo test --workspace --exclude extenddb-storage-cassandra on stable and on 1.88.0 (1,772 passed each), and cargo test -p extenddb-storage-cassandra --lib (61 passed). The Cassandra integration tests and the new file on Cassandra were not run locally.

CI runs the new file in run-integration, run-integration-sqlite, mongodb-pytest and cassandra-pytest.

Files

crates/core/src/expression/update_evaluator.rs is 642 lines on main and grows by one, the call. The check lives in the new 156-line update_nesting.rs.

Merging

git merge-tree --write-tree is clean against the 18 open PRs that change crates/core/src/expression/ or crates/core/src/validation/mod.rs and merge with main. #416, #395 and #200 conflict with main already.

UpdateItem and the TransactWriteItems Update checked the nesting depth of
ExpressionAttributeValues but not of the item the update produced, so a SET
of a 2-level map at a 30-level path stored an attribute nested past the
limit, on every backend. Amazon DynamoDB refuses an update whose result nests
an attribute more than 32 levels (31 M or L wrappers around a scalar) and
writes nothing: UpdateItem answers ValidationException `Nesting Levels have
exceeded supported limits`, and a TransactWriteItems Update is cancelled with
a ValidationError reason carrying that text (measured 2026-10-25 at 19:53 UTC
and 2026-10-26 at 00:55 UTC, each case twice). A failing condition is reported
first.

apply_update_validated, which every backend's update and transactional update
path calls after its condition check, now checks the top-level attributes the
SET actions wrote. REMOVE, ADD and DELETE cannot deepen a value, and other
attributes are not checked, so an item stored deeper than the limit before
this change stays updatable through them.
Dual-target wire tests from the 2026-10-25 and 2026-10-26 measurements: SET
into a deep map or list, at the top level, through list_append and through
if_not_exists, up to and past the limit; a failing condition next to a result
that is too deep; an update of another attribute on an item at the limit; and
the TransactWriteItems Update cancellation reason.
ExtendDB's ADD creates the missing maps on its path, so an ADD can nest an
attribute past the limit (`ADD #m.#k...#k.#a.#b.#c :one` on a value 30
wrappers deep stored 32). The check now covers the top-level attribute of
each ADD target as well as each SET target.

Each top-level attribute is walked once however many actions write into it,
and the walk matches maps and lists directly instead of boxing an iterator
per node. With 200 SET actions into one list of 4,000 maps, UpdateItem took
78 ms against 36 ms on main before this; the walk now runs once.
The service refuses `ADD path(29).#a.#b.#c :one` on a value 30 wrappers
deep for its 33-element path (measured 2026-10-26 at 01:26 UTC, twice);
ExtendDB refuses it for the depth of the maps the ADD would create. The test
compares the error code and the stored item only. The stored-depth helper
now takes the deepest branch, so the list_append case sees an appended
element.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UpdateItem: a result nested more than 32 levels is applied; the service refuses it

3 participants