[LeeroyHannigan]

Fixes #572.

On MongoDB, a condition on a map member was compiled to a filter path that skipped the map's M field, so it never matched. A condition that held was refused, and attribute_not_exists or <> on a member that was present let the write through.

What happened on main

On a table with no index and no stream, while the server's cache holds a fresh entry recording that the table has no index (60 seconds after a write), an UpdateItem or DeleteItem condition that the analyzer accepts is compiled to a MongoDB filter and checked by MongoDB. Items are stored in their typed form, so a map's members sit under its M field and a list's elements under its L field:

• #m.#k compiled to item_data.m.k; the member is stored at item_data.m.M.k.
• #m.#l[0] compiled to item_data.m.l.L.0; the element is stored at item_data.m.M.l.L.0.
• #l[0].#k compiled to item_data.l.L.0.k; the member is stored at item_data.l.L.0.M.k.

attribute_exists, comparisons, begins_with, contains and attribute_type on those paths answered ConditionalCheckFailedException when they held. attribute_not_exists(#m.#k) and #m.#k <> :v on a present member let the write through. Top-level attributes and list indexes were compiled correctly.

The change

A member name after the top-level attribute now gets an M segment, as a list index already got an L segment. The rule reaches the stored value for maps in maps, lists in maps, maps in lists and lists in lists. A member named M, L, S or 0 is looked up inside the parent's M document, so it cannot be read as a type tag or an index.

What the service answers

Measured on 2026-10-25 at 20:51, 20:52 and 21:13 UTC in us-east-1, each request sent twice with the same answer: 21 shapes on map members, list elements inside maps and map members inside list elements, on UpdateItem and DeleteItem, holding and failing, plus 52 further shapes covering missing and wrong-type intermediates, out-of-range indexes, members named M, L, S, 0, x$, $x and a.b, empty maps, binary, BOOL and NULL members, sets and lists inside maps, and < and > on nested strings. On the 50 condition rows of the second set that ExtendDB can also run, SQLite matches the service on all 50 and MongoDB at the head on 47; the 3 it misses are the U+0000 rows under "Not changed here". MongoDB on the build before the fix matches 29.

Tests

• crates/storage-mongodb/src/condition_tests.rs: unit tests for the compiled paths of map members, list elements inside maps, map members inside list elements, an element of a list inside a list (l[2][1] compiles to item_data.l.L.2.L.1), members named M, L, S and 0 (item_data.m.M.M, item_data.m.M.L, item_data.m.M.S and item_data.m.M.0), and a four-element path (m.mm.n.k compiles to item_data.m.M.mm.M.n.M.k). The exists_at() helper takes PathElement values, so a member named 0 can be written as a name and not as an index. Each of these changes to the path builder fails a test: removing the M segment, adding it before the first name, adding it before an index, leaving it out for members named M, L, S or 0, leaving it out for an all-digit member name, leaving out the L segment of the second of two consecutive indexes, and leaving the M segment out from the fourth path element on.
• tests/test_condition_nested_paths.py: 26 dual-target wire tests. The 21 in HOLDS, FAILS and the DeleteItem tests are from the 2026-10-25 measurements. The 5 in DERIVED_HOLDS, equality on the members named M, L, S and 0 of a map t and on g[2][1], an element of a list inside a list, were encoded from ExtendDB's in-Rust condition evaluator and then measured on Amazon DynamoDB on 2026-10-28 with the test's item, names and values, two samples per row: each applied its update, and #t.#M = :other and #g[2][1] = :other answered ConditionalCheckFailedException and left the item. Each test puts its own item right before the conditional write, so on MongoDB the write runs while the cache entry is fresh and reaches the compiled filter.
• crates/storage-mongodb/tests/pushdown_parity.rs: the property harness now generates items whose attributes can hold maps and lists nested two deep, with members named a, b, M, L, S and 0, and conditions on paths of up to three elements; when the item has an attribute, three in four of the paths are replaced by paths the item has (about three in five of all generated paths, since one item in five is empty). It compares evaluate_condition with the compiled filter as the crate's BSON interpreter evaluates it, on each. The harness changes in this PR because it generated single-name paths over flat items and could not see this PR's defect: with main's path builder, the old harness passes and the new one fails. The case budget is unchanged at 1024 per property.

Run locally on 2026-10-25:

• MongoDB (mongo:7 replica set): the new file 21 passed at the head; on the commit before the fix 13 failed and 8 passed, the same rows in two runs. The 8 that pass on the old build are rows whose answer the old path also gives (existence tests below a map member that answer "absent", list indexes, failing conditions).
• MongoDB: 281 other condition, update and delete wire tests pass at the head. Three transaction isolation rows fail without EXTENDDB_TEST_MONGODB_CONTAINER; CI sets it, and with it set they are the expected failures main already marks.
• SQLite and PostgreSQL: the new file 21 passed on each.
• 6,000 generated conditions on nested items (paths and values drawn from the item, AND and OR of up to three leaves, names a, b, M, L, S, 0, N), sent with retries off: MongoDB at the head and SQLite gave the same answer for all 6,000, and MongoDB's profiler shows each one was checked by the compiled filter. The build before the fix differed from SQLite on 472.
• cargo +1.97.0 fmt --all -- --check, cargo +1.97.0 clippy -p extenddb-storage-mongodb --all-targets -- -D warnings, and cargo test -p extenddb-storage-mongodb --lib condition (17 passed).

Run locally on 2026-10-28, on the final tree:

• MongoDB (mongo:7 replica set): the new file 26 passed. The path builder runs only for the compiled filter, and changing it fails the t and g rows: leaving out the M segment for members named M, L, S and 0 fails the four t rows, leaving it out for an all-digit member name fails the #t.#zero row, and leaving out the L segment of the second of two consecutive indexes fails the #g[2][1] row.
• pushdown_parity.rs with 50 fixed seeds (102,400 cases): no difference between the compiled filter and evaluate_condition. With the M segment removed as on main, added only after a name, or left out for members named M, L, S and 0, or with the second index's L segment left out, both properties failed on all 40 seeds run for each change, and on all 200 seeds run for the two changes that need an index inside the path.
• pushdown_parity.rs in a debug build: 0.41 s for its 20 tests, up from 0.27 s before the change (median of 7 runs).
• cargo +1.97.0 test -p extenddb-storage-mongodb -- --include-ignored against the replica set: the 147 unit and live tests, the 18 interpreter self-tests and the 20 harness tests passed. cargo +1.88.0 check -p extenddb-storage-mongodb --all-targets passed.
• cargo +1.97.0 fmt --all -- --check, cargo +1.97.0 clippy -p extenddb-storage-mongodb --all-targets -- -D warnings, and cargo test -p extenddb-storage-mongodb --lib condition (18 passed; with the M segment left out from the fourth element on, 17 passed and the typed-fields test failed).

CI runs the new file in run-integration, run-integration-sqlite, mongodb-pytest and cassandra-pytest.

Not changed here

A map member whose name holds U+0000 answers InternalServerError on MongoDB, in PutItem and in a condition on it, where the service accepts both. That is on main and unchanged here; #368 escapes such names in storage.

Files

condition.rs was 759 lines on main. The first commit moves its test module unchanged into condition_tests.rs, loaded with #[path] as backup_engine.rs loads its test files. condition.rs is now 505 lines. docs/design/13-storage-mongodb.md section 5.2 now says what the parity harness generates.

Merging

git merge-tree --write-tree is clean against #571 and #368, and against the other open PRs that touch crates/storage-mongodb. #571 counts AND, OR and NOT nesting; the M segment lengthens a field name and adds no nesting, so its count is unchanged.

[robinnsc]

Looks correct to me. On a local MongoDB I ran about 900 conditions (nested maps, a[0].b, a.b[1], deep mixes, literal and aliased names, members named M/L/S/0) as UpdateItem and DeleteItem against a stream-enabled table as the reference. Head matched on all of them; the build before the fix differed on 295. Plans stay IXSCAN {pk:1}, one key and one doc examined.

Two things outside the diff, both already on main, so probably for a separate issue rather than this PR:

• begins_with/contains with U+0000 in the operand returns InternalServerError ("Regular expression cannot contain an embedded null byte"), top-level or nested. #368 gates NUL in member names but not in the operand.
• A long operand does the same once the escaped pattern goes past about 32 KB (Mongo error 51091), e.g. 40,000 a or 20,000 .. Refusing pushdown in the analyzer for either case would send it to the in-Rust evaluator.

[robinnsc]

The description mentions member rows for x$, $x and a.b measured on the service, but I couldn't find them in this file. Might be worth adding them, maybe with a decoy member a next to the a.b one. As far as I can tell nothing covers the per-element ./$ check in unsafe_attribute_reason past the first element: changing it to elements.iter().take(1) passes every test. Before this PR a nested path never matched anyway, but now that it reaches real data, #m.#x with #x = "a.S" would compile to item_data.m.M.a.S if that check slipped. (Adding a.b to MEMBER_NAMES in the parity harness probably won't work, since that property compiles without going through is_pushable.)

[robinnsc]

Could consider having the MongoDB leg check that the compiled filter actually ran, e.g. a profiler entry with item_data. in its filter. Right now, if the cache TTL or the put's cache renewal changes, every row would still pass through the in-Rust evaluator, and the file would quietly stop covering #572. Not a big deal since it's dual-target, just a thought.