Repository navigation
fix(expressions): validate in the service's order, with its messages - #399
LeeroyHannigan wants to merge 17 commits into
Conversation
| // itself; whether that function may stand alone, whether its name exists, | ||
| // and how many operands it takes are semantic questions answered after | ||
| // the grammar pass (see `validate_condition_semantics`), so a syntax error | ||
| // further along still wins, as on Amazon DynamoDB. |
There was a problem hiding this comment.
What does "wins" mean in this context?
There was a problem hiding this comment.
"Wins" meant precedence between two errors in one expression, and the comment did not say so. The grammar pass in parser.rs reports every syntax error first; the semantic checks (function name, whether a function may stand alone, operand count) run afterwards in validate_condition_semantics (crates/core/src/expression/semantics/condition_rules.rs:31), over the whole parsed tree. So for foo(pk) AND the response is the syntax error at the end of the text, not the unknown function name that appears first.
The service has the same precedence. Measured 2026-10-23 on PutItem: foo(pk) AND, ATTRIBUTE_EXISTS(pk) AND, begins_with(pk, :n) AND (with :n a number) and attribute_exists(#x) AND (with #x undefined) each answer Invalid ConditionExpression: Syntax error; token: "<EOF>", near: "AND"; foo(pk) alone answers Invalid function name; function: foo. The four cases are pinned in TestConditionFunctions.test_a_syntax_error_beats_the_semantic_errors in tests/test_expression_semantics.py (343c2fb).
I rewrote the comment to say this without the word (crates/core/src/expression/parser.rs:136-144, and the same wording in update_parser/mod.rs:279-286), in a8d83c6.
|
|
||
| let mut args = Vec::new(); | ||
| if *pos < tokens.len() && tokens[*pos] != Token::RParen { | ||
| // `f()` is a syntax error on the service, reported at the `)`. |
There was a problem hiding this comment.
A syntax error because there are no arguments? Please clarify.
There was a problem hiding this comment.
Yes. The service's grammar needs at least one operand between the parentheses, so attribute_exists() and foo() are refused at the ) as a syntax error, Syntax error; token: ")", near: "()", before the function name or the operand count is looked at. Measured 2026-10-23 for both spellings on PutItem (ConditionExpression); the UpdateItem case SET x = foo() was measured 2026-10-14. One operand too few is a different error: begins_with(pk) parses and the semantic pass reports Incorrect number of operands for operator or function; operator or function: begins_with, number of operands: 1 (also measured 2026-10-23).
The check is at crates/core/src/expression/parser.rs:241-251 (and update_parser/mod.rs for SET operands); the wire cases are foo(), attribute_exists() and begins_with(pk) in TestConditionFunctions in tests/test_expression_semantics.py. I rewrote the comment to say this in a8d83c6.
| /// Per-table capacity totals in the order the tables first appear in the | ||
| /// request. | ||
| /// | ||
| /// Amazon DynamoDB's own order of the `ConsumedCapacity` entries of a |
There was a problem hiding this comment.
I can't tell if this is saying ExtendDB's behavior (in terms of sorting) is now equivalent to DynamoDB's, or if ExtendDB is providing firmer ordering guarantees than DynamoDB (and why).
There was a problem hiding this comment.
It is now the service's order. DynamoDB documents no order for the ConsumedCapacity entries of a transaction, so I measured it on 2026-10-23 with three tables created on 2026-10-22, 2026-10-23 11:01 and 2026-10-23 14:45 UTC, with names that sort opposite to the creation order. For the request orders [newest, middle, oldest], [middle, newest, oldest] and [oldest, newest, middle], TransactGetItems and TransactWriteItems both returned the tables oldest first. The order follows neither the request nor the names, it follows the tables' creation time.
PerTableCapacity (now crates/engine/src/per_table_capacity.rs) orders the entries by creation time, oldest first, with name order breaking a tie. TableKeyInfo carries the creation time from the catalog row the handlers already read for the key schema, so there is no extra catalog read. Before this PR the handlers aggregated into a HashMap and the order changed between processes and backends.
Two tables tie when their stored creation times are equal, and the precision of that value differs per backend: nanoseconds on SQLite, microseconds on PostgreSQL, milliseconds on MongoDB and Cassandra. MongoDB was writing whole seconds, so three tables created inside one second got the same value and came back in name order; that write now keeps the millisecond part. A catalog row written by an earlier build still holds whole seconds. tests/test_transact_capacity_order.py creates three tables whose names sort opposite to their creation order and asserts the order on both operations; it runs against the service and against ExtendDB, and it passes on SQLite, PostgreSQL and MongoDB. docs/differences-from-dynamodb.md has the row under Capacity and Throttling, with the precision per backend.
| }; | ||
|
|
||
| // Validate begins_with operand types upfront (before any rows are read). | ||
| if let Some(ref f) = filter { |
There was a problem hiding this comment.
Where are these checks happening now?
There was a problem hiding this comment.
The expression form moved them into the semantic pass. The legacy QueryFilter path lost them, which was a regression in this PR; it is fixed in 343c2fb and 4132394.
FilterExpression: the begins_with and ordering operand checks run inside the semantic pass, at request time, before the table lookup. The call is parse_condition_checked at crates/engine/src/query.rs:170, which runs validate_condition_semantics (crates/core/src/expression/semantics/condition_rules.rs:31); the begins_with operand type is at condition_rules.rs:131 and the ordering operand types at condition_rules.rs:54 and 60. The checks are in the walk because the service reports them in expression order with the other semantic errors (#x = :v AND begins_with(pk, :n) with an undefined #x reports the name, begins_with(pk, :n) AND #x = :v reports the operand type; both in TestConditionFunctions). Covered by tests/test_expr_operand_type_and_size.py and tests/python/test_begins_with_type_check.py.
QueryFilter (legacy): desugar_filter bypasses the parser, so the removal left that path with no request-time check; an empty table returned 200 and a non-empty one failed on the first row evaluated. The service checks each legacy condition at request time, before the table lookup, with its own messages. I measured the full operator by type grid on 2026-10-23: One or more parameter values were invalid: ComparisonOperator BEGINS_WITH is not valid for N AttributeValue type and the same text for every other refused pair (EQ and NE take every type; BEGINS_WITH takes S and B; CONTAINS and NOT_CONTAINS take S, N, B, BOOL and NULL; the ordering operators, BETWEEN and IN take S, N and B), plus the value count (Invalid number of argument(s) for the <op> ComparisonOperator), mixed types (AttributeValues inside AttributeValueList must be of same type), an inverted BETWEEN range and an unknown operator name. validate_legacy_condition in crates/engine/src/legacy_filter.rs now runs those checks on each QueryFilter, ScanFilter and KeyConditions entry before it is desugared, with the service's text; the Invalid FilterExpression: wording is gone. The grid is pinned in the engine's unit tests (legacy_filter_tests.rs) and one row per message on the wire in TestLegacyFilterOperandType in tests/test_expr_operand_type_and_size.py.
| } | ||
|
|
||
| // Validate begins_with operand types upfront (before any rows are scanned). | ||
| if let Some(ref f) = filter { |
There was a problem hiding this comment.
Similar question to query.rs: where are these checks happening now?
There was a problem hiding this comment.
The answer is the same as for Query. FilterExpression: inside the semantic pass through parse_condition_checked at crates/engine/src/scan.rs:197, before the projection, the unused check and the table lookup (scan.rs:300); the begins_with and ordering operand checks are at crates/core/src/expression/semantics/condition_rules.rs:131, 54 and 60.
ScanFilter (legacy): the check had been dropped. 343c2fb and 4132394 restore it in desugar_filter (validate_legacy_condition in crates/engine/src/legacy_filter.rs) with the service's messages, measured 2026-10-23 on an empty table for every operator and type: BEGINS_WITH with a number answers One or more parameter values were invalid: ComparisonOperator BEGINS_WITH is not valid for N AttributeValue type, LT with a boolean names BOOL, BETWEEN with a list then a number names L, and the value count, mixed types, inverted BETWEEN range and unknown operator get their own texts. The ScanFilter rows are in TestLegacyFilterOperandType in tests/test_expr_operand_type_and_size.py.
jcshepherd
left a comment
There was a problem hiding this comment.
Some of the agent-generated comments are read pleasantly but actually are ambiguous which makes it hard to determine whether the code is even solving a real problem. I flagged a couple. Also some operand validation code in query.rs and scan.rs was deleted and it's not obvious that it is still being done somewhere (assuming it was needed).
| /// | ||
| /// Returns `DynamoDbError::ValidationException` naming the first undefined | ||
| /// reference, names before values, in the order the expressions are given. | ||
| pub fn validate_refs_defined( |
jcshepherd
left a comment
There was a problem hiding this comment.
Dead code in resolver.rs: otherwise okay.
…ts messages Issue #108 lists 106 requests whose response differs from Amazon DynamoDB. Every expectation in this change was measured against the service (us-east-1, 2026-10-14); the probes and captured responses are in the issue's notes, and every new test file was run against the service before it was run here. Semantic validation of expressions. After the grammar pass, the service walks the parsed expression once, in expression order, checking each node after its operands, then checks overlapping paths, then unused names and values. A new `validate_update_semantics` and `validate_condition_semantics` in the core crate follow that walk; the parsers now do grammar only, keep function names as written, and record section order for the walk. - UpdateExpression: `+` and `-` take numbers, `list_append` takes lists, `ADD` takes a number or a set, `DELETE` takes a set; a placeholder of the wrong type is refused with the service's type vocabulary (`S`, `N`, `L` for operators and functions; `STRING`, `NUMBER`, `Binary`, `BOOL`, `NULL`, `LIST`, `MAP` for ADD and DELETE). Function names are case-sensitive: `IF_NOT_EXISTS(x, :v)` and `foo(x)` are `Invalid function name`, a condition function is `not allowed in an update expression`, `list_append` is refused as an operand of a function or of `+`, both functions take two operands, the first operand of `if_not_exists` is a document path. A repeated section is reported after that section's own actions (`SET a = :v SET b = :q` reports `:q`). Previously `IF_NOT_EXISTS`, `DELETE s :string`, and `SET a = :string + :n` were accepted and written. - ConditionExpression and FilterExpression: the six condition functions are matched as written (`ATTRIBUTE_EXISTS(pk)` was accepted and evaluated); `if_not_exists` and `list_append` are `not allowed in a condition expression`; a boolean function in operand position and `size` in condition position are `not allowed to be used this way`; operand counts are fixed per function; `attribute_exists` and `attribute_not_exists` take a document path; `begins_with` takes a string or binary and `attribute_type` a string naming a type, both checked at request time with the operation's prefix. - Every unused name or value is listed in one set, sorted, so the message is the same on every backend; it named one key chosen by hash order. Request stage on the read APIs. Query and Scan check the FilterExpression, then the KeyConditionExpression (with its `Invalid KeyConditionExpression:` prefix, and ahead of the key schema match, so an undefined `#n` is reported as such), then the ProjectionExpression, then unused names and values, all before the table is looked up. An undefined `:q` in a filter returned 200 on an empty table. Limit and Segment checks precede the expressions on Scan. Single-item order. UpdateItem checks the whole ConditionExpression before the UpdateExpression. PutItem, UpdateItem, and DeleteItem wrap every expression-stage error in `1 validation error detected: ` and omit the `: ConditionExpression is null` suffix; the read, batch, and transaction APIs return the messages bare. The ` expression size: N` suffix follows the operation. Three existing dual-target tests asserted the unprefixed text and fail against the service; they are corrected. Transactions. TransactGetItems and TransactWriteItems validate in phases, each over every item before the next: per item, expressions (UpdateExpression before ConditionExpression), unused names and values, then the item's 400 KB size; then table lookup; then the 4 MB total; then key schema, as a cancellation with ValidationError on the first offending item only and None on every other (it flagged every offender); then duplicates. An undefined reference is the request error the service returns rather than a per-item cancellation reason, and a key schema mismatch leaves other items' conditions unevaluated. TransactWriteItems returns `"ItemCollectionMetrics": {}` when SIZE is requested and no table has a local secondary index, and its ConsumedCapacity entries are in first-appearance order rather than per-process hash order. Key-shaped messages. GetItem, DeleteItem, UpdateItem, BatchGetItem, and BatchWriteItem (Put and Delete) refuse a key with a missing, mistyped, or extra attribute with `The provided key element does not match the schema`; PutItem keeps `Missing the key pk in the item` and `Type mismatch for key`. BatchWriteItem returns `"ItemCollectionMetrics": {}` under SIZE with no LSI. Table ARNs in `TableName` (eight #108 rows) stay unsupported by decision: an ExtendDB table ARN names ExtendDB's account and region, so a table that streams or replicates to Amazon DynamoDB would answer to two ARNs. docs/differences-from-dynamodb.md records this. Remaining #108 gaps after this change, all stated in the PR: the parser's syntax wording (`Syntax error; token: X, near: Y`), the ResourceNotFound message text, the 4 MB message (the service's carries its own payload size), the service's rendering of over-limit list values, and the table ARN rows. Testing: tests/test_transact_validation_order.py (31 cases), tests/test_update_expression_rules.py (37), and tests/test_expression_semantics.py (155) pass against the service and on SQLite, PostgreSQL, and MongoDB; 164 of the 186 cases in the first and third fail on main. The expression, validation, transaction, batch, query, and scan suites pass on all three backends (767 on SQLite and PostgreSQL, 751 on MongoDB with the four #387 cases unchanged). Of the 106 #108 rows, 83 now return the service's status, error type, and message (30 on main). cargo test --workspace 1,500 passed; fmt; clippy -D warnings on 1.97.0; cargo +1.88.0 check. Closes part of #108.
… three comments The move of the FilterExpression operand-type checks into the semantic pass dropped them for the legacy QueryFilter and ScanFilter parameters, which bypass the parser. A ScanFilter with BEGINS_WITH and a number, or LT and a boolean, was accepted at request time and refused only when a row was evaluated, so an empty table returned 200. The desugared legacy filter now runs the same begins_with and ordering operand checks as before, with the FilterExpression prefix, before any row is read. Wire tests cover both parameters on an empty table. Comments reworded after review: the parsers state that grammar errors are reported before semantic errors and why the semantic checks are deferred; the empty operand list comment names the measured cases; the PerTableCapacity doc states that first-appearance order is a firmer guarantee than the service gives, with the measured orders.
…'s message A ScanFilter or QueryFilter condition whose AttributeValueList holds a value of a type its ComparisonOperator does not take is refused at request time with the service's text, `One or more parameter values were invalid: ComparisonOperator <op> is not valid for <type> AttributeValue type`, naming the first such value. Measured against Amazon DynamoDB on 2026-10-23 on an empty table: BEGINS_WITH with N, LT and GT with BOOL, and BETWEEN with L then N, which names the L. The check runs in desugar_filter on each condition before it is desugared; the `Invalid FilterExpression:` wording that the previous commit restored is gone. attribute_type_code is exported from extenddb-core for the type tag. Pin the condition-expression order for a syntax error after a semantic error: `foo(pk) AND`, `ATTRIBUTE_EXISTS(pk) AND`, `begins_with(pk, :n) AND` and `attribute_exists(#x) AND` each answer the syntax error on the service (measured 2026-10-23), and the test asserts that no semantic message is returned. Document the order of ConsumedCapacity entries in transaction responses. With two tables the service returned the same table first for both request orders (measured 2026-10-23), so the order does not follow the request; ExtendDB keeps first appearance in the request on every backend. No wire test asserts a multi-table order.
… and messages Measured the full ComparisonOperator by AttributeValue type grid against Amazon DynamoDB on 2026-10-23 for ScanFilter, QueryFilter and KeyConditions, with the value counts, mixed-type lists, inverted BETWEEN ranges and unknown operator names. validate_legacy_condition in legacy_filter.rs now applies the measured rules to every legacy condition before it is desugared and before the table is looked up: the operator name (`Invalid comparison operator`), the value count (`Invalid number of argument(s) for the <op> ComparisonOperator`), the first value's type (EQ and NE take every type; BEGINS_WITH takes S and B; CONTAINS and NOT_CONTAINS take S, N, B, BOOL and NULL; LE, LT, GE, GT, BETWEEN and IN take S, N and B), that every value has the first value's type (`AttributeValues inside AttributeValueList must be of same type`), and that a BETWEEN lower bound is not above its upper bound. Each message carries the `One or more parameter values were invalid: ` prefix. KeyConditions get the same checks ahead of the table lookup, and a range key operator the service does not index (NE, NULL, NOT_NULL, CONTAINS, NOT_CONTAINS, IN) is refused with `Attempted conditional constraint is not an indexable operation`. The engine's own count messages for these parameters are gone. compare_orderable is exported from extenddb-core for the range check. The grid is pinned in legacy_filter_tests.rs and one wire row per message in TestLegacyFilterOperandType. The ConsumedCapacity order note now cites the three-table measurement: 16 transactions in eight request orders all listed the tables in name order.
Amazon DynamoDB documents no order for the ConsumedCapacity entries of TransactGetItems and TransactWriteItems. Measured on 2026-10-23 with three single-hash-key tables whose names sort opposite to their creation order, in three request orders: both operations listed the tables by creation time, oldest first, whatever order the request named them in. TableKeyInfo now carries the table's creation time. Every backend reads it from the catalog row or document it already fetches for the key schema (SQLite and PostgreSQL creation_date_time, MongoDB creation_date_time, Cassandra created_at), so no request makes another catalog read. PerTableCapacity moves from capacity_helpers.rs to per_table_capacity.rs, takes the TableKeyInfo of each operation, and lists tables oldest first, with name order breaking a tie on the same creation time. Both transaction handlers use it, including the idempotent replay path of TransactWriteItems, which now has a key_info accessor on PreparedOp. tests/test_transact_capacity_order.py creates three tables whose names sort opposite to their creation order, waiting for ACTIVE between them, and asserts the order for both operations at TOTAL and INDEXES over the three measured request orders; it runs against the service and against ExtendDB. The row in docs/differences-from-dynamodb.md states the measured order and ExtendDB's tie rule.
create_table wrote the catalog creation time as whole seconds, so two tables created inside the same second carried the same value. The transaction ConsumedCapacity order sorts tables by that value and falls back to name order on a tie, which the service does not do, and tests/test_transact_capacity_order.py failed 5 of 6 on this backend. The write now keeps the millisecond part, which is the precision bson::DateTime stores. Every reader already converts through timestamp_millis, so DescribeTable, ListTables, the backup and restore paths and the transaction key-info fetcher return the sub-second value, the same as the PostgreSQL backend. Catalog rows written by earlier builds keep whole seconds and can still tie.
The ConsumedCapacity order row said two tables created in the same millisecond are listed in name order. The precision of the stored creation time differs per backend, so the tie does too: nanoseconds on SQLite, microseconds on PostgreSQL, milliseconds on MongoDB and Cassandra, and whole seconds for a MongoDB catalog row written before the precision fix. The row now says which, and names the backends the order was measured on. The service-side measurement no longer claims the three tables had a single hash key, which the recorded run does not state. Same wording in the test docstring and the PerTableCapacity doc comment. The empty-operand-list comment in the condition parser now cites the 2026-10-23 measurement as well as the 2026-10-14 one.
semantics.rs had grown to 819 lines. It is now a module directory with one file per rule family: update_rules.rs for the update-expression rules, condition_rules.rs for the condition-expression rules, refs.rs for the name and value usage checks both passes share, errors.rs for the message builders, and tests.rs for the tests. mod.rs keeps the module documentation, the two function-name tables and the two public entry points. No behaviour change. The functions moved unedited; the only edits are the per-file imports, pub(super) on the helpers the siblings call, and the test module's imports in place of its use super::*. The nine semantic-pass tests keep their names and pass.
update_parser.rs had grown from 571 lines to 630. It is now a module directory: mod.rs holds the parser at 417 lines and tests.rs holds the 21 tests at 215 lines. No behaviour change. The test module moved unedited, keeping its use super::*, and the tests keep their names. One comment inside it lost an em dash.
The update-parser ADD/DELETE doc comment and the scan Segment check comment carried an em dash. Both are now two clauses.
TransactWriteItems looked the tables up one item at a time and ran four checks inside that loop, so the first item's defect hid a missing table on a later item and the answer changed with the item order. The service reports them in two places, measured with the items in each order. Before any table is looked up, in the pass that also checks the expressions: the nesting of every ExpressionAttributeValues value of the item, used or not, on every kind of item (worded "ExpressionAttributeValues contains invalid value: ... for key :v"), and for a Put the attribute name size, then the item size, then the item nesting. After every table is looked up and after the 4 MB total, in one pass over the items in order: a key that does not fit the schema, then a key attribute named in an Update (a ValidationError cancellation reason on that item, not a top-level error), then an empty key value (a top-level ValidationException). The first failing item decides. The empty value rule is left out of the key shape check this pass uses, so a mistyped range key is reported ahead of an empty hash key. tests/test_transact_write_phase_order.py pins each placement with the items in both orders, and the order of the checks inside one item.
check_key_condition_refs collected the references of a parsed KeyConditionExpression into sets, sorted them, and named the smallest name, then the smallest value. With two undefined references the message named an alphabetical pick, and any name beat any value. The service names the first reference in the text, names and values alike: "#b = :p AND #a > :q" names #b, "#a = :p AND #b > :q" names #a, "pk = :q AND #n = :v" names :q, and "pk = :z AND sk > :y" names :z. parse_key_condition_expr now takes the request's expression maps and walks the tokens in order once the grammar pass succeeds, so the order is the text's. The separate check function and its call in Query are gone. validate_refs_defined had no caller and documented an order it did not implement (it sorted both lists); it is removed with its re-export. tests/test_request_stage_checks.py pins the five expressions above on the wire.
Four paths gave a text or an order the service does not.
A KeyConditionExpression clause with an identifier followed by "(" always
got "Invalid function name". The service answers by name. contains,
attribute_exists, attribute_not_exists and attribute_type get "Invalid
operator used in KeyConditionExpression" with no function name, on the sort
key and on the hash key. size gets "KeyConditionExpressions cannot contain
nested operations". Any other name, begins_with in another case included,
gets "Invalid function name; function: <name>". The text comes from
key_condition_function_refusal, next to the function name lists.
Scan answered "Value '0' at 'limit' ..." for a Limit under 1. The service
answers "Value at 'Limit' failed to satisfy constraint: Member must have value
greater than or equal to 1" for Scan as it does for Query, on an existing and
on a missing table, ahead of every expression error. Query already did.
An ADD with a string, map or boolean value, and a DELETE with a value that is
not a set, sent as legacy AttributeUpdates, went through the UpdateExpression
semantic pass and came back as "Invalid UpdateExpression: Incorrect operand
type ..." for a request that carried no expression. The service answers "1
validation error detected: One or more parameter values were invalid: ADD
action is not supported for the type S" and "... DELETE action with value is
not supported for the type N", ahead of the table lookup and key validation.
attribute_updates.rs checks the value types before the desugaring, in
attribute name order.
An empty ConditionExpression on a ConditionCheck was dropped as "no
condition" and then refused with a text the service does not use. The service
answers "Invalid ConditionExpression: The expression can not be empty;". Put,
Delete and Update still treat an empty string as no condition.
tests/test_request_stage_checks.py pins each of them on the wire, and unit
tests cover the function name texts and the value type texts.
Four comments described code that had moved or never worked that way.
scan.rs said Segment, Limit and Select are checked after the expressions.
Limit and the Segment range checks run first.
transact_get_items.rs said a ValidationError reason per offending item. Only
the first offending item carries one; every other item reports None.
storage-sqlite ddl.rs said the creation time column holds milliseconds,
while per_table_capacity.rs and the differences table said nanoseconds on
SQLite. CreateTable writes RFC 3339 with up to nanosecond digits, and the
value the transaction handlers order by is an f64 of epoch seconds, which
resolves about 0.24 microseconds. All three now say that.
The comments in transact_write_items.rs that called an empty or oversize key
an up-front top-level error went with the code they sat on in the commit that
moved the key checks.
validate_unused_attributes sorts the unused names and values so the message
is the same on every backend and process, and no test failed without the
sort. The new unit test builds fresh maps twenty times, each iterating in a
fresh random order, and expects {:a, :m, :z} and {#a, #m, #z}.
The first commit of this series checked every ExpressionAttributeValues value of a transaction item against the item's nesting limit. That refused a ConditionCheck value 32 wrappers deep, which the service accepts (an existing test pins it). The service accepts 32 wrappers in an ExpressionAttributeValues value and refuses 33, as a top-level ValidationException worded "ExpressionAttributeValues contains invalid value: ... for key :v". validate_expression_value_nesting_depth applies that threshold. A value an Update stores with SET becomes an attribute of the item, so 32 wrappers pass that check and then break the item limit. The service answers a TransactionCanceledException with a ValidationError reason "Nesting Levels have exceeded supported limits" on every such item, after the lookup, the 4 MB total, the key pass and the duplicate check. stored_value_nesting_reasons reports it there. Before, the same value was refused as a top-level ValidationException inside the table lookup loop, so a missing table on a later item was hidden. The service also reports the other items' condition results next to this reason; ExtendDB reports None for them, because no condition has been evaluated at that point. key_pass_failure now returns the whole list of reasons, so the handler no longer builds it. tests/test_transact_value_nesting.py pins the 31, 32 and 33 wrapper boundaries for each place a value can sit and the position of the reason among the other checks.
…essions Scan checked that Segment and TotalSegments go together, and that Segment is below TotalSegments, ahead of every expression check. On 2026-10-27 the service answered an undefined filter value, an unused value and an undefined projection name ahead of those three mismatches, and a missing table after them. Only the range checks on Segment, TotalSegments and Limit come first. scan.rs keeps the range checks where they were and moves the three pairing checks between the unused names and values check and the table lookup. Their texts are unchanged. The Segment assertion in test_parameter_checks_come_before_the_expressions expected the TotalSegments message ahead of an undefined filter value, which fails against the service; it now expects the filter error. TestScanSegmentChecks in tests/test_request_stage_checks.py pins the messages, the position after the filter, the unused values and the projection, the position ahead of the table lookup, and that Limit still comes first.
… reason cover The Scan comment said Select stays after the existence check; only the Select checks that need the index do, and validate_select_projection runs ahead of the table lookup. The transact_write_helpers header cited a 500-line limit that both files are over; the limit is 600. The doc comment on stored_value_nesting_reasons read as if ExtendDB reported the other items' condition results beside the nesting reason as the service does; it reports None for them, and the differences document now carries the row (measured 2026-10-27).
17b805d to
1454cc9
Compare
…e service does Legacy QueryFilter, ScanFilter and KeyConditions values are now held to the service's rules that need the key schema, and to its number check, on every backend (issue #477, stacked on #399): - a legacy N value that is not a number answers the number message before the operator, count, type and range checks on that entry, inside NS, L and M too; - a KeyConditions value whose type is not the key's AttributeDefinitions type answers "Condition parameter type does not match schema type" (the range key answered 500 before, the hash key 200); - a non-indexable operator on any KeyConditions entry, the hash key included, is refused at request time; - an empty string on a key through KeyConditions is refused with the shared key message, hash key first, after the schema checks; - a QueryFilter entry on a key of the queried table or index is refused after every other Query check; - BatchGetItem charges each absent key the minimum read and lists every requested table in ConsumedCapacity, in table name order. Which entry the service names when several legacy entries are invalid is not a function of the request; the differences document records it.
What
Issue #108 lists 106 requests whose response differs from Amazon DynamoDB. This change makes ExtendDB validate expressions and transactions in the service's order and return the service's messages for the groups below. Every expectation was measured against the service (us-east-1, 2026-10-14), and every new test file was run against the service before it was run against ExtendDB.
Semantic validation of expressions. After the grammar pass, the service walks the parsed expression once, in expression order, checking each node after its operands, then checks overlapping paths, then unused names and values. New
validate_update_semanticsandvalidate_condition_semanticsincrates/core/src/expression/semantics/(one file per rule family) follow that walk. The parsers now do grammar only, keep function names as written, and record section order for the walk.+and-take numbers,list_appendtakes lists,ADDtakes a number or a set,DELETEtakes a set. A placeholder of the wrong type is refused with the service's type vocabulary (S,N,Lfor operators and functions;STRING,NUMBER,Binary,BOOL,NULL,LIST,MAPfor ADD and DELETE). Function names are case-sensitive:IF_NOT_EXISTS(x, :v)andfoo(x)areInvalid function name, a condition function isnot allowed in an update expression,list_appendis refused as an operand of a function or of+, both functions take two operands, and the first operand ofif_not_existsis a document path. A repeated section is reported after that section's own actions, soSET a = :v SET b = :qreports:q. Before this changeIF_NOT_EXISTS(x, :v)was accepted and written, andDELETE s :stringwas accepted whensdid not exist.SET a = :string + :nwas already refused when it was evaluated (An operand in the update expression has an incorrect data type); it is now refused before the table is read, with the service's text.ATTRIBUTE_EXISTS(pk)was accepted and evaluated);if_not_existsandlist_appendarenot allowed in a condition expression; a boolean function in operand position andsizein condition position arenot allowed to be used this way; operand counts are fixed per function;attribute_existsandattribute_not_existstake a document path;begins_withtakes a string or binary andattribute_typea string naming a type, both checked at request time with the operation's prefix.Legacy conditions. ScanFilter, QueryFilter and KeyConditions bypass the expression parser, so they get their own request-time check, ahead of the desugaring and the table lookup, with the rules measured against the service on 2026-10-23 for every ComparisonOperator and AttributeValue type: the operator name (
Invalid comparison operator), the value count (Invalid number of argument(s) for the <op> ComparisonOperator), the first value's type (ComparisonOperator <op> is not valid for <type> AttributeValue type), that every value has the first value's type, and that a BETWEEN lower bound is not above its upper bound, each with theOne or more parameter values were invalid:prefix. A range key operator the service does not index (NE, NULL, NOT_NULL, CONTAINS, NOT_CONTAINS, IN) in KeyConditions is refused withAttempted conditional constraint is not an indexable operation. Before this change a ScanFilter with BEGINS_WITH and a number returned 200 on an empty table.Request stage on the read APIs. Query and Scan check the FilterExpression, then the KeyConditionExpression (with its
Invalid KeyConditionExpression:prefix and ahead of the key schema match, so an undefined#nis reported as such), then the ProjectionExpression, then unused names and values, all before the table is looked up. An undefined:qin a filter returned 200 on an empty table. Limit and Segment checks precede the expressions on Scan. AKeyConditionExpressionfunction name is answered by name:contains,attribute_exists,attribute_not_existsandattribute_typegiveInvalid operator used in KeyConditionExpressionwith no function name,sizegivesKeyConditionExpressions cannot contain nested operations, and any other name,Begins_Withincluded, givesInvalid function name; function: <name>. The first undefined#nameor:valuein the text is the one named, names and values alike. ALimitunder 1 on Query and Scan gives1 validation error detected: Value at 'Limit' failed to satisfy constraint: Member must have value greater than or equal to 1, ahead of the table lookup and every expression. On Scan, the range checks onSegmentandTotalSegmentscome first, and the checks that the two go together and thatSegmentis belowTotalSegmentscome after the expressions and the unused names and values and before the table lookup.Single-item order. UpdateItem checks the whole ConditionExpression before the UpdateExpression. PutItem, UpdateItem, and DeleteItem wrap every expression-stage error in
1 validation error detected:and omit the: ConditionExpression is nullsuffix; the read, batch, and transaction APIs return the messages bare. Theexpression size: Nsuffix follows the operation (present on a Scan FilterExpression and a transaction ConditionExpression, absent on a Query FilterExpression and the single-item ConditionExpression). Eleven assertions in three existing dual-target test files asserted the unprefixed text and fail against the service (test_expression_error_prefix.py3,test_expression_param_usage.py6,test_expression_size_limit.py2); they are corrected, and one assertion is added to the third file. LegacyAttributeUpdateswith anADDof a value that is not a number or a set, or aDELETEwith a value that is not a set, are refused with1 validation error detected: One or more parameter values were invalid: ADD action is not supported for the type S(andDELETE action with value is not supported for the type N), ahead of the table lookup and key validation. AConditionCheckwith an emptyConditionExpressiongivesInvalid ConditionExpression: The expression can not be empty;.Transactions. TransactGetItems and TransactWriteItems validate in phases, each over every item before the next. TransactWriteItems checks, one item after another and before any table is looked up, the nesting of the item's
ExpressionAttributeValues(32 wrappers pass, 33 do not), its expressions (UpdateExpression before ConditionExpression), the unused names and values, and for a Put the attribute name size, the 400 KB item size and the item nesting. Then it looks the tables up, checks the 4 MB total, and makes one pass over the items in order that stops at the first failing item: a key that does not fit the schema, then a key attribute named in an Update (both a cancellation withValidationErroron that item andNoneon every other item), then an empty key value (a top-level ValidationException). Duplicate targets come next, and last an Update that stores a value nested as deep as an item may not hold, which is aValidationErrorreason on every such item. TransactGetItems checks the projections, looks the tables up, checks the key schema (first offending item only) and then duplicate keys. An undefined reference is the request error the service returns rather than a per-item cancellation reason, and a key schema mismatch leaves the other items' conditions unevaluated. TransactWriteItems returns"ItemCollectionMetrics": {}when SIZE is requested and no table has a local secondary index, and the ConsumedCapacity entries of both transaction APIs are ordered by table creation time, oldest first, as measured on the service (section below).Key-shaped messages. GetItem, DeleteItem, UpdateItem, BatchGetItem, and BatchWriteItem (Put and Delete) refuse a key with a missing, mistyped, or extra attribute with
The provided key element does not match the schema. PutItem keepsMissing the key pk in the itemandType mismatch for key. BatchWriteItem returns"ItemCollectionMetrics": {}under SIZE with no LSI.Table ARNs in
TableName(eight #108 rows) stay unsupported by decision: an ExtendDB table ARN names ExtendDB's account and region, so a table that streams or replicates to Amazon DynamoDB would answer to two ARNs.docs/differences-from-dynamodb.mdrecords this. Open PR #200 takes the opposite position and needs reconciling before either merges.This is one PR with three pieces that can be read on their own. The legacy condition validator (
legacy_filter.rs,legacy_filter_tests.rs, and the 79 wire cases intests/test_expr_operand_type_and_size.py) is in aef6e1b, 475465a and 90fdc32. The ConsumedCapacity order is in 68dcd0a, 217b5f9 and b7ba331: thecreation_date_timefield onTableKeyInfo, the column read in the SQLite, PostgreSQL and Cassandra storage crates, the MongoDB catalog write that keeps milliseconds, andtests/test_transact_capacity_order.py. The decision to keep table ARNs inTableNameunsupported is one row indocs/differences-from-dynamodb.md, added in 29229c2 with the semantic pass. Commits ac76c66 and 964f3c3 only move code: the semantic pass split by rule family, and the update parser tests into their own file. The six commits from af5f681 to cbb04f0 change the transaction phases, the Query key condition checks, four request stage texts, the Scan parameter order, and add tests for each.Why
Closes part of #108. A client that maps on the service's error codes and messages, or that reads
CancellationReasons,ItemCollectionMetrics, orConsumedCapacity, saw different responses from ExtendDB on these requests, and request shapes the service refuses were accepted: a function name in the wrong case, aDELETEof a string on an absent attribute, a repeated update section, overlapping update paths, a nestedlist_append, and anif_not_existswhose first operand is not a path.Of the 106 rows in #108, 83 returned the service's status, error type, and message (30 on
main) when measured on this branch on 2026-10-14, before the legacy condition work above. The 24 that remained are in these groups:Syntax error; token: X, near: Y; 14 rows);ResourceNotFoundmessage text (3 rows);1 validation error detectedmessage (2 rows: 101 transaction items, 26 batch writes).The 4 MB transaction message also remains: the service's text carries its own serialized payload size.
Testing done
tests/test_expression_semantics.py(new, 159 cases): operand types, function names, the walk order, the unused set, condition function rules, Query and Scan request-stage checks, key-shaped messages, BatchWriteItemItemCollectionMetrics.tests/test_transact_validation_order.py(new, 31 cases): the transaction phases, first-offender cancellation, item size and 4 MB positions.tests/test_update_expression_rules.py(new, 37 cases): repeated sections, operands, overlap, the single-item prefix. Every expected response was captured from the service; all three files pass against the service and on SQLite, PostgreSQL, and MongoDB. 164 of the 186 cases in the first two fail onmain.test_transact_absent_item_isolationcases of fix(postgres): lock missing items that TransactWriteItems checks or deletes #387 unchanged). The remaining full-suite failures on a dev-mode SQLite server (metrics, import and export, gsi async, misc operations) are identical onmainand need infrastructure the dev server lacks.semantics/tests.rs(10) pin the walk order and the per-node rules inside one expression. The orders between expressions and stages (condition before update, the Query stages, the transaction phases) are pinned by the wire tests. At 4f4fc21:cargo test --workspace2,021 passed on stable; on 1.88.0, 1,784 passed with the Cassandra crate excluded, because the local Cassandra node was held by another workload (that crate passed on stable in the same run);cargo fmt --checkandcargo clippy --all-targets -- -D warningson 1.97.0 clean.tests/test_transact_capacity_order.py(new, 6 cases) and the legacy condition cases intests/test_expr_operand_type_and_size.pyandtests/test_expression_param_usage.py: measured on the service on 2026-10-23. The eight test files this PR adds or changes pass on MongoDB at 4f4fc21 (391 passed), including the ordering test with tables created within one second of each other.main, one client, two runs (p50): TransactWriteItems with 100 Puts 23.8 to 23.9 ms against 24.0 to 24.1; UpdateItem with 50 SET actions 1.48 to 1.50 ms against 1.47; PutItem with a 20-clause condition 1.17 to 1.19 ms against 1.18; an UpdateItem with an invalid expression 0.88 ms against 0.88 to 0.89.tests/test_transact_write_phase_order.py(new, 16 cases),tests/test_request_stage_checks.py(new, 30 cases) andtests/test_transact_value_nesting.py(new, 6 cases) pin the transaction check order, the key condition, Limit, Segment, legacyAttributeUpdatesand empty expression texts, and the value nesting thresholds. Every expectation was measured on the service on 2026-10-27, two samples each, and all 52 cases pass against the service and against a SQLite dev-mode server. 861 tests pass on the dev-mode server across the transaction, expression, query, scan, item, batch and update files. At 95e90f8,cargo test -p extenddb-core -p extenddb-enginepasses (530 and 187),cargo fmt --checkandcargo clippy --all-targets -- -D warningson 1.97.0 are clean for core, engine and storage-sqlite, andcargo checkon 1.88.0 passes for core and engine. The workspace totals quoted in the next bullet were not re-run.ConsumedCapacity order per backend
TransactGetItems and TransactWriteItems return one ConsumedCapacity entry per table, ordered by the table's creation time, oldest first. Two tables with the same stored creation time are listed in name order.
The precision of the stored creation time differs per backend, so the point at which two creates tie differs with it:
f64of epoch seconds.Measured on SQLite, PostgreSQL and MongoDB. Cassandra reads the same catalog column, and the six cases of
tests/test_transact_capacity_order.pypass on it in the Cassandra integration job at 4f4fc21.Known differences
These are differences from the service on the legacy condition paths that this PR does not change, measured against the live service on 2026-10-23 and filed as #477. Since #429, main answers items 5 and 7 with the service's message for a table's own key attributes; the other six are present on main.
Measured against the live service on 2026-10-27 and present on main:
ConditionExpressionon a transaction Put, Delete or Update, and on PutItem, UpdateItem or DeleteItem: the service answersInvalid ConditionExpression: The expression can not be empty;, ExtendDB treats it as no condition.Attribute name is too large, must be less than 65536 bytes, ExtendDB words it differently. The boundary agrees.ADDof an empty list: the service accepts it, ExtendDB refuses it.SegmentandTotalSegmentsrange checks: the service answersValue at 'TotalSegments' failed to satisfy constraint: ...andValue at 'Segment' ..., ExtendDB uses another wording, and its text for aSegmentat or aboveTotalSegmentsends with the two values.ExpressionAttributeValuesand no expression that uses them: the service answersExpressionAttributeValues can only be specified when using expressions, ExtendDB adds: FilterExpression is null.ValidationErrornext toConditionalCheckFailed), and withNonefor them on ExtendDB.Rebase on main
Rebased onto
mainat f49c5f8 on 31 October. The branch keeps its 17 commits in the same order; the merge commit f9355f5 is gone, and the first commit carries what that merge had resolved (table_key_info_uncachedintransact_get_items.rs, the imports intransact_write_items.rs, and the table ARN row of the differences document). The commit ids in the What section are the ones on this head. The ids in Testing done and in the ConsumedCapacity section (4f4fc21, 95e90f8) name the heads those runs were made at, before the rebase.The main PRs this branch was resolved against, and what changed in its code because of them:
crates/core/src/expression/mod.rs: fix(auth): bind LeadingKeys for a refused Query's partition operand #511 widened thekey_conditionre-export toKeyClauseOperand,KeyCondition,SortKeyCondition,key_condition_clause_operandsandparse_key_condition, and this branch addscompare_orderableto theevaluatorre-export. The file keeps both lines. fix(auth): bind LeadingKeys for a refused Query's partition operand #511's rule is unchanged for every key condition that parses.begins_withwritten in another case no longer parses, so it binds no LeadingKeys, as for any condition that does not parse.crates/storage-cassandra/tests/direct/stream_ownership.rs: fix(streams): resolve the shard against its stream inside the GetRecords read #543 added this test after this branch gaveTableKeyInfoitscreation_date_timefield, so its oneTableKeyInfoliteral did not build. The commit that adds the field (68dcd0a) setscreation_date_time: 0.0there, as it does instreams.rs.crates/engine/src/query.rs: fix(query): classify and order key conditions on a multi-attribute GSI the way the service does #429 addedcheck_key_value_typeandrefuse_begins_with_on_numberabovevalidate_name_refs_in_expr. This branch removesvalidate_name_refs_in_expr, because its one call, the FilterExpression name check, is replaced by the semantic pass thatparse_condition_checkedruns on the filter. The file keeps fix(query): classify and order key conditions on a multi-attribute GSI the way the service does #429's two functions. fix(query): classify and order key conditions on a multi-attribute GSI the way the service does #429's key condition classification inhandle_querysits where this branch moved the key condition handling, after the table and index lookup and before the Select checks.git range-diff 9d132022..17b805da f49c5f82..1454cc93: the first commit differs by the former merge's resolutions and by #429's lines in thehandle_querycode it moves and above the function it removes, and the fifth by thestream_ownership.rsline above. The other fifteen carry the same diff.At 1454cc9:
cargo fmt --all --checkandcargo clippy --locked --workspace --all-targets -- -D warningson 1.97.0 clean;cargo check --locked --workspaceon 1.88.0 passes;cargo test --locked --libpasses for core (726), engine (192) and server (237), #429's 20 unit tests inkey_condition.rsandleading_keysamong them. The eleven test files this PR adds or changes, plustests/test_fgac_leading_keys.pyand #429'stests/test_gsi_multi_range_query.py, pass on a SQLite server and on a PostgreSQL server built from this head: 537 cases passed on each, and one case oftests/test_gsi_multi_range_query.pyskipped, the one that runs only on a backend that refuses a condition on a later RANGE attribute. Against Cassandra 4.1 with password authentication, as the Cassandra integration job runs them,extenddb-storage-cassandra's unit tests (75) and direct storage tests (181) pass.CI passed all 35 checks on
1454cc932.Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).ADR / RFC: n/a. The change alters wire-visible error text and response members toward the live service, which CONTRIBUTING lists under the RFC trigger; it is a parity fix measured against the service rather than a protocol proposal. A maintainer's call on whether an ADR is wanted.
Breaking changes
Requests the service refuses and ExtendDB accepted are now refused: function names in the wrong case in an UpdateExpression or a condition, a
DELETEof a non-set value on an absent attribute, a repeated update section, overlapping update paths, a nestedlist_append, anif_not_existswhose first operand is not a path, and an undefined:valueor#namein a Query or Scan filter (previously a 200 when no row was evaluated). Requests that evaluation already refused are now refused before the table is read, with the service's text:+or-on a non-numeric placeholder,ADDof a wrong type, andlist_appendof a non-list. A Scan withSegmentand noTotalSegmentsnow reports an expression error ahead of the missing parameter, as the service does. Error messages for these groups change to the service's text. Key mismatch messages on the key-shaped APIs change toThe provided key element does not match the schema.ItemCollectionMetricsis present as{}on TransactWriteItems and BatchWriteItem under SIZE with no LSI. TheConsumedCapacityarray of both transaction APIs is ordered by table creation time. ScanFilter, QueryFilter and KeyConditions conditions the service refuses are refused at request time with the service's messages.By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.