Skip to content

fix(query): classify and order key conditions on a multi-attribute GSI the way the service does - #429

Open
LeeroyHannigan wants to merge 2 commits into
mainfrom
fix/gsi-multi-range-query
Open

LeeroyHannigan wants to merge 2 commits into
mainfrom
fix/gsi-multi-range-query

Conversation

@LeeroyHannigan

@LeeroyHannigan LeeroyHannigan commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #381.

What the service does

Measured on 2026-10-21 against live DynamoDB with the issue's schema (GSI with HASH h and RANGE a, b, c, d) and a two-HASH index:

  • Results are ordered over every RANGE attribute in key-schema order, then the base table key; ScanIndexForward = false returns the exact mirror. LastEvaluatedKey carries every key attribute.
  • A key condition is served when every HASH attribute has an equality and the RANGE conditions cover a leading run of the RANGE attributes, with an equality on every one before the last; the last may use <, <=, >, >=, BETWEEN or begins_with. Clause order is free.
  • Refusals, in the order the service applies them: a second condition on one key attribute (KeyConditionExpressions must only contain one condition per key); more conditions than key attributes (The number of query conditions (6) exceeds the number of key attributes defined in the schema (5)); a HASH attribute without an equality (HASH key attributes h1, h2 must have equality conditions specified in the query); a RANGE gap (RANGE key attributes a must have equality conditions specified in the query because a condition is present on key attribute b); a value whose type differs from the attribute definition (One or more parameter values were invalid: Condition parameter type does not match schema type); a condition on an attribute outside the key schema, named by the first RANGE attribute without a condition (Query condition missed key schema element: a). begins_with on a number attribute is Incorrect operand type for operator or function; operator or function: begins_with, operand type: N.

What ExtendDB did

KeyCondition::resolve_multipart returned early for one HASH attribute, so the parser's assignment stood: results were ordered by the first RANGE attribute then the table key, every key condition with three or more clauses was refused as Query condition missed key schema element: h, and h = :h AND b = :b was accepted and returned nothing.

Changes

  • crates/core/src/expression/key_condition.rs: resolve_multipart classifies every clause by name and applies the rules above in the service's order. resolve_multipart_with takes a per-clause value check, which the engine supplies with the attribute definitions, so a wrong-typed value is reported after the structural rules and before the non-key rule, as measured.
  • crates/engine/src/query.rs: the multi-attribute path calls the classifier with the type check; the single-attribute path runs the same type check ahead of its missing-key check (a wrong-typed sort key value answered 500 before). begins_with on a number attribute is refused with the measured wording.
  • crates/engine/src/legacy_filter.rs: KeyConditions on a multi-attribute schema go through the same classifier.
  • crates/storage-sqlite and crates/storage-postgres: every RANGE condition is applied to its column; ORDER BY and the row-comparison cursor cover all RANGE columns.
  • crates/storage/src/util/key.rs, crates/storage-mongodb, crates/storage-cassandra: these two backends filter the first RANGE attribute of such an index only. A condition on a later RANGE attribute is refused with a ValidationException naming the attribute and the backend, instead of being applied to the wrong column.
  • crates/storage-cassandra/src/data/query.rs: on main, every Query on a Cassandra index with two or more RANGE attributes answers 500 (measured on ec81e9d4: hash-only, first-RANGE equality and first-RANGE range conditions all return Order by currently only supports the ordering of columns following their declared order in the PRIMARY KEY from Cassandra). The index table clusters on every RANGE column ahead of base_pk, so the ORDER BY now names every index RANGE column in declared order; those queries answer 200 with the page in clustering order. A Query with an ExclusiveStartKey on such an index also answered 500 and is now refused with a ValidationException (ExclusiveStartKey on index gsi: the Cassandra backend pages a Query over an index with one RANGE attribute only), because the cursor compares the first RANGE attribute only. docs/differences-from-dynamodb.md records both backends.

Verification

tests/test_gsi_multi_range_query.py, 60 cases, every expectation measured on the service: order, pages and LastEvaluatedKey, each accepted shape, each refusal and its text, legacy KeyConditions, the value type rule, and the MongoDB and Cassandra refusal.

target result
live DynamoDB 60 passed
SQLite (dev mode) 60 passed, 1 skipped
PostgreSQL 16 60 passed, 1 skipped
MongoDB 7 39 passed, 22 skipped (served-result cases)
Cassandra 4.1 the first push of this branch errored at the module fixture (61 errors, the hash-only Query answered 500 as on main); with the ORDER BY change the fixture's Query answers 200 locally. GSI propagation did not run in the local Cassandra setup used for that check (the queue rows stayed pending on main as well), so the full run on Cassandra is the CI job's
main at ec81e9d4, SQLite 30 of the 30 then-existing cases failed

Regression on the branch servers: test_multipart_gsi.py, test_query_scan.py, test_conditional_operator.py, test_expression_param_usage.py, test_item_operations.py and the other test_query*.py files, 309 cases on SQLite and 180 on PostgreSQL, all pass. A 67-request probe comparing the branch with the service case by case leaves 13 differences, all present on main and outside this change: the parser's wording for two non-equality clauses (#427), the 2048-byte hash key and 1024-byte aggregate range key limits on Query, and an extra page with a LastEvaluatedKey when Limit lands on the last item.

Negative controls on the committed tree, each restored with git checkout: the eight classifier mutations (missing-HASH order, RANGE duplicate detection, non-equality duplicate detection, RANGE gap window, non-equality excluded from the gap, early return widened, non-key refusal disabled, count refusal disabled) each fail their unit test; the type check disabled fails 6 wire cases; the begins_with rule disabled fails 1; the MongoDB refusal disabled fails 18 wire cases on wrong items or order.

Gates: cargo fmt --check; cargo +1.97.0 clippy --all-targets -D warnings on core, engine, storage and the four storage crates; cargo +1.88.0 build on the same; unit tests core 521, engine 103, storage 72, storage-sqlite 101, storage-postgres 46, storage-mongodb 96, storage-cassandra 52. No dependency change.

Not in this change

Pre-existing, filed separately: #427 (the parser refuses a lone non-equality, or two non-equalities, before the key schema is known, so the wording differs from the service); Query does not enforce the key value size limits the service applies (Size of hashkey has exceeded the maximum size limit of2048 bytes, Aggregated size of all range keys has exceeded the size limit of 1024 bytes); a Limit that lands on the last item returns a LastEvaluatedKey and one more empty page where the service returns none. Full support for several RANGE attributes on the MongoDB and Cassandra backends (typed fields and clustering for every RANGE attribute) is a separate change.

Overlap with other open PRs

…I the way the service does

On a global secondary index with one HASH attribute and several RANGE attributes, Query ordered results by the first RANGE attribute and then the table key, refused every key condition with three or more clauses as "Query condition missed key schema element: h", and accepted a condition that skipped the first RANGE attribute while returning no items. KeyCondition::resolve_multipart returned early when the schema had one HASH attribute, so the parser's assignment stood: the first equality became the partition key and the rest sat where the engine's HASH-count check refused them.

Measured against the service on 2026-10-21, with the issue's schema (HASH h, RANGE a, b, c, d) and with two HASH attributes:

- Results are ordered over every RANGE attribute in key-schema order, then the base table key; ScanIndexForward = false is the exact mirror. LastEvaluatedKey carries every key attribute.
- A key condition is served when the HASH attributes all have an equality and the RANGE conditions cover a leading run of the RANGE attributes with an equality on every one before the last; the last may use <, <=, >, >=, BETWEEN or begins_with. Clause order in the expression is free.
- Refusals, in the order the service applies them: a second condition on one key attribute ("KeyConditionExpressions must only contain one condition per key"); more conditions than key attributes ("The number of query conditions (6) exceeds the number of key attributes defined in the schema (5)"); a HASH attribute without an equality ("HASH key attributes h1, h2 must have equality conditions specified in the query", every missing one named); a RANGE gap ("RANGE key attributes a must have equality conditions specified in the query because a condition is present on key attribute b"); a value whose type differs from the attribute definition ("One or more parameter values were invalid: Condition parameter type does not match schema type"); a condition on an attribute outside the key schema, named by the first RANGE attribute without a condition ("Query condition missed key schema element: a"). begins_with on a number attribute is "Incorrect operand type for operator or function; operator or function: begins_with, operand type: N". The single-attribute wording "Query condition missed key schema element" stays on single-attribute schemas.

resolve_multipart now classifies every clause by name against the HASH and RANGE attribute lists and applies those rules in that order; the value type check runs inside it through resolve_multipart_with, with the engine supplying the attribute definitions. The engine's single-attribute path gets the same type check ahead of its missing-key check, which turns a 500 on a wrong-typed sort key value into the service's refusal. Legacy KeyConditions on a multi-attribute schema go through the same classifier. The SQLite and PostgreSQL backends apply every RANGE condition to its column and order and page over all RANGE columns; the cursor keeps its shape. The MongoDB and Cassandra backends filter the first RANGE attribute only and now refuse a condition on a later one instead of applying it to the wrong column. On Cassandra every Query on an index with several RANGE attributes answered 500 before this change: the index table clusters on every RANGE column and CQL rejects an ORDER BY that skips one, so the ORDER BY now names every index RANGE column in declared order, and a Query with an ExclusiveStartKey on such an index is refused instead of answering 500, because the cursor compares the first RANGE attribute only. docs/differences-from-dynamodb.md records both backends.

tests/test_gsi_multi_range_query.py pins 60 cases, all measured on the service: order, pages and LastEvaluatedKey, every accepted shape, every refusal and its text, legacy KeyConditions, and the MongoDB and Cassandra refusal. 60 pass on the service and on SQLite and PostgreSQL; on MongoDB 39 pass and the 22 served-result cases skip; on Cassandra the same 22 skip. On main 30 of the 30 then-existing cases failed. Unit tests in core cover the classifier rules; the storage plan and cursor tests cover the ORDER BY and cursor columns.

Closes #381
Conflicted file and resolution:

docs/differences-from-dynamodb.md: main appended two rows (GSI
throughput on a PROVISIONED table, projected attributes across all
indexes) and the Backup and Restore section after the multi-part keys
row; this branch appended one row on Query over a GSI with several RANGE
attributes at the same place. The merged table keeps main's two rows,
then this branch's row, then main's Backup and Restore section.

Auto-merged files checked by hand:

crates/storage/src/util/mod.rs: main's backup_id and catalog_version
re-exports and this branch's refuse_conditions_beyond_first_range
re-export both kept.

crates/storage-mongodb/src/data_engine.rs: main's new MongoEngine test
fixture fields and this branch's range-condition refusal in the query
path do not overlap.
@LeeroyHannigan

Copy link
Copy Markdown
Collaborator Author

Measured on the service on 2026-10-23 while working #427 (each row twice, identical answers), for the placement of the type check this PR adds:

  • Multi-attribute index (multi-idx, HASH h, RANGE a b c d), h > :n with :n = {"N": "1"}: HASH key attributes h must have equality conditions specified in the query. The HASH wording precedes the type check on a multi-attribute index.
  • Single-attribute schema, pk > :n with :n = {"N": "1"} on an S key: One or more parameter values were invalid: Condition parameter type does not match schema type. The type check precedes the no-HASH-equality wording there.
  • pk > :n with an unused :y: Value provided in ExpressionAttributeValues unused in expressions: keys: {:y}; pk > :n with FilterExpression="nope(": Invalid FilterExpression: Syntax error; token: "<EOF>", near: "(". Both precede the type check.
  • sk >= :v with :v = {"N": "1"} on an S sort key, with or without an ExclusiveStartKey: the type sentence. ExtendDB answers InternalServerError for the sort key bound today; worth confirming the check covers the sort key clause and not only the partition value.
  • begins_with(pk, :n) with :n = {"N": "1"} on an N key: Invalid KeyConditionExpression: Incorrect operand type for operator or function; operator or function: begins_with, operand type: N.

#479 conflicts with this branch in crates/engine/src/query.rs (the partition key block); whichever merges second gets rebased, and the resolution keeps both orders above.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Query on a GSI with one HASH and several RANGE attributes: wrong sort order, and three or more key conditions are rejected

2 participants