[LeeroyHannigan]

Closes #366. Merge after #428: the LSI paragraph in docs/design/04-component-storage.md is written against the section 7 text #428 introduces (the **... Query:** headings, explicit ASC and DESC, the hash-only and no-base-sort-key paragraphs). Merged on top of #428 it replaces #428's LSI paragraph; merged before it, section 7 would describe two different LSI cursors until #428 is rebased. The conflict is one hunk, plus two #428 sentences that merge cleanly but describe the LSI cursor this change removes:

1. Conflict hunk in section 7: take this branch's LSI Query: paragraph and SQL block in place of docs(design): describe the query cursor as the row comparison the code emits #428's LSI Query: paragraph; keep docs(design): describe the query cursor as the row comparison the code emits #428's Hash-only index: paragraph and the "On a base table without a sort key" paragraph after it unchanged.
2. Section 7 intro: replace "For the Scan, base table Query, GSI and hash-only index shapes below, the ordering index serves the page in order with no sort step; the LSI shape is described after them." with "For every shape below, the ordering index can serve the page in order with no sort step."
3. Section 5.1: replace "so a GSI cursor is a row comparison the ordering index serves in one seek (the LSI case is in section 7)." with "so a GSI or LSI cursor is a row comparison the ordering index serves in one seek."

What the service does

Measured 2026-10-21 on DynamoDB with a (pk, sk) table and an LSI on (pk, lsk), 9 items in three groups of equal lsk: a Query on the LSI returns the items ordered by lsk, the items inside a group in one deterministic order (the same across two tables), and a reverse Query is the exact mirror of the forward one. Page boundaries and LastEvaluatedKey ({lsk, pk, sk}) follow that order. The order inside a group is not byte order of the base sort key; that difference is #425 and is unchanged here.

What ExtendDB did

Every index table whose index has a sort key has an ordering index on (pk, sk, base_pk, base_sk) on PostgreSQL and SQLite. A GSI Query orders by (sk, base_pk, base_sk), the index without the constant pk, so the database walks the index and stops at the Limit. An LSI Query ordered by (sk, base_sk) with a cursor over the same columns. The planners cannot tell that base_pk is constant inside an LSI partition, so the index served sk only. PostgreSQL added an Incremental Sort over the index scan (a Sort when lsk = :l left one value); SQLite added a temporary B-tree for the last ORDER BY term and seeked a resumed page on sk alone. On a 50,000-item partition with 50 items per lsk value, a Limit 100 page read 151 rows and 154 buffers on PostgreSQL to return 101; a page resumed inside a group read 130 rows and 133 buffers.

Changes

• crates/storage-postgres/src/data/query_scan.rs: order_by_columns puts base_pk between the index sort key and the base sort key for every index with a sort key (the is_lsi parameter is gone from it and from build_pagination_where); an LSI cursor binds the queried partition key as base_pk; cursor unit tests updated.
• crates/storage-postgres/src/data/query.rs: PaginationBinds::BaseSkOnly removed (no remaining user).
• crates/storage-sqlite/src/data/query_scan.rs: the same column change in query_order_columns; append_query_pagination takes the queried partition key and binds it for an LSI; the statement builder moves from query_impl into query_statement with no behaviour change, so tests can run EXPLAIN QUERY PLAN on the statement a Query sends; cursor unit tests updated plus a bind test; new plan_tests module.
• crates/storage-postgres/tests/lsi_query_plan.rs: new storage-level plan test. It reads the statement the engine sent from pg_stat_activity and asserts that the generic plan and the custom plan are an index walk: no Sort or Incremental Sort node, no Filter under the index scan, for a first page and a resumed page with the partition alone, with lsk = :l and with lsk BETWEEN, in both directions. It checks the page rows and that a start key naming another partition returns the same page. It skips below PostgreSQL 16 (EXPLAIN (GENERIC_PLAN)).
• .github/workflows/integration.yml: --test lsi_query_plan added to the PostgreSQL storage-level step (pgvector/pgvector:pg16).
• docs/design/04-component-storage.md: the LSI Query paragraph in section 7, with the statement, the bind, and the sort step the ordering index avoids.

The cursor token format and the LastEvaluatedKey shape do not change.

Verification

Servers built from this branch, PostgreSQL 16.15 and SQLite in memory. Plans from the servers' own statements: PostgreSQL EXPLAIN (ANALYZE, BUFFERS) EXECUTE and EXPLAIN (GENERIC_PLAN); SQLite EXPLAIN QUERY PLAN.

Check	PostgreSQL	SQLite
LSI pages, 21 request shapes (first, resumed, FilterExpression, =, >, <, <=, >=, begins_with, BETWEEN, engine chunks), both directions	Index Scan or Index Scan Backward on the ordering index, cursor in the Index Cond, no sort node, custom and generic plan; rows read equal the LIMIT (was Incremental Sort or Sort, up to 130 rows for LIMIT 98)	SEARCH ... USING INDEX idx_order_*, no temporary B-tree, resumed seek on (sk_s, base_pk, base_sk_s) (was sk_s only plus a temporary B-tree); 1,196 VM opcodes per LIMIT 98 page, was 2,892
GSI, base table Query and Scan plans	unchanged	unchanged
Pages of the 52,000-item table before and after (every item key and LastEvaluatedKey)	identical	identical
9-item probe against the service output: items per page, group order, Count, LastEvaluatedKey keys	match on every query; order inside a group differs (#425)	same
tests/test_query_scan.py + tests/test_multipart_gsi.py	126 passed	126 passed
LSI cases in test_item_operations.py, test_table_operations.py, test_put_item_create_race.py	3 passed	3 passed
key-type matrix (pk, lsk, sk in N, B, S combinations; full, paged, BETWEEN, COUNT queries both directions)	0 problems	0 problems, output identical to PostgreSQL
crates/storage-postgres/tests/lsi_query_plan.rs (20 plans)	1 passed, 7 s	n/a
plan_tests (12 LSI statements, 4 GSI statements, paging both directions)	n/a	3 passed
cursor unit tests	8 passed	8 passed

Negative controls

Each mutation applied to the committed tree, the tests run, the file restored with git checkout, crates/ clean afterwards.

Mutation	Fails	Passes
PostgreSQL: base_pk filtered out of the LSI ORDER BY (all shapes)	lsi_query_plans_as_an_index_walk_in_both_directions: first page, forward=true, generic plan: the plan sorts the rows the index scan reads (Incremental Sort)	8 cursor unit tests
PostgreSQL: base_pk filtered out of the LSI ORDER BY only when the key condition has a sort key condition	same test: lsk = :l, first page, forward=true, generic plan: the plan sorts the rows the index scan reads (Sort, Sort Key: base_sk_s)	8 cursor unit tests
PostgreSQL: base_pk replaced by its bound value in the LSI cursor comparison	same test: resumed page, forward=true, generic plan: the plan filters rows after the index scan instead of seeking on them (Index Cond sk_s >= $2, Filter on the row comparison)	8 cursor unit tests
PostgreSQL: the start key's partition key bound instead of the queried one	same test: resumed page, forward=true: a start key naming partition q moved the page boundary
SQLite: base_pk filtered out of the LSI ORDER BY (all shapes)	plan_tests::lsi_query_walks_the_ordering_index_without_a_sort: LSI pk only forward=true resumed=false: the plan sorts in a temporary B-tree	10 (GSI plan test, paging test, cursor unit tests)
SQLite: base_pk filtered out of the LSI ORDER BY only with a sort key condition	same test: LSI lsk = :l forward=true resumed=false: the plan sorts in a temporary B-tree	10
SQLite: base_pk removed from the LSI cursor columns and bind	lsi_cursor_compares_the_ordering_index_columns, order_by_columns_match_the_cursor_columns_for_every_shape, lsi_cursor_binds_the_queried_partition_key_as_base_pk	8, the paging test among them: the rows do not change
SQLite: the start key's partition key bound instead of the queried one	lsi_cursor_binds_the_queried_partition_key_as_base_pk: left ["l", "other", "s"], right ["l", "p", "s"]	10

Gates

cargo fmt --all -- --check clean. cargo +1.97.0 clippy -p extenddb-storage-postgres -p extenddb-storage-sqlite --all-targets -- -D warnings clean. cargo +1.88.0 test --no-run on both crates builds. cargo test -p extenddb-storage-sqlite: 104 passed, 1 ignored. cargo test -p extenddb-storage-postgres against PostgreSQL 16.15: lib 45, doc_version_literals 3, key_collation 3, lsi_query_plan 1, put_create_race 9, stream_shard_retention 1, twi_conflict 8, vector_control_plane 42, all passed. No dependency change.

Not in this change

• An LSI ExclusiveStartKey whose partition key differs from the queried partition. The service returns HTTP 400 ValidationException, The provided starting key is outside query boundaries based on provided conditions. ExtendDB accepts it and resumes inside the queried partition, on main and on this branch; the storage layer does not read the start key's partition key, so the refusal belongs in request validation. Filed separately.
• The order of items that share an index sort key value: the service's order is not byte order of the base sort key (SQL backends: LSI Query orders items that share an index sort key differently from the service #425).
• A Query with Limit equal to the number of remaining items: the service returns the last page without a LastEvaluatedKey; ExtendDB returns one and then an empty page. Pre-existing, filed separately.
• Cassandra's LSI cursor is still (index sk, base sk); its clustering order is a different mechanism.

[LeeroyHannigan]

Rebased on main at abf8082; conflicts in .github/workflows/integration.yml, resolved by keeping both test entries: the PostgreSQL storage-level step runs gsi_online_build, gsi_stale_writer and lsi_query_plan, and its comment describes both.

The earlier merge of main is folded into the one commit. crates/storage-postgres (lsi_query_plan included, on PostgreSQL 16) and crates/storage-sqlite tests pass.