[LeeroyHannigan]

What

ExtendDB's backfill starts after a 1 s wait and a write barrier, so it has no stage that corresponds to the allocation phase. A CREATING index on PostgreSQL now answers as the service does in the backfilling phase:

1. UpdateTable returns once the catalog row is committed, with the index CREATING. The build task is detached instead of awaited. It backfills and flips the row to ACTIVE when the backfill commits.
2. Query and Scan on a CREATING index answer ValidationException "Cannot read from backfilling global secondary index: {name}". Before, they answered "The table does not have the specified index".
3. DeleteTable while a GSI is CREATING answers ResourceInUseException with the measured message, on a deletion-protected table too, where the service gives the same answer ahead of the protection refusal. Before, it deleted an unprotected table under the running build.
4. One server process runs at most four GSI builds at once (GSI_BUILD_SLOTS in crates/storage-postgres/src/data/gsi_build.rs), and the recovery worker's builds count toward the four. UpdateTable never waits for a slot. A build that finds none waits in its task with no database session open and its index CREATING, and slots go to waiting builds in the order they asked. Without the limit, a client that added indexes to one table after another started every build at once, each with a database session of its own for its ownership lock. In a storage-level test against a local PostgreSQL 16 container, 30 UpdateTable calls in a row on 30 tables of 300 items, with each build held after its first page, left 42 sessions on the database 3 s after the last call (11 before) and 30 ownership locks held. With the limit, the same 30 calls left 16 sessions and 4 ownership locks at that point, the most seen at any sample was 18 sessions, and all 30 indexes were ACTIVE 12.8 s after the builds were released.

Differences that remain, with the service's measured answers:

• TableStatus is ACTIVE in the UpdateTable response and for the whole build. The service reports UPDATING until Backfilling turns true, 16 to 39 s after the call, then ACTIVE.
• GSI descriptions have no Backfilling member. The service reports Backfilling false in the UpdateTable response and until that point, then true until the index is ACTIVE.
• A Delete of the creating index removes it at once: the response no longer lists it, and a Query naming it gets "The table does not have the specified index". The service refused that Delete in the allocation phase. In the backfilling phase it accepted it and showed the index DELETING for about 5 s, and a Query meanwhile got the backfilling text.
• For the first seconds after Backfilling turns true, the service still answers a Query with "The table does not have the specified index" (0 and 2 s after, on one table), and later with the backfilling text (about 45 s after). ExtendDB answers the backfilling text for as long as the index is CREATING.

The same four are rows in docs/differences-from-dynamodb.md.

These answers are unchanged and match the service's backfilling phase: a second index Create gets LimitExceededException, a Delete of the creating index is accepted, and PutItem and stream changes are accepted. A switch to PROVISIONED billing with table throughput only is accepted. The service was sent that switch with a GSI Update carrying the index's throughput as well, and accepted it; ExtendDB was compared without the GSI Update, which PostgreSQL refuses (see "Not changed"). ExtendDB also answers LimitExceededException to a Delete of a different index; on the service only a Delete of an index that did not exist was measured, and it got LimitExceededException.

Restart and failure handling uses the existing recovery worker and build ownership lock. When a build slot is free, UpdateTable takes the build's ownership lock (a session advisory lock on its own connection) before the catalog commit, and the build task holds it for its lifetime. A build that waits for a slot takes the lock when the slot comes. Until then its CREATING row has no owner, and the recovery worker on this or another server may take the index and build it instead. Once it holds the lock, the waiting build reads the index row again and goes ahead only if the index is still CREATING. If the server stops during a build, the lock is released with the connection, and the recovery worker, which runs a pass at startup and then every 60 s, takes ownership of each CREATING GSI whose lock is free and runs the build again over the partial index table. The backfill upserts, and the orphan sweep removes leftovers. A transient error leaves the index CREATING for the recovery worker. A deterministic error removes the index, as the recovery worker already did, so DescribeTable shows it gone and a later UpdateTable can create it again. The caller no longer receives that error in the UpdateTable response. The service answers UpdateTable before its backfill starts, so it cannot return a backfill failure in that response either.

I chose this over handing every build to the recovery worker, which would start a build up to 60 s late, and over a new persisted failed state, which the service does not report and which would need a catalog migration.

docs/design/04-component-storage.md section 6.2 and the module doc of data/gsi_build.rs now describe the build task.

Not changed

UpdateTable with a GSI Update action is refused on PostgreSQL with ValidationException "UpdateGlobalSecondaryIndex is not yet supported", whether the index is CREATING or ACTIVE, as before this PR. The service accepted that action in the backfilling phase.

UpdateTable holds a pool connection for its catalog transaction while it takes a second one to create a new GSI's data table, so on a server whose catalog and data share one pool, as many concurrent GSI Creates as the pool has connections can each hold one and time out waiting for another; this PR does not change that.

This PR does not change SQLite. SQLite builds the index inside UpdateTable and answers with it ACTIVE. On a 20,000-item table the call took 0.62 s and 0.67 s in two runs of a local release build. Its response reports the index ACTIVE where the service reports CREATING.

Overlap with open pull requests

#690 also adds test binaries to the PostgreSQL storage-level step of .github/workflows/integration.yml and test hooks to crates/storage-postgres/src/lib.rs. The two branches conflict in those two places only. Whichever merges second keeps both sides.

Why

On PostgreSQL, UpdateTable with a GSI Create spawned the index build and awaited it inside the request (crates/storage-postgres/src/update_table.rs), so the response came back only after every existing item had been indexed. The issue reports 23 s for a 3,000-item table under load, and a client ReadTimeoutError at boto3's default 60 s read timeout on a 20,000-item table while the build went on in the server.

DynamoDB answers UpdateTable at once with the new index CREATING and backfills in the background.

On a 20,000-item table on a local PostgreSQL 16 container, the backfill on this branch took 5.6 s from the UpdateTable call to the index reporting ACTIVE. Before this change the UpdateTable response waited for the backfill to finish; the pre-change build was not timed separately on this host. With this change the call returned in 24 ms with the index CREATING. Both figures were measured before the build limit (item 4 under "What") was added. A build that finds a free slot starts the way it did before.

Service behaviour

Measured on Amazon DynamoDB in us-east-1 on 30 October 2026, on four on-demand tables: two of 2,000 items and two of 500 items, one of the 500-item tables created with deletion protection on. Each measurement added a GSI with UpdateTable and polled DescribeTable every 1 or 2 s, six index builds in all:

• UpdateTable returned 200 in under 100 ms with TableStatus UPDATING, the new index CREATING and Backfilling false.
• 16 to 39 s after the call (16.3, 24.3, 24.4, 26.2, 26.2 and 38.5 s), TableStatus went back to ACTIVE while the index stayed CREATING with Backfilling true. The service's own messages call the first stage the resource allocation phase and the second the backfilling phase.
• On the 2,000-item tables the index became ACTIVE 16 min 39 s and 16 min 33 s after the call. ItemCount and IndexSizeBytes stayed 0 until then.

Answers while the index was CREATING:

Request	Allocation phase	Backfilling phase
Query or Scan on the index	ValidationException "The table does not have the specified index: gsi1"	ValidationException "Cannot read from backfilling global secondary index: gsi1"
PutItem, UpdateItem	200	200 (PutItem)
UpdateTable adding a second GSI	LimitExceededException "Subscriber limit exceeded: Only 1 online index can be created or deleted simultaneously per table"	the same
UpdateTable deleting the creating GSI	ResourceInUseException "Attempt to change a resource which is still in use: Index creation is in resource allocation phase. Retry deletion during backfilling phase or when the index is active. Table: {table} Index: gsi1"	200; the index went DELETING and was gone about 5 s later
UpdateTable enabling a stream	ResourceInUseException "Attempt to change a resource which is still in use: Can't change stream status when an index is being created. Table: {table} Indexes: [gsi1]"	200
UpdateTable to PROVISIONED billing, with table throughput and a GSI Update carrying the index's throughput	ResourceInUseException "Attempt to change a resource which is still in use: Index is being created but is not backfilling yet. Table: {table} Index: gsi1"	200
DeleteTable	ResourceInUseException "Attempt to change a resource which is still in use: Cannot delete table while indexes are being created, updated, or deleted."	the same
DeleteTable on the deletion-protected table	the same ResourceInUseException. With no index change running, the same table answered ValidationException "Resource cannot be deleted as it is currently protected against deletion. Disable deletion protection first."	the same ResourceInUseException

The backfilling-phase Query and Scan answers on the two 2,000-item tables came about two minutes after UpdateTable. On the deletion-protected table, during the build of its second index, Queries 0 and 2 s after Backfilling turned true still got "The table does not have the specified index", and Queries about 45 s after it got the backfilling text.

Each allocation-phase answer was measured on two tables, except DeleteTable: once on an unprotected table, and twice on the deletion-protected one. In the backfilling phase, PutItem, the second Create, the stream change, the billing change and the Delete of the creating index were measured once each, Scan on two tables, Query on three, and DeleteTable five times on two tables.

Closes #654

Testing done

Tests

New crates/storage-postgres/tests/gsi_async_build.rs, using a hook behind the crate's test-hooks feature (hold_gsi_backfill_for_tests) that holds one table's GSI build at its start or after a given number of backfill pages:

• update_table_returns_while_the_backfill_runs: with the build held, UpdateTable returns within 10 s with the index CREATING, and DescribeTable agrees. A read of the index gets the backfilling message. DeleteTable gets the ResourceInUse message and the table stays. After one page is released, the test writes behind and ahead of the scan (a key change, a delete, an insert, an item losing the key attribute). On release the index reaches ACTIVE and holds each base item that carries the key attribute once.
• a_restart_during_the_backfill_still_reaches_active: a first engine on its own runtime adds the GSI with the build held after one page. Once at least 256 rows are in the index, that runtime is shut down, and the index stays CREATING. After more writes, reconcile_creating_gsis on a second engine finishes the build, and the index matches the base table with each item once.

New crates/storage-postgres/tests/gsi_build_task.rs, for what the build task does after UpdateTable has answered:

• a_build_that_fails_every_time_removes_the_index: one stored item is made undecodable. UpdateTable answers with the index CREATING, the build fails on that item, and the index and its data table are removed. Once the item is restored, the same Create builds the index.
• a_build_that_fails_for_a_passing_reason_stays_creating: an outside transaction holds the base table's lock, so every write barrier attempt times out. The index stays CREATING with its data table, the build's ownership lock is released, and the next recovery sweep publishes the index.
• the_recovery_sweep_leaves_a_running_build_alone: a recovery sweep during a held build returns at once with nothing built.
• delete_table_of_a_protected_table_during_a_build_is_in_use: on a deletion-protected table, DeleteTable gets the protection refusal before the build, the ResourceInUse message during it, and the protection refusal again once the index is ACTIVE.

New crates/storage-postgres/tests/gsi_build_slots.rs. Its first three tests run on an engine given one build slot, with a first table's build held after its first page:

• a_build_waits_for_a_free_slot_holding_no_session: UpdateTable on a second table returns within 10 s with the index CREATING. That index refuses a read with the backfilling message, has 0 rows after 3 s, and holds no ownership lock. Once the first index is ACTIVE, the second build starts under its own ownership lock, and both indexes end holding exactly the base items.
• the_recovery_sweep_waits_for_a_free_slot: a recovery sweep takes the waiting index and builds nothing while the slot is taken. When the slot frees, the index's own task finds the sweep owning it and leaves it, and the sweep publishes it.
• a_waiting_build_leaves_an_index_a_peer_published: a second engine on the same database stands in for another server. While the second build waits, that engine's recovery sweep takes the index and publishes it. The second index's build entry is then held, so a build that started there would keep its ownership lock. When the slot frees, the waiting task takes the lock, finds the index ACTIVE and releases the lock, and 3 s later no ownership lock is held.

The fourth, an_engine_runs_at_most_gsi_build_slots_builds, runs on an engine with its own four slots: UpdateTable on five tables, each build held after its first page, leaves four builds running with one ownership lock each, and the fifth index with 0 rows and no owner after 3 s. All five indexes then reach ACTIVE holding exactly the base items.

The new files are listed in the PostgreSQL storage-level test step of integration.yml.

Tests that assumed the build had finished when UpdateTable returned now wait for the index to be ACTIVE: add_gsi in gsi_online_build.rs and gsi_stale_writer.rs, and in the wire suite update_table_attribute_definitions and gsi_on_hash_only_table, through a new test_base::wait_for_index_active. Without that, two wire tests failed against a server built from this branch: one got the backfilling ValidationException and the other got LimitExceededException on its next index add.

Negative controls

Each was run on the committed tree and restored with git:

• UpdateTable awaiting the build again: update_table_returns_while_the_backfill_runs fails with "UpdateTable waited for the held backfill instead of returning".
• data/ddl.rs as before this change: the same test fails because the read answered IndexNotFound.
• delete_table.rs as before this change: the same test fails because DeleteTable answered Ok.
• The deletion-protection check ahead of the CREATING check again: delete_table_of_a_protected_table_during_a_build_is_in_use fails because DeleteTable answered the protection refusal during the build.
• Every build failure treated as transient: a_build_that_fails_every_time_removes_the_index fails because the index stayed CREATING.
• Every build failure treated as permanent: a_build_that_fails_for_a_passing_reason_stays_creating fails because the index was removed.
• The ownership lock dropped before the build task starts: the_recovery_sweep_leaves_a_running_build_alone fails because the sweep started a second build and did not return.
• The waiting build task taking a slot of its own instead of one of the engine's: both slot tests fail because the second build wrote 256 rows while the first held the only slot.
• The recovery worker taking no slot: the_recovery_sweep_waits_for_a_free_slot fails the same way.
• The waiting build task building without taking ownership: both slot tests fail, one because the second build ran without its ownership lock and the other because the sweep published nothing.
• UpdateTable waiting for a slot: both slot tests fail because UpdateTable did not return within 10 s.
• The waiting build task building without reading its index row again: a_waiting_build_leaves_an_index_a_peer_published fails because the task held the published index's ownership lock to build it a second time.
• The engine's build slots unbounded (Semaphore::MAX_PERMITS permits): an_engine_runs_at_most_gsi_build_slots_builds fails because the fifth build wrote its first page.

Results

On this branch, against PostgreSQL 16 with pgvector in a local container:

• The 13 storage-level test binaries in the CI list, release build: 201 passed, 0 failed, 2 ignored.
• cargo fmt --all -- --check and cargo clippy -p extenddb-storage-postgres --all-targets -- -D warnings on 1.97.0, cargo check --locked --workspace --all-targets, and cargo +1.88.0 check --locked -p extenddb-storage-postgres --all-targets pass with no warnings.

On an earlier revision of this branch, based on 1c0523c90 and without the waiting build's re-read of its index row or the last two gsi_build_slots tests:

• The wire suite (tests/rust) against a release server: 512 passed, 0 failed.
• The Python GSI tests (test_gsi_async, test_gsi_attribute_definitions, test_gsi_online_backfill, test_multipart_gsi, test_table_operations) against the same server: 48 passed, 0 failed.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk
  format, or public CLI surface, an RFC has been accepted or is linked
  below. Otherwise, an ADR captures the decision (link below).

CI ran the workspace tests on Rust 1.88.0 and on stable, and the integration suites on all four backends, including the wire suite and the Python suite (without its import and export tests) against a PostgreSQL server. All of them passed. Locally I ran the steps listed under Results. Clippy ran with -D warnings, without the pedantic group. The change adds no command, flag, catalog migration or trait method. The build task's design is in docs/design/04-component-storage.md section 6.2.

ADR / RFC: n/a

Breaking changes

• UpdateTable with a GSI Create returns with the index CREATING. A client that reads the index right after UpdateTable returns has to wait for it to be ACTIVE, as on the service.
• Query and Scan on a CREATING index answer ValidationException "Cannot read from backfilling global secondary index: {name}", and DeleteTable while a GSI is CREATING answers ResourceInUseException.
• A build failure no longer comes back in the UpdateTable response. A deterministic failure removes the index, and a transient one leaves it CREATING for the recovery worker.
• One server process runs at most four GSI builds at once. A further build waits for a slot with its index CREATING.

---

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.