[LeeroyHannigan]

What

The relation names belong to the storage backend, so the comparison moved there.

• DiagnosticsStore (crate extenddb-storage) gains check_storage_relations and drop_orphaned_relations, with defaults that report the backend does not support them. Other backends are untouched.
• The PostgreSQL store implements both in a new data/catalog_check.rs. Expected names come from table_storage_relations, the list DeleteTable drops, for the rows of tables, indexes and vector_indexes in any status, through the crate's own naming functions. The data relations are listed from pg_tables in the data database's current schema by the _ddb_ prefix the naming functions now share.
• The check lists the relations, reads the catalog, lists the relations again and reads the catalog a second time. A relation is reported orphaned when both listings have it and no row names it. A relation is reported missing when the second listing lacks it and both catalog reads name it from a row that, with its table, is in neither CREATING nor DELETING. When the first listing has a relation the first read does not name, the check waits for the transactions writing indexes to end and reads the catalog again before it compares (see the lock section below). A check that finds each listed relation named takes no lock.
• --fix waits for the indexes writers the same way, reads the catalog under that lock, and releases the lock before it drops anything. Each reported relation that the read does not name is then dropped in a transaction of its own with a lock timeout of 1 s, the bound SYNC_DROP_LOCK_TIMEOUT sets for the server's own synchronous drops, using DROP TABLE without IF EXISTS. The command prints Dropped; Already gone when another session dropped the relation first (SQLSTATE 42P01); Kept with the reason; or FAILED with the error. For a relation another session holds longer than 1 s, the error reads "another session held a lock on it for more than 1000 ms, so it was kept; run the command again". If the index writers do not finish within 30 s, the command reports that, drops nothing and says to run it again.
• A catalog without data_database_connection_string, or with an empty value, is refused with the message main gives: "No data database connection string in settings. Run extenddb init." Both store methods refuse it, so the command does not look for data relations in the catalog's own database. The data pool connects with statement logging off, so a drop that waits out its lock timeout adds no slow-statement line to the report.
• cmd_catalog_check.rs calls the store for checks 1 and 2 and reports each missing relation with the table or index that owns it. Checks 3 and 4, the PID refusal, the counting and the exit code are unchanged.
• docs/troubleshooting.md: the entry from feat(storage): SnapshotEngine, a consistent table snapshot on every backend [backup/restore 4/4] #395 that told operators not to use --fix now describes the check, what it waits for, the reports that can be wrong while a server runs (below) and the drop outcomes. It says that --fix treats each _ddb_ relation in the data database's current schema as this deployment's, and its manual query now filters on current_schema().

The lock on indexes

Each relation name comes from a UUID that the request inserting its catalog row generates for that row: in create_table.rs for a table and the GSIs, LSIs and vector indexes created with it, which restore also uses, and in update_table.rs for a GSI or vector index added later. No path inserts a row for an id it did not generate, so a row inserted later cannot name a relation that exists already. The recovery paths in gsi_build.rs and vector_index.rs create a relation only for a row that has committed.

CreateTable and a vector index add commit the catalog rows first and create the relations after. A GSI add in UpdateTable is the only path that commits its relation before its row: it inserts the index row, creates and commits the relation on the data database, and then commits the catalog transaction, which holds ROW EXCLUSIVE on indexes from the insert to its end. A SHARE lock on indexes is granted only after each such add whose relation exists has committed or rolled back, and a catalog read taken after the grant sees the rows of those that committed. No row names a relation that this read does not name, and no row inserted later can name it.

The command requests the lock in attempts of 100 ms, 50 ms apart, for up to 30 s. While an attempt waits, new writes to indexes queue behind it, so a server write waits on the command for at most 100 ms plus the one catalog read the lock is held for. The command holds no other lock while it waits, so it cannot take part in a deadlock, and the GSI add's next statement, an update of its tables row, does not wait on it. The transaction runs at READ COMMITTED whatever default_transaction_isolation the database, role or connection sets, and sets its lock timeout with SET LOCAL, which takes no snapshot, so the read after the grant sees rows committed during the wait.

Servers do not hold migrate's advisory lock, so taking it would not exclude a running CreateTable or UpdateTable.

#708 (fix/654-pg-async-gsi-build), the fix for #654, stops UpdateTable from waiting for the GSI build and leaves the add's order of insert, create and commit as it is, so the lock covers a GSI add there too. Its cleanup of a failed build removes the CREATING row, waits GRACE_PUBLISH (1 s) and then drops the relation, in the order main uses, so a check in that second reports the relation orphaned. The two branches add adjacent lines to the storage-level test list in integration.yml and to the test-hooks re-exports in crates/storage-postgres/src/lib.rs; merging both keeps the lines of each.

Reports that can be wrong while a server runs

The command refuses to run while it finds the server's PID file, but it does not see a server started without --write-pid-file, one on another host, or one using another run_dir or port. With a server running, the reports below can be wrong, and a later run does not repeat them:

• Orphaned: the relations of a table, GSI or vector index whose catalog row a server has removed and whose drop has not committed when the second listing runs. This is the case for a synchronous DeleteTable between its catalog commit and its data commit, for an UpdateTable that deletes a GSI or vector index before its drop runs, and for a failed GSI build during its GRACE_PUBLISH second. A --fix that drops such a relation first causes the server no error, because the server's drops use IF EXISTS.
• Missing: the relations of a table whose row is ACTIVE before its relations commit. A CreateTable inserts the row ACTIVE when control_plane_delay_seconds is 0. Otherwise the row moves from CREATING to ACTIVE once that delay (0.25 s by default) has passed since its catalog transaction began, whether or not the relations have committed. --fix drops only relations reported orphaned, so a wrong missing report leads to no drop.

The command does not report a GSI or vector index stuck in CREATING: check 4 covers tables only, and check 2 does not count a CREATING index row as expected.

Why

extenddb catalog-check expected data relations named _ddb_{account}_{table} and _ddb_{account}_{table}__gsi__{index}. The storage layer names them from catalog ids: _ddb_{table_id}, _ddb_{index_id} and _ddb_vec_{index_id} for vector indexes, which the command did not know. It also counted only tables in ACTIVE or CREATING as live.

Measured on main 1c0523c90 (2026-10-30, PostgreSQL 16 with pgvector, a separate data database as extenddb init creates it, one table with a GSI, an LSI and a vector index): the report listed all four live relations as orphaned and three names that never existed as missing; --fix dropped the four relations; GetItem then returned ResourceNotFoundException and Query on either index returned ValidationException: The table does not have the specified index, while DescribeTable still reported the table ACTIVE.

On SQLite the command exits before any check with Cannot connect to catalog: error with configuration: relative URL without a base, because it opens a PostgreSQL pool unconditionally (#177). It reports and drops nothing there, so no SQLite change is needed.

Closes #652

Testing done

The runs in this section were made against PostgreSQL 16 with pgvector and a separate data database, on 2026-10-30 unless an item gives another date.

With a binary built from 4556ffcd4, on the deployment from the reproduction with the table recreated: the report is HEALTHY (exit 0). With a planted _ddb_00000000-0000-4000-8000-00000000dead and an unrelated operator_scratch, the report lists the planted relation only; --fix prints Dropped for it and nothing else; the four live relations and operator_scratch remain; GetItem returns the item and Query on gsi1 and lsi1 returns 3 and 1 items.

The loops below used a binary built from 90d429449, whose code matches the branch head (the later commits change tests, one comment and docs), with a server running on the same deployment. The command ran with a config for another port, so the PID refusal did not apply. Four traffic workers each created a table with a GSI, an LSI and a vector index, set its TTL, tagged it, wrote three items, added a second GSI, queried both indexes and deleted it, in a loop; a fifth wrote items to one table.

• Report loop: 300 runs 0.1 s apart from 20:23:52Z to 20:26:45Z, with no planted relation. All 300 exited 0 (median 466 ms, max 930 ms). The traffic made 365 CreateTable, 364 UpdateTable, 364 DeleteTable and 8,045 PutItem calls, all answered 200. 40 of the CreateTable calls took longer than 250 ms (max 480 ms), so the missing-report case above could occur during the loop; no run reported it.
• The same report loop on 2026-10-31, against a build of fe791ce87 on a busier host where 188 of 349 CreateTable calls took longer than 250 ms (p50 255 ms, max 808 ms): 3 of 300 runs exited 1, each in a case listed under "Reports that can be wrong while a server runs". Two runs reported missing relations for a table whose CreateTable took 558 ms in one run and 435 ms in the other. The third reported orphaned relations for two tables whose DeleteTable calls were between their catalog commit and their data commit. How many runs report a wrong result depends on how long CreateTable and DeleteTable take on the host. The server log has no deadlock and no error line.
• The same loop with an earlier binary (16c62f521), which requested the lock on each run in attempts of 200 ms, had 1 run of 300 exit 1, at 20:10:56Z: the four relations of a table whose CreateTable took 308 ms were reported missing. A waiting lock request can hold up a CreateTable's insert into indexes for one attempt, which counts against the 0.25 s delay. The check now requests the lock only when it finds an unnamed relation.
• --fix loop: 120 runs 0.3 s apart from 20:29:20Z to 20:31:20Z, each with a planted relation that no row names. Each run reported and dropped that relation and nothing else; no run kept or failed a drop or reported a relation missing. The traffic made 253 CreateTable, 252 UpdateTable and 7,457 PutItem calls, all answered 200, and the server log has no deadlock and no error line.
• A planted relation held by a psql session in ACCESS SHARE mode for 6 s: --fix printed FAILED to drop ...: another session held a lock on it for more than 1000 ms, so it was kept; run the command again: error returned from database: canceling statement due to lock timeout and exited 1 after 1.33 s. After the session ended, --fix printed Dropped.
• With the data_database_connection_string setting absent, the report and --fix both printed Error: No data database connection string in settings. Run `extenddb init`. and exited 1.

Tests

crates/storage-postgres/tests/catalog_check/ (main.rs, report.rs, fix.rs, support.rs), run as --test catalog_check in the PostgreSQL storage-level step of integration.yml. Each test initialises a deployment through PostgresBootstrapper with a separate data database, creates a table with gsi1, lsi1 and vec1 (vec1 when pgvector is installed) and three items through the engine, and drives PostgresCatalogStore as the DiagnosticsStore. Without EXTENDDB_TEST_PG_CONNECTION_STRING the tests skip.

The check:

• live relations are neither orphaned nor missing;
• a renamed GSI relation is reported missing with index gsi1 of table 123456789012/orders as its owner, and the renamed one orphaned;
• rows in any status keep their relations out of the orphan list;
• with the relations removed, a table, index or vector index in CREATING or DELETING is not reported missing, and all of them are once the rows are ACTIVE again;
• with a write to indexes left open, a check that finds each relation named finishes in under 2 s;
• a catalog without the data database setting, or with an empty one, is refused by both methods with main's message, and the planted relation stays;
• with the check held after its first catalog read, a DeleteTable, a relation another session drops and a relation created then are not reported, and the next check reports the new relation.

The drop:

• --fix drops a planted relation, keeps an unrelated table, keeps the relations of a table created after the check when they are passed as stale candidates, and leaves the items and index rows intact;
• a GSI add that has inserted its row and committed its relation is in flight while the drop runs. The drop waits for it holding no lock on tables or vector_indexes; the add's tables write finishes in under 900 ms and another indexes write in under 1 s; the add commits while a lock attempt waits; the relation is kept. The test runs at the default isolation and again with the store's sessions defaulting to REPEATABLE READ;
• the same add in flight during the check: its relation is not reported orphaned;
• a planted relation held by another session: the drop reports it FAILED with the lock message within 5 s, a CreateTable sent meanwhile finishes within 5 s, and once the session ends the drop succeeds;
• a relation another session drops after the check reported it comes back as Already gone.

A unit test in data/catalog_check.rs checks that a row in CREATING or DELETING, or one that is absent, does not count as settled.

Negative controls

Each mutation was applied to a clean committed tree, the catalog_check tests were run, and the file was restored with git checkout and the tree checked clean before the next.

Mutation	Tests that failed
No SHARE lock on indexes before the read	the two drop-during-add tests and the check-during-add test
The lock on tables, indexes and vector_indexes, as before	the two drop-during-add tests (the waiting session held a lock on tables)
No retry after an attempt times out	the two drop-during-add tests and the check-during-add test
The lock requested on each check	the test with an open indexes write
The transaction starts with SELECT set_config(...) and no READ COMMITTED	the drop-during-add test under REPEATABLE READ (relation dropped), in 3 runs of 3
The drop ignores the names the read under the lock gives	the stale-candidate test and the two drop-during-add tests
Drops without their own lock timeout	the held-relation test
DROP TABLE IF EXISTS again	the already-gone test
Orphans taken from the second listing	the test that holds the check after its first read
Orphans without the second-listing filter	the test that holds the check after its first read
Missing without the second catalog read	the test that holds the check after its first read
A DELETING row counts as expected	the CREATING and DELETING test
A CREATING row counts as expected	the CREATING and DELETING test
The prefix check in the drop replaced by false	the stale-candidate test
The prefix check in the drop disabled with if false &&	the stale-candidate test
The listing without the prefix filter	9 of the 13 tests
A catalog without the setting falls back to the catalog pool	the refusal test

An earlier control, run when the test file had its first four tests, built the expected names from account and table names instead of ids; all four tests failed.

Gates

• cargo +1.97.0 fmt --all --check: clean.
• cargo +1.97.0 clippy -p extenddb-storage -p extenddb-storage-postgres -p extenddb-app -p extenddb --all-targets -- -D warnings: no warnings.
• cargo test --locked -p extenddb-storage -p extenddb-app: 138 and 34 passed. cargo test --locked -p extenddb-storage-postgres --lib: 65 passed.
• The PostgreSQL storage-level step of integration.yml, cargo test --release -p extenddb-storage-postgres with its 11 test binaries and EXTENDDB_TEST_PG_REQUIRED=1, against PostgreSQL 16 with pgvector: 204 passed, 2 ignored, none failed (catalog_check: 13 passed).
• cargo +1.88.0 check --locked -p extenddb-storage -p extenddb-storage-postgres -p extenddb-app --all-targets: clean.
• cargo check --locked --workspace --all-targets: clean.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace)
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy -- -W clippy::pedantic)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk
  format, or public CLI surface, an RFC has been accepted or is linked
  below. Otherwise, an ADR captures the decision (link below).

CI ran the workspace tests on Rust 1.88.0 and on stable, and the integration suites on all four backends. All of them passed. Locally I ran the crates listed under Gates. Clippy ran with -D warnings, without the pedantic group. The change adds two methods with default bodies to DiagnosticsStore, so other backends compile unchanged, and adds no command, flag or on-disk format. There are no breaking changes.

ADR / RFC: n/a

---

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.