[LeeroyHannigan]

Summary

#666 put SQLite and PostgreSQL on one numbered list of catalog migrations with a ledger in every store. This PR moves Cassandra and MongoDB onto the same list and the same runner. All four backends now read their catalog status from their ledgers, refuse a catalog that does not match the binary when serve starts, and take the migration lock before any change. With every backend on the runner, this PR removes the code that served a backend without catalog migrations: the bootstrapper's migration methods, the old branches of init, migrate and verify, serve's resource-tag marker check and OperationsEngine::catalog_version.

It is stacked on #666, and its diff is against feat/catalog-migrations. The design is ADR-0023, which now has Cassandra and MongoDB rows. The first three commits are the code and tests, then CI with the check scripts and the committed schemas, then the docs. The fourth to sixth change test code only: the fourth fixes two tests that failed on the first CI run, the fifth sends each migrator of the two-node test to its own node, and the sixth tightens the proxy test's first check. All three are described under Testing.

Fixes #637 (schema agreement on every node before the next statement). Refs #636, #346, #470 (permissions under CassandraAuthorizer).

The migrations on each backend

Id	Cassandra	MongoDB
1 baseline	One file for the catalog keyspace and one for each account keyspace, with every statement qualified by its keyspace. Brings every released catalog to the v1.0 schema.	Code for the catalog and data databases. "Already exists" is recognised by its error code (48) only. Creates the data indexes an init that stopped part way left out.
2 resource_tag_condition_keys	The shared policy rewrite.	The shared policy rewrite.
3 sort_key_definitions	Not applicable: the Cassandra backend first shipped after the #259 fix.	Not applicable: MongoDB's form of that defect is #648.
4 index_table_base_key	Not applicable: index tables are keyed by their own primary key.	Not applicable: index collections get their compound index when they are created.

Each store records what it applied in its ledger, and the first time migration 1 is applied the runner compares the result with the expected definitions (system_schema on Cassandra, collection options and index keys on MongoDB) and records nothing on a difference. Migration 1 also deletes the marker row 0.1.x servers require, and writes the legacy schema_history rows (Cassandra) or 001_initial document (MongoDB) so a 0.1.x migrate finds nothing to do.

What changes for operators

• extenddb migrate, migrate --yes, --list, --forget, --reconverge, --lock-timeout and verify work on Cassandra and MongoDB as they do on SQLite and PostgreSQL, with the same exit codes.
• serve on Cassandra and MongoDB refuses a catalog whose status is not Current or AheadCompatible, before it opens the engine, and prints the command that resolves it. On MongoDB, serve did not read the catalog version before.
• migrate --yes and init refuse a keyspace or database that is not a catalog, including another program's keyspace whose settings table has the catalog's columns, and write nothing to it. On Cassandra the refusal names what the keyspace holds.
• migrate --yes and --forget read the status under the lock as its holder. On a catalog behind migration 2 that read scans every IAM principal's policies, and each request it sends counts as progress, so the lock is renewed through the scan.
• A run that stopped or failed part way finishes when migrate --yes runs again. When migration 2 cannot read or write a policy, its error ends with the first failure's text, such as a lapsed lock or a node that did not answer.
• Cassandra:
  • The migration lock is a row in the catalog's settings table, written by lightweight transactions at SERIAL, with a 60 s TTL renewed every 15 s while the run makes progress. Each server registers at QUORUM and renews at LOCAL_QUORUM, and migrate --yes reads the registrations at ALL, waits up to --lock-timeout for running servers, then exits 6 naming them. A server that cannot renew its registration for 30 s exits 1 and removes its PID file.
  • Each request under the lock is bounded by 10 s and each schema statement by 20 s. After every schema statement, the runner waits until every live node reports the same schema version. A node that does not answer is not counted as agreeing.
  • init creates the catalog keyspace and its settings table, then takes the lock and runs the rest under it, the default account's migrations included. Under an authorizer other than AllowAllAuthorizer it grants the application role its permissions, and migrate grants them on an account keyspace it creates. The upgrade manual gives the statements for deployments created before 1.0.
  • CreateAccount runs the account keyspace's migrations through the runner, under fix(storage-cassandra): serialize migrations to prevent keyspace contamination #453's lock.
• MongoDB:
  • The migration lock and the server registrations are documents written with majority write concern, compared against the replica set primary's clock. The same wait, refusal and 30 s rule apply.
  • A user with readWrite on extenddb_catalog and extenddb_data and no other role can run every command.
  • Commands connect with 10 s connect and server selection timeouts, and every request under the lock is bounded by 10 s.
• When several account keyspaces have the same migration pending, the status names it once with the number of stores.

What changes for contributors

• docs/schema/cassandra.txt and docs/schema/mongodb.txt hold the current schema, and CI fails when a fresh install differs.
• CONTRIBUTING.md "Adding a catalog migration" now covers all four backends.
• CI gains, for Cassandra, steps for tests/migrations.rs and tests/migrations_stopped.rs, a two-node job, a CassandraAuthorizer job and the catalog checks; for MongoDB, the catalog migration tests and the catalog checks (schema, lock, access control, released catalogs, the last 0.1.x image). Both set their *_REQUIRED variable so the tests fail rather than skip without a server.
• test_table_operations_work_on_a_catalog_without_v005 (fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores #384) is removed. A catalog without migration 1 reads as Behind, and serve refuses it, so no table operation runs on one.

Testing

Run before the push on 30 October on one host, against Cassandra 4.1 containers (one node, two for the two-node job), MongoDB 7 replica sets, PostgreSQL and SQLite:

• Every job of the test and integration workflows on all four backends, at the head where both backends were first merged onto feat(migrations): one list of catalog migrations with a ledger in every store, for SQLite and PostgreSQL #666:
  • clippy (1.97.0) on the seven feature builds, the 1.88 build and cargo test --workspace (2715 passed);
  • the Rust integration suite on Cassandra, PostgreSQL, SQLite and MongoDB (512 each), and the PostgreSQL storage tests (314);
  • pytest on SQLite (1486 passed), PostgreSQL (1479) and MongoDB (1478, plus one failure of test_export_dynamodb_json, which passed three times out of three alone), and the --comprehensive suite on each (340);
  • the schema, lock and older-release checks on PostgreSQL and SQLite, the PostgreSQL lock check behind PgBouncer as well, and the schema, lock, access-control and released-catalog checks on MongoDB.
• Cassandra:
  • at the final code, the unit tests, tests/migrations.rs and tests/migrations_stopped.rs;
  • the lock, older-release and CassandraAuthorizer checks, with a binary built before the last round, which changed migration 2's error text, test code and comments;
  • the schema check at the merged head;
  • the two-node job at the final code: tests/migrations.rs on a two-node cluster;
  • pytest with two workers against the merged head's server (1471 passed, 1 failed), and the --comprehensive suite (340 passed). The failure is test_racing_puts_on_an_lsi_table_leave_only_the_final_entry, a create-race test from fix(postgres): order a PutItem that loses a create race after the winner #388 that fails as storage-cassandra: a PutItem that loses the create race can leave its LSI row behind #414 describes: racing puts leave a stale index entry beside the final one. Run alone with this PR's server it failed 2 of 5 runs, and its GSI twin failed 4 of 6 runs alone with feat(migrations): one list of catalog migrations with a ledger in every store, for SQLite and PostgreSQL #666's server. This PR does not change Cassandra's item or index write path.
• Each test added in the last fix rounds was run with its fix reverted and failed: the lease on init's default-account run, the agreement wait's progress, the status read under the lock, the version row of a stopped run, the schema statement bound, and migration 2's error text.
• Failures from outside this PR: test_ttl_enable_rejects_asynchronous_gsi, which fails the same way on feat(migrations): one list of catalog migrations with a ledger in every store, for SQLite and PostgreSQL #666's head and on main (Cassandra ttl_integration: test_ttl_enable_rejects_asynchronous_gsi asserts a refusal wording #384 replaced; masked by continue-on-error #566), in a step CI runs with continue-on-error (Feat/storage cassandra in tree #338), and the backup-store conformance tests, which refuse a backup root under the test host's /tmp because another user owns it. This PR does not touch that crate.

The first CI run, on the third commit, passed 33 of its 36 checks. Two Cassandra tests failed, and the third failure is the job that requires every Cassandra job to pass. The fourth commit changes test code only:

• the_status_read_under_the_lock_keeps_it_through_a_long_scan stopped in its fixture, which created 4000 users with create_user, 32 at a time. create_user inserts a user with a lightweight transaction, and every fixture user is in the same account's partition, so the transactions contend for it. The CI log does not name the failing statement: create_user reports a driver error from either of its writes, the transaction and the self-service policy insert, as Database error. The transaction is the one statement the new fixture no longer sends. The fixture now inserts the user row without the transaction, with the self-service policy create_user adds. On a Cassandra node limited to one CPU, tests/migrations_stopped.rs failed in both runs with the old fixture and passed in both with the new one.
• a_bodys_request_to_a_node_that_stops_answering_fails_at_the_request_timeout, in the two-node job: the driver read the second node from system.peers_v2 and sent the request that follows the proxy going silent straight to that node, which answered in 7 ms. The driver sends requests to the nodes in turn, and on the local two-node cluster that request fell to the proxy, so the old test passed there in 8 runs out of 8. The test now uses a session whose query plans hold only the proxy. On the local cluster the old session's plan held both nodes, and the new session's plan holds the proxy only.
• At the fourth commit, tests/migrations.rs passed on one node and twice on two nodes, and tests/migrations_stopped.rs passed on one node and twice on the node limited to one CPU. clippy (1.97.0) and rustfmt pass on the crate.

The second CI run, on the fourth commit, passed all 36 checks. The fifth commit also changes test code only:

• two_migrators_started_together_apply_a_legacy_catalog_once gives each migrator a session whose query plans hold its own node only, so on two nodes the two migrators always reach the catalog through different coordinators. Before, each migrator's session sent requests to both nodes in turn. The schema agreement wait still asks every node for its version, on a connection to each. On the local two-node cluster, each such session's plan held one node, a different one for each, and the agreement wait finished through both.
• That session gives up on the connect after 10 s, as CassandraEngine::create_session does, so a node that cannot be reached fails a test after 10 s instead of at the job's timeout. The proxy test's session comes from it.
• The proxy test checks that the request sent before the proxy goes silent is answered by the node, which reports that the keyspace does not exist. With the proxy silenced before that request, the test now fails at that check; the earlier check accepted any error, the timeout included.
• The fixture binds password_hash to null, as create_user does.
• At the fifth commit the tests ran on a host whose load average was between 29 and 48 on 16 cores. The changed tests passed in every run: the two-migrator and proxy tests in three runs of tests/migrations.rs on two nodes and one on one node, and the two tests that use the fixture in two runs of tests/migrations_stopped.rs on one node. tests/migrations.rs passed in full on one node and in one of the three runs on two nodes. The other runs had five failures in four tests this commit does not change, each on a timing bound: a schema statement past its 20 s bound three times, and a lock left by a killed holder taken again 8.9 s and 26 s after the kill, against a bound of 8.5 s. Each of the four passed twice when run alone. clippy (1.97.0) and rustfmt pass on the crate.

The third CI run, on the fifth commit, passed all 36 checks after one rerun of mongodb-pytest, whose --comprehensive run failed test_all_writers_succeed as #577 describes. The sixth commit also changes test code only:

• The proxy test's first check requires the proxy's address in the answer as well as "does not exist": the error a node returns names the server the driver read it from. On the local two-node cluster, a session built as before the fifth commit failed that check in three runs out of three, answered by the second node, and the pinned session passed it in three out of three.
• pinned_session resolves its contact point first, so a host name works, as it does for CassandraEngine::create_session, and its doc comments say what its query plans hold.
• At the sixth commit, tests/migrations.rs passed on one node, and on two nodes with host names as contact points. clippy (1.97.0) and rustfmt pass on the crate.

Measured:

• Cassandra, 75000 IAM users with one dynamodb:ResourceTag inline policy each and migration 2 pending, migrate --yes with production lease times. A binary that read the status under the lock with no lease stopped renewing the lock three minutes in and exited 1 after 308 s, with migration 2 not recorded. This PR's binary then applied migration 2 on the same catalog in 815 s, the ledger row names all 75000 rewrites, and verify passes.
• MongoDB, 9000 users the same way: before the fix that keeps the lock through migration 2's scan, migrate --yes exited 1 after 75 s; with it, 0 after 125 s with migration 2 recorded.
• Cassandra, the node paused for 50 s while init ran the default account's migrations: init stopped at its next schema statement's 20 s bound and recorded nothing. Before the fix, the account run went on writing 22 s past the lock's deadline.
• Cassandra under CassandraAuthorizer: the upgrade manual's GRANT statements for deployments created before 1.0 give the application role the same ten permissions init grants.