Deferred indexes -- background-build

[michal-morzywolek]

Summary

Adds a new opt-in mechanism for building long-running indexes outside the upgrade window. Indexes declared with .deferred() are queued in a DeferredIndexes infrastructure table during the upgrade step; the adopter later drives them to completion by running the Runnables returned from DeferredIndexService.getBuildTasks().

The build task is self-healing: it observes physical state via a new SqlDialect.isIndexValid on every invocation, so a routine JVM restart mid-CREATE INDEX does not boot-loop the application. Only operator-caused corruption of a COMPLETED row (missing or INVALID physical) surfaces as fatal drift, and even then every anomaly across the schema is collected and reported in a single IllegalStateException per boot cycle.

API additions (all additive)

• New public interfaces in org.alfasoftware.morf.upgrade.deferredindexes:
  • DeferredIndexService -- getBuildTasks(): List<DeferredIndexBuildTask> + getProgress(): Map<DeferredIndexStatus, Integer>
  • DeferredIndexBuildTask extends Runnable -- adds getTableName()/getIndexName() identity + getStatus()/getAttemptsCount()/getErrorMessage() snapshot getters for adopter-side filtering.
• New default methods on SqlDialect:
  • Optional<Boolean> isIndexValid(Connection, String, String) -- implemented for PostgreSQL (pg_index.indisvalid), Oracle (USER_INDEXES.STATUS), both H2s (existence in INFORMATION_SCHEMA.INDEXES). Default empty; adopters on MySQL/SQL Server get graceful fallback.
  • Optional<String> setLockTimeoutSql(Duration) / resetLockTimeoutSql() -- PostgreSQL-only (SET lock_timeout = <ms> + RESET lock_timeout), used to bound the DROP INDEX wait when reconciling an in-flight leftover.
  • boolean deferredIndexBuildRequiresAutoCommit() -- PostgreSQL returns true (CREATE INDEX CONCURRENTLY can't run in a transaction block); the build task flips and restores autocommit around the work.
• New infrastructure upgrade step CreateDeferredIndexes, new DeferredIndexes table (11 columns, PK on id, unique index on (tableName, indexName), non-unique on status for fast non-terminal lookup).

Adopters not opting into the feature see zero behaviour change: UpgradeConfigAndContext.setDeferredIndexCreationEnabled(true) is required to activate the pipeline, and dialects returning supportsDeferredIndexCreation()=false silently strip the .deferred() flag and build immediately.

Interaction with _PRF (ignored) indexes

When the visitor's emitPhysicalIndexIfNeeded sees a target index whose columns + isUnique match a _PRF* entry in UpgradeConfigAndContext.setIgnoredIndexes, it emits RENAME INDEX PRF -> newName instead of CREATE INDEX newName -- regardless of the deferred flag. For a .deferred() target this materialises the physical immediately (cheaper than a background CREATE); the row is still registered as PENDING and self-heals to COMPLETED on the adopter's next build pass via isIndexValid. See integration guide Section 11.1 for the full behaviour matrix.

Docs

Two standalone documents ship with the feature:

• Adopter integration guide -- covers the getBuildTasks() recommended loop pattern, CommonJ Work wrapper, single-node requirement, state matrix (12-cell self-heal table), per-dialect notes, _PRF rename interaction, and known limitations.
• Design / testing / risks doc for the JIRA ticket -- glossary, requirements, solution overview, per-task algorithm, self-heal state matrix, INT-R1..INT-R3 regression scenarios + INT-01..INT-70 positive/edge scenarios (each row lists the specific test method backing it), technical approach, risks, assumptions & decisions.

Test plan

• mvn clean verify green across all 10 modules
• 38 integration tests in TestDeferredIndexesIntegration (real H2, end-to-end from Upgrade.performUpgrade through adopter's getBuildTasks() loop)
• Unit test suites for every new production class (builder, service, session, statements, enricher, registration policy, deferred-index POJO, build-task holder)
• Per-dialect surface tests: TestPostgreSQLDeferredIndexSupport (13), TestOracleDeferredIndexSupport (8), TestH2DeferredIndexSupport (7 tests x both H2 modules)
• Latest merge from origin/main (16 commits, incl. the PRF-rename optimisation) integrated; three-way conflict in AbstractSchemaChangeVisitor.visit(ChangeIndex) resolved by lifting the PRF-rename optimisation into the shared emitPhysicalIndexIfNeeded helper (used by both AddIndex and ChangeIndex for symmetric behaviour). PRF x deferred intersection tests added (INT-66..INT-70) to cover the full-flow scenarios neither side had integration coverage for.
• Checkstyle + SpotBugs pass

PRF-rename correctness fix

Reviewing the merge surfaced a defect in the first cut of the _PRF interaction, now fixed on the branch.

Satisfying a declared-deferred index by renaming a shape-matching PRF leaves it physically present the instant the upgrade script runs, but the row was still registered PENDING. That produced the one state DeferredIndexSession's central invariant excludes — registered, non-terminal, and physically present — so isAwaitingBuild returned a false positive for an index already in the database, and the visitor suppressed the DDL of any later change to it. Reproduced across upgrades, in the explicitly-supported case where the adopter has not drained the build queue before the next upgrade:

• removeIndex — DROP suppressed, physical orphaned
• renameIndex — RENAME suppressed, row records a name the database does not have
• changeIndex — DROP of the from-index suppressed, so the replacement CREATE hit Index "PRODUCT_NAME_1" already exists and failed the upgrade outright

Fixed by registering such rows as COMPLETED with completedTime set (DeferredIndexSession.registerCompletedIndex), which both restores the invariant and means a PRF-materialised index never enters the build queue — getBuildTasks() and getProgress() no longer report phantom outstanding work. Three integration tests were written first and confirmed failing (INT-71..INT-73), plus session-level tests pinning the invariant directly (INT-74).

There is no known outstanding limitation in this area.