fix(trino): recover rollouts and gate tenant readiness

[EDsCODE]

Summary

Trino rollouts can stall on obsolete candidates, and newly provisioned tenants can accept queries before their catalog or storage is usable.

• Retire outdated, never-admitted candidates through Gateway's identity and generation guards; preserve successful admissions and immutable receipts.
• Keep capacity reserved until cleanup and Gateway retirement complete, including recovery after lost responses.
• Require a catalog-store snapshot containing the tenant before opening its admission barrier, then wait for member acknowledgements.
• Verify a new Hoglake catalog's storage with a bounded write/read/delete using its tenant role. Failed checks remain pending for reconciliation.
• Existing catalogs skip the initialization probe. Unrelated catalog updates do not re-admit existing tenants.

Before:

flowchart LR
  A[Blueprint changes] --> B[Old candidate retries admission] --> C[Replacement blocked]
  D[New tenant enabled] --> E[Admission can precede catalog and storage readiness]
  classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
  classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
  class A,B,D phBlue;
  class C,E phYellow;

Loading

After:

flowchart LR
  A[Blueprint changes] --> B[Read Gateway state]
  B --> C[Active: resolve admission]
  B --> D[Preparing: claim guarded retirement]
  D --> E[Confirm cleanup, create replacement]
  F[New Hoglake tenant] --> G[Verify tenant storage] --> H[Publish catalog]
  H --> I[Members acknowledge containing revision] --> J[Admit tenant]
  classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
  classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
  class A,B,D,F,G,H,I phBlue;
  class C,E,J phYellow;

Loading

Validation

• Regression tests cover admission races, authority loss, checkpoint recovery, late catalog publication, controller restarts, storage denial, and probe cleanup.
• The existing E2E assertions remain unchanged. They exposed catalog visibility and storage access failures.
• Local just test-trino '' passed across the control-plane, admin, provisioner, and provisioning packages.
• Local just lint passed; Kubernetes-tagged lint found no new issues against the previous PR head. Fresh CI is pending.
• Production rollout and end-to-end service authentication remain unverified.

Deployment

Deploy the updated Duckgres controller through the normal rollout. Existing Gateway retirement guards provide the required coordination.
The control-plane role must assume tenant storage roles and reach S3; the managed composition already grants that trust.
The runbooks describe readiness and recovery verification. No UI changes.

🤖 Agent context

Codex (GPT-6) implemented this change with delegated implementation and safety review. Skills: debugging-ci-failures, writing-pr-descriptions, and signed-git-publish.

[github-actions]

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area	Added	Changed	Deleted
Test files	3	4	0
E2E/journey files	0	0	0
Workflow files	0	0	0

Signals

• Test cases: +20 / -0
• Assertions: +86 / -0
• Skips or known failures added: 0
• Workflow continue-on-error added: 0
• Workflow path filters added: 0
• Test commands removed from justfile: 0
• E2E/journey retry lines added: 0

Coverage risk: neutral or increased

No coverage-reduction warnings detected.