fix(backup): restore snapshot jobs and verify remote backup bytes by devantler · Pull Request #5406 · devantler-tech/platform

devantler · 2026-10-26T23:46:51Z

Why

The daily backup cannot run on its current storage. Moving company Email credentials must wait until recovery is proven.

What

Move snapshot jobs to the prepared recovery storage while preserving the original. Require matching downloaded backup bytes before reporting success; complete the separate restore proof before moving credentials.

Part of #5376

Part of #5376. Preserve the original claim, require active storage binding, and check actual downloaded bytes before remote retention. Restore acceptance remains separate.

…t-cutover-5376-23e2

Represent partial changes without declaring incomplete workloads. Confirm the indexed field names and old values before replacement. Normalized production render remains identical; no scanner rules or runtime privileges change.

…t-cutover-5376-23e2 # Conflicts: # scripts/validate-eks-ci-role-policy/approved-surface-history.md

Preserve the existing snapshot authentication role and token path with a bounded explicit projection. Disable automatic mounting so the new tool installer and remote mirror receive no Kubernetes token. Preserve both reviewed authority records while merging current main.

devantler · 2026-10-27T01:21:50Z

@coderabbitai review

coderabbitai · 2026-10-27T01:21:57Z

⚠️ Action not completed

Review rate limited.

coderabbitai · 2026-10-27T01:22:04Z

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 54 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

Source mode: Auto
Repository: devantler-tech/platform
Branch: codex/openbao-snapshot-cutover-5376-23e2
File: .coderabbit.yaml
Commit: 2985369
Layers: Repository YAML (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Advanced
Run ID: 0d5235ce-1f4f-4f86-a5e1-8503ef638a7c

Commits

Reviewing files that changed from the base of the PR and between 6da2e00 and 2985369.

Files selected for processing: 15

.github/workflows/ci.yaml
docs/dr/openbao.md
k8s/bases/infrastructure/vault-backup/cron-job.yaml
k8s/bases/infrastructure/vault-backup/job.yaml
k8s/clusters/base/flux-kustomization-infrastructure.yaml
k8s/providers/hetzner/infrastructure/kustomization.yaml
k8s/providers/hetzner/infrastructure/vault-backup-recovery/patches/use-recovery-storage-for-daily-snapshot.yaml
k8s/providers/hetzner/infrastructure/vault-backup-recovery/patches/use-recovery-storage-for-initial-snapshot.yaml
k8s/providers/hetzner/infrastructure/vault-backup-recovery/persistent-volume-claim.yaml
scripts/tests/vault-backup-recovery/readback_test.go
scripts/tests/vault-backup-recovery/storage_test.go
scripts/tests/vault-backup-recovery/token_test.go
scripts/validate-eks-ci-role-policy/approved-surface-history.md
scripts/validate-eks-ci-role-policy/approved-surface.txt
scripts/validate-eks-ci-role-policy/main.go

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

devantler · 2026-10-27T01:55:53Z

@codex review

chatgpt-codex-connector · 2026-10-27T01:56:07Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-27T23:37:59.903243Z	`02d0491`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector · 2026-10-27T01:57:33Z

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 2985369ac0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

devantler · 2026-10-27T03:50:42Z

Blocker: none

Cleared 2026-10-27: Scope correction: snapshot repair is a direct dependency of the maintainer-requested Email custody and transparent-storage recovery work. It must not be parked as unrelated work merely because production Email availability #5412 remains open. Current main 094218a is merged at signed head 7d2dd43; the sole CI-selector conflict preserves both backup test families. Local backup, recovery and EKS policy suites plus Actionlint pass. Own current-head CI, review, and protected delivery remain unfinished readiness work; #5376 still requires an actual remote restore proof.

Previous record (historical):

> 🤖 Generated by the Agentic Engineer

<!-- pr-blocker-record -->

**Blocker:** devantler-tech/platform#5412 | upstream | last-verified 2026-10-27: The production Email startup incident is still open and the live platform health gate is failing. This PR is outside the active recovery and is not queued. Resume protected deployment after the incident and platform health recover, with fresh current-head validation, review and user evaluation.

devantler · 2026-10-27T22:35:29Z

@codex review

devantler · 2026-10-27T23:15:25Z

The current-head request at 22:35:29 UTC received its provider acknowledgment at 22:35:44 UTC. More than 35 minutes later, fresh review objects, threads and conversation evidence still contain no substantive result for this head, and the no-gate guard admits progression. This records a stalled request, not a successful review. The head still requires a substantive review before promotion.

devantler · 2026-10-27T23:16:37Z

devantler · 2026-10-27T23:16:38Z

@cursor review

cursor

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7d2dd43. Configure here.}

devantler · 2026-10-27T23:35:52Z

@codex review

chatgpt-codex-connector · 2026-10-27T23:37:58Z

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: 02d04917d4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

devantler · 2026-10-27T23:52:19Z

Ready at 02d04917d4ae025e36dfd8f582f59a104c011b95: all 53 reported checks are settled without failures, all five review threads are resolved, and Codex's current-head receipt is clean. Closing-keyword verification examined seven commits without prematurely closing the parent recovery issue.

User evaluation: the actual native regression job exercised pinned snapshot tooling for both production workloads. Matching remote reads passed; corrupt content and a complete body followed by a transport failure were refused. The rendered replacement-storage handoff passed, and actual isolated restores passed matching, corrupt-snapshot, different-record and wrong-unseal-key controls.

This activates the retained replacement snapshot cache and requires verified remote bytes before a snapshot job can succeed. Existing data and the original retained claim are preserved. Deployment, fresh production snapshot completion and the independent restore from #5470 remain required on #5376/#5407 before Email credential custody changes. The ordinary protected queue remains authoritative.

devantler added 2 commits October 27, 2026 00:45

fix(backup): activate retained snapshot storage with verified readback

3834c5d

Part of #5376. Preserve the original claim, require active storage binding, and check actual downloaded bytes before remote retention. Restore acceptance remains separate.

Merge remote-tracking branch 'origin/main' into codex/openbao-snapsho…

6f3963e

…t-cutover-5376-23e2

github-project-automation Bot added this to 🌊 Project Board Oct 26, 2026

github-project-automation Bot moved this to 🫴 Ready in 🌊 Project Board Oct 26, 2026

devantler mentioned this pull request Oct 26, 2026

Prove credential recovery from an independent backup #5407

Open

6 tasks

github-advanced-security Bot found potential problems Oct 26, 2026

View reviewed changes

devantler added a commit that referenced this pull request Oct 27, 2026

chore: agent-claim #5406 nonce=1ad9cb2603f591084c3329620fe5f803

c079655

devantler added 2 commits October 27, 2026 01:06

Merge remote-tracking branch 'origin/main' into codex/openbao-snapsho…

a7b1a5e

…t-cutover-5376-23e2 # Conflicts: # scripts/validate-eks-ci-role-policy/approved-surface-history.md

devantler added a commit that referenced this pull request Oct 27, 2026

chore: agent-claim #5406 nonce=d82c66d5c371bdf15c9c23eac89ddc48

ee93c2c

devantler added the blocked label Oct 27, 2026

chore: merge current main into backup recovery

7d2dd43

devantler added a commit that referenced this pull request Oct 27, 2026

chore: agent-claim #5406 nonce=6d6074fd8b0a6d69ab7c52e3aa33c5b5

b608fd3

devantler removed the blocked label Oct 27, 2026

devantler mentioned this pull request Oct 27, 2026

feat(backup): prove recovery from the independent snapshot store #5470

Merged

cursor Bot reviewed Oct 27, 2026

View reviewed changes

Comment thread k8s/providers/hetzner/infrastructure/vault-backup-recovery/persistent-volume-claim.yaml

fix(backup): preserve mirrored snapshot backup routing

02d0491

devantler added a commit that referenced this pull request Oct 27, 2026

chore: agent-claim #5406 nonce=ee42da3e471106565dc0946db392377d

96af8af

devantler marked this pull request as ready for review October 27, 2026 23:52

devantler added this pull request to the merge queue Oct 27, 2026

Merged via the queue into main with commit 64f6fc3 Oct 28, 2026
53 checks passed

devantler deleted the codex/openbao-snapshot-cutover-5376-23e2 branch October 28, 2026 05:24

github-project-automation Bot moved this from 🫴 Ready to ✅ Done in 🌊 Project Board Oct 28, 2026

Conversation

devantler commented Oct 26, 2026

Why

What

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devantler commented Oct 27, 2026

Uh oh!

coderabbitai Bot commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Oct 27, 2026

Review limit reached

Uh oh!

devantler commented Oct 27, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot commented Oct 27, 2026

Uh oh!

devantler commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devantler commented Oct 27, 2026

Uh oh!

devantler commented Oct 27, 2026

Uh oh!

devantler commented Oct 27, 2026

Uh oh!

devantler commented Oct 27, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

devantler commented Oct 27, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 27, 2026

Uh oh!

devantler commented Oct 27, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai Bot commented Oct 27, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 27, 2026 •

edited

Loading

devantler commented Oct 27, 2026 •

edited

Loading