Skip to content

fix: switch backup CronJob from distopia-db-r to distopia-db-rw - #207

Merged
ro80t merged 2 commits into
mainfrom
fix/backup-use-rw-service
Aug 28, 2026
Merged

fix: switch backup CronJob from distopia-db-r to distopia-db-rw#207
ro80t merged 2 commits into
mainfrom
fix/backup-use-rw-service

Conversation

@ro80t

@ro80t ro80t commented Aug 28, 2026

Copy link
Copy Markdown
Member

Summary

  • distopia-db-r ("any" instance Service) consistently refused pg_dump's connection in real production (Connection refused, reproduced across the scheduled run and multiple manual retries), while distopia-db-rw kept working fine for the app the entire time.
  • Ruled out NetworkPolicy (same-namespace traffic was always allowed, both before and after PR fix: allow cnpg-system namespace through CNPG's NetworkPolicy #206's cnpg-system exception) and resource pressure (kubectl top showed the instance nowhere near its CPU/memory limits) as causes.
  • Root cause on the -r Service specifically remains unclear, but -rw is proven reliable in this environment, so k8s/db/backup-cronjob.yaml's DB_HOST now points at it instead. Revisit -r vs -rw if instances is ever raised above 1 for HA.

Test plan

  • kubectl create job --from=cronjob/distopia-db-backup distopia-db-backup-manual -n distopia completes successfully against distopia-db-rw
  • The next scheduled run (0 18 * * *) also completes successfully

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Improved PostgreSQL backup reliability by connecting backups through the read-write primary database endpoint.
    • Prevented recurring connection refusals encountered with the previous endpoint in production.

distopia-db-r ("any" instance) consistently refused pg_dump's connection
in real production (Connection refused, reproduced across the scheduled
run and multiple manual retries), while distopia-db-rw kept working fine
for the app the entire time. Ruled out NetworkPolicy (same-namespace
traffic was always allowed, before and after the cnpg-system exception)
and resource pressure (kubectl top showed the instance nowhere near its
limits) as causes -- root cause on the -r Service specifically remains
unclear, but -rw is proven reliable, so use it instead. Revisit if
instances is ever raised above 1 for HA.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

github-actions Bot pushed a commit that referenced this pull request Aug 28, 2026
@coderabbitai

coderabbitai Bot commented Aug 28, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

Next included review available in 23 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ea5dc0fa-eacd-4214-9903-e48f628ced2b

📥 Commits

Reviewing files that changed from the base of the PR and between 02e6834 and 219376f.

📒 Files selected for processing (1)
  • .github/workflows/ci.yml

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 4fb7613a-f8d3-4072-9867-e05850d7b7ba

📥 Commits

Reviewing files that changed from the base of the PR and between 6e0f91f and 02e6834.

📒 Files selected for processing (1)
  • k8s/db/backup-cronjob.yaml

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The PostgreSQL backup CronJob now connects to distopia-db-rw instead of distopia-db-r. Its comment documents the production connection issue and notes when to reassess the endpoint.

Changes

Backup endpoint

Layer / File(s) Summary
Update backup database host
k8s/db/backup-cronjob.yaml
The pg-dump container now uses the distopia-db-rw service. The comment documents the endpoint behavior and the single-instance condition.

Estimated code review effort: 2 (Simple) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 02e68

The backup job now targets the database endpoint that has been reliable in this environment. No actionable merge-blocking risk remains beyond normal checks and review.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: switching the backup CronJob from the read endpoint to the read-write endpoint.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/backup-use-rw-service

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

github-actions Bot pushed a commit that referenced this pull request Aug 28, 2026
@ro80t
ro80t merged commit 98b40e1 into main Aug 28, 2026
20 checks passed
@ro80t
ro80t deleted the fix/backup-use-rw-service branch August 28, 2026 09:03
github-actions Bot pushed a commit that referenced this pull request Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant