Skip to content

fix(lb): capacity-safe rolling target replacement and UID-owned cleanup - #70

Merged
pando85 merged 12 commits into
masterfrom
fix/reconciliation-safety-followup
Oct 13, 2026
Merged

pando85 merged 12 commits into
masterfrom
fix/reconciliation-safety-followup

Conversation

@pando85

@pando85 pando85 commented Oct 13, 2026 •

Copy link
Copy Markdown

Capacity-safe, action-aware reconciliation

This is a focused follow-up to the kube-rs controller refactor, addressing pre-existing target replacement behavior without introducing a persistent state machine.

Behavior

  • Hetzner LB ownership remains tied to the Kubernetes Service UID, including best-effort cleanup of deleting Services without a finalizer.
  • Reject oversized desired configurations before mutating Hetzner resources. Count unmanaged/non-IP targets against the provider's occupied capacity, but never remove them.
  • Pick one deterministic target operation from freshly observed provider state: add a missing target if capacity exists; otherwise remove one obsolete IP target to free a slot; finally remove any remaining obsolete IPs.
  • Await the returned Hetzner target Action (2-second fixed polling, 60-second end-to-end timeout including stalled status HTTP requests). An accepted HTTP response alone is not treated as completion; Action errors and timeouts propagate into the existing retry/backoff policy.
  • After each completed mutation, requeue in 1 second. Recompute current desired nodes and observe the LB again before attempting another mutation. No persistent Action IDs, no target operation queue.
  • Await asynchronous LB type-change Actions before observing capacity again. Don't plan target updates using a stale LB type.
  • Publish Kubernetes Service ingress status as soon as Hetzner assigns an LB address, even while target reconciliation is pending; do not memoize incomplete target state.
  • No additional readiness logic: Hetzner LB health checks determine whether a backend serves traffic.

Failure handling

Existing per-Service exponential backoff (5s through 300s) and project-wide HTTP 429 gate remain unchanged. No exponential backoff for running Hetzner Actions. Unknown outcomes after polling timeouts are not marked successful; a later reconciliation reads provider state again.

Tests

  • Deterministic add-first selection with free capacity and remove-one-first at capacity
  • Sequential completion and re-observation; converged state generates no API writes
  • Failed addition does not prematurely remove an existing target
  • Unmanaged targets consume capacity and are never removed
  • Repeated single-step reconciliation respects target capacity across combinations
  • Polling a running Action, surfacing an asynchronous Action failure, and timing out stalled HTTP status requests

Known trade-offs

  • At full capacity, replacing a target requires first removing an obsolete target, temporarily reducing the backend count. Zero-downtime replacement in that case requires spare capacity or resizing.
  • Action success is not a health-readiness guarantee; Hetzner LB health checks handle backend readiness.
  • A process crash during an in-flight Action loses the transient Action ID; subsequent reconciliation relies on provider re-observation and existing retry handling.
  • Async completion for other operations (e.g., listener changes and LB deletion) is outside this targeted follow-up.

@forkline

forkline Bot commented Oct 13, 2026

Copy link
Copy Markdown

Fixed by running cargo fmt in commit 1068169 (collapsed the multi-line usize::try_from(...).unwrap_or(0) and removed the extra blank line before the closing brace in src/lb/mod.rs). CI is now green on this branch.

@forkline

forkline Bot commented Oct 13, 2026

Copy link
Copy Markdown

✅ Fix pushed:

The CI failure was cargo fmt reformatting src/lb/mod.rs. Commit 1068169 already contains the fix and is pushed; verified cargo fmt --check passes and all checks (pre-commit, docker-smoke, helm-unittest) are green on the branch head. Posted a brief comment on PR #70.

@pando85
pando85 marked this pull request as draft October 13, 2026 17:56
@pando85 pando85 changed the title fix(controller): minimize target replacement disruption and unnecessary cleanup draft: evaluate pre-existing target replacement and cleanup behavior Oct 13, 2026
@forkline

forkline Bot commented Oct 13, 2026

Copy link
Copy Markdown

⏸️ CI Auto-Fix paused

A user commit was detected after an automated fix. CI auto-fix is now paused
to prevent conflicts with your changes.

To resume CI auto-fix, use the /reset or /reset-ci-auto-fix command.

@pando85 pando85 changed the title draft: evaluate pre-existing target replacement and cleanup behavior fix(lb): capacity-safe rolling target replacement and UID-owned cleanup Oct 13, 2026
@pando85
pando85 force-pushed the fix/reconciliation-safety-followup branch from 5a12f4b to 3dd72d0 Compare October 13, 2026 20:42
@pando85
pando85 force-pushed the fix/reconciliation-safety-followup branch from 3dd72d0 to 7d8c471 Compare October 13, 2026 20:59
@forkline

forkline Bot commented Oct 13, 2026

Copy link
Copy Markdown

⚠️ Task Interrupted: AI Provider Did Not Complete the Request

Your AI provider started processing this task but did not return a complete final response.

What to try:

  1. Check that your AI provider and model are working normally
  2. Verify your provider credentials and account limits in Forkline Settings
  3. Try again in a few minutes
  4. If this keeps happening, try a different model or provider

Runner: da4

@pando85
pando85 marked this pull request as ready for review October 13, 2026 21:14
@pando85
pando85 merged commit 70667ee into master Oct 13, 2026
3 checks passed
@pando85
pando85 deleted the fix/reconciliation-safety-followup branch October 13, 2026 21:17
@pando85 pando85 mentioned this pull request Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant