You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(lb): capacity-safe rolling target replacement and UID-owned cleanup - #70
This is a focused follow-up to the kube-rs controller refactor, addressing pre-existing target replacement behavior without introducing a persistent state machine.
Behavior
Hetzner LB ownership remains tied to the Kubernetes Service UID, including best-effort cleanup of deleting Services without a finalizer.
Reject oversized desired configurations before mutating Hetzner resources. Count unmanaged/non-IP targets against the provider's occupied capacity, but never remove them.
Pick one deterministic target operation from freshly observed provider state: add a missing target if capacity exists; otherwise remove one obsolete IP target to free a slot; finally remove any remaining obsolete IPs.
Await the returned Hetzner target Action (2-second fixed polling, 60-second end-to-end timeout including stalled status HTTP requests). An accepted HTTP response alone is not treated as completion; Action errors and timeouts propagate into the existing retry/backoff policy.
After each completed mutation, requeue in 1 second. Recompute current desired nodes and observe the LB again before attempting another mutation. No persistent Action IDs, no target operation queue.
Await asynchronous LB type-change Actions before observing capacity again. Don't plan target updates using a stale LB type.
Publish Kubernetes Service ingress status as soon as Hetzner assigns an LB address, even while target reconciliation is pending; do not memoize incomplete target state.
No additional readiness logic: Hetzner LB health checks determine whether a backend serves traffic.
Failure handling
Existing per-Service exponential backoff (5s through 300s) and project-wide HTTP 429 gate remain unchanged. No exponential backoff for running Hetzner Actions. Unknown outcomes after polling timeouts are not marked successful; a later reconciliation reads provider state again.
Tests
Deterministic add-first selection with free capacity and remove-one-first at capacity
Sequential completion and re-observation; converged state generates no API writes
Failed addition does not prematurely remove an existing target
Unmanaged targets consume capacity and are never removed
Repeated single-step reconciliation respects target capacity across combinations
Polling a running Action, surfacing an asynchronous Action failure, and timing out stalled HTTP status requests
Known trade-offs
At full capacity, replacing a target requires first removing an obsolete target, temporarily reducing the backend count. Zero-downtime replacement in that case requires spare capacity or resizing.
Action success is not a health-readiness guarantee; Hetzner LB health checks handle backend readiness.
A process crash during an in-flight Action loses the transient Action ID; subsequent reconciliation relies on provider re-observation and existing retry handling.
Async completion for other operations (e.g., listener changes and LB deletion) is outside this targeted follow-up.
Fixed by running cargo fmt in commit 1068169 (collapsed the multi-line usize::try_from(...).unwrap_or(0) and removed the extra blank line before the closing brace in src/lb/mod.rs). CI is now green on this branch.
The CI failure was cargo fmt reformatting src/lb/mod.rs. Commit 1068169 already contains the fix and is pushed; verified cargo fmt --check passes and all checks (pre-commit, docker-smoke, helm-unittest) are green on the branch head. Posted a brief comment on PR #70.
pando85
changed the title
fix(controller): minimize target replacement disruption and unnecessary cleanup
draft: evaluate pre-existing target replacement and cleanup behavior
Oct 13, 2026
pando85
changed the title
draft: evaluate pre-existing target replacement and cleanup behavior
fix(lb): capacity-safe rolling target replacement and UID-owned cleanup
Oct 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Capacity-safe, action-aware reconciliation
This is a focused follow-up to the kube-rs controller refactor, addressing pre-existing target replacement behavior without introducing a persistent state machine.
Behavior
Failure handling
Existing per-Service exponential backoff (5s through 300s) and project-wide HTTP 429 gate remain unchanged. No exponential backoff for running Hetzner Actions. Unknown outcomes after polling timeouts are not marked successful; a later reconciliation reads provider state again.
Tests
Known trade-offs