Repository navigation
upgrade-drain: retry a coordinator call that answers Unavailable - #1781
Merged
Merged
Conversation
The host-fleet pre-upgrade drain failed on its first call: list_hosts got `Unavailable` (connection refused). The coordinator rollout had finished seconds before, so the Service still routed to a stopping pod. The drain loop retries its actions, but a failed list_hosts or list_sessions at the top of a pass ended the hook. Every call of the drain's Coordinator client now goes through one retry: on `Unavailable` it tries again every 2 s, up to 30 attempts (about a minute). Any other status fails at once. The retry counts attempts, so it reads no clock (ADR 0098). Both hooks (`upgrade-drain` in the coordinator and host-operator images, and `upgrade-roll`) use this client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e
The coordinator answers `Unavailable` for two different things: it is not reachable, and its own call to a host failed. The retry treated both the same, so an evict, snapshot or resume of a session on a dead host waited a minute per call inside its pass, and held the other actions of that pass, where it failed in seconds before. The drain already sends a failed action again on its next pass. `list_hosts` and `list_sessions` keep the retry: their failure ends the hook. Cordon, the checkpoint read, snapshot, resume and evict get one try. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A host-fleet
helm upgradefailed at its pre-upgrade drain hook: the hook's first call,list_hosts, gotUnavailable(connection refused). The coordinator Deployment had reported its rollout complete seconds earlier and both new pods were listening, so the Service still routed to a stopping pod. A rerun of the deploy passed.The two fleet reads of the drain's
Coordinatorclient (engram-upgrade-drain),list_hostsandlist_sessions, now retry: onUnavailablethey try again every 2 s, up to 30 attempts (about a minute). Any other status fails at once. The budget counts attempts, so the code reads no clock (ADR 0098). A failure of one of these reads ends the hook, so they are the calls that must wait for the coordinator. Both hooks use this client:upgrade-drain(coordinator and host-operator images) andupgrade-roll.The actions (cordon, the checkpoint read, snapshot, resume, evict) get one try, as before. The coordinator also answers
Unavailablewhen its own call to a host fails, and the drain already sends a failed action again on its next pass; a wait inside the call would hold the other actions of that pass for a minute per call on a session whose host is dead.Test plan
Ok);PermissionDeniedfails after 1 call; retries stop at the attempt limit; an action gets one try.cargo clippy -p engram-upgrade-drain -p engram-host-operator --all-targets -D warningscargo nextest run -p engram-upgrade-drain: 8/8🤖 Generated with Claude Code
https://claude.ai/code/session_017XkqDqWbQdPXdy94AdFV5e