Repository navigation
fix(auth): skip reserved credentials in retry and cooldown-wait checks (HOUSE-145) - #34
Conversation
…s (HOUSE-145) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
✅ No blocking findingsWalkthroughWhen quota reserves are configured, the retry and cooldown-wait checks now treat a reserved credential as unavailable, matching what selection already does. Before this change, a request that arrived while the usable accounts were in a short transient cooldown counted the reserved account as ready. It retried with no wait, selection excluded that account again, and the request failed at once with
Details
First review of this pull request. @unfret review · @unfret status · @unfret help |
✅ Merged into
|
Summary
With a quota reserve active, a request that arrived while the usable accounts were in a short transient cooldown failed at once with
503 auth_unavailableinstead of waiting and retrying. The retry and cooldown-wait checks counted the reserved account as ready, so the retry ran with zero wait and selection then excluded the same account again.shouldRetryAfterErrorWithAttempted retryAllowed # may another round run? m.retryRoundAvailabilityForAuth closestCooldownWaitWithAttempted # how long to wait first? m.retryRoundAvailabilityForAuth + if m.quotaReserveReached(auth, now) + return not available isAuthBlockedForModel ...retryRoundAvailabilityForAuthbecomes aManagermethod so both scans apply the same reserve rule that selection already applies.Evidence
Incident on x11, 2026-11-20 15:09 to 15:15 EST. OpenAI's Codex backend returned intermittent
503 upstream connect erroron both usable Codex accounts. The proxy returned 256 responses with status 503, and 181 of them took under 50ms without reaching OpenAI. The third Codex account was held back by its 20% reserve. Config:request-retry: 1,max-retry-interval: 10,transient-error-cooldown-seconds: 5.4e348c7e, new tests):TestExecuteWaitsForCooldownWhenRemainingCredentialIsReserved: FAIL,auth_unavailable: no auth available (last upstream error: upstream connect error), the same message as in production.TestShouldRetryAfterErrorIgnoresReservedCredentials: FAIL in both cases,(0s, true), which means an immediate retry.231cffaf): both tests PASS. The second request waits for the earliest usable account's recovery, then succeeds. The reserved account gets zero executor calls.The Execute test uses
testing/synctest, so it does not depend on wall-clock timing.Merge Danger
Door: Two-way. Revert the commit to restore the old behavior. It has no state or config changes.
Blast Radius: Retry timing.
This applies only when
routing.quota-reservesis configured. A request that arrives while every non-reserved account is cooling now waits up tomax-retry-interval, instead of failing in about 1ms. When every eligible account is reserved, or the request is pinned to a reserved account, it still fails at once. Home mode rejects reserves, so it is unaffected.Acceptance criteria
max-retry-interval) and then retries, instead of failing at once."Verified by
TestExecuteWaitsForCooldownWhenRemainingCredentialIsReserved: it fails before the fix and passes after.Verified by
TestShouldRetryAfterErrorIgnoresReservedCredentials, with two cases: every credential reserved, and a pin to a reserved credential. It also checks the zero reserved-account calls in test 1.Covered by the two tests above, in
sdk/cliproxy/auth/quota_reserve_retry_wait_test.go.Not yet. This follows the merge and is recorded on the ticket.
Gate results
Run on head
231cffafon studio2:go build -o test-output ./cmd/servergo vet ./sdk/cliproxy/auth/go test -count=1 ./sdk/cliproxy/auth/...go test -count=1 ./...sdk/cliproxyTestServiceHeartbeatFailoverDoesNotDrainBlockingScope(see below)gofmt -l .internal/pluginhost/host.go(see below)TestServiceHeartbeatFailoverDoesNotDrainBlockingScopefails 5 of 5 runs on main4e348c7etoo. Its message is "new subscription did not start while the old scope remained active". studio2's load average was about 30 at the time.internal/pluginhost/host.gois listed bygofmt -l .on main too. This PR does not touch that file.Not verified
Linear: HOUSE-145
🤖 Generated with Claude Code
Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.