Skip to content

fix(auth): skip reserved credentials in retry and cooldown-wait checks (HOUSE-145) - #34

Merged
mergify[bot] merged 1 commit into
mainfrom
HOUSE-145-lane
Nov 21, 2026
Merged

mergify[bot] merged 1 commit into
mainfrom
HOUSE-145-lane

Conversation

@ericlitman

@ericlitman ericlitman commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

With a quota reserve active, a request that arrived while the usable accounts were in a short transient cooldown failed at once with 503 auth_unavailable instead of waiting and retrying. The retry and cooldown-wait checks counted the reserved account as ready, so the retry ran with zero wait and selection then excluded the same account again.

 shouldRetryAfterErrorWithAttempted
   retryAllowed                      # may another round run?
     m.retryRoundAvailabilityForAuth
   closestCooldownWaitWithAttempted  # how long to wait first?
     m.retryRoundAvailabilityForAuth
+      if m.quotaReserveReached(auth, now)
+        return not available
       isAuthBlockedForModel ...

retryRoundAvailabilityForAuth becomes a Manager method so both scans apply the same reserve rule that selection already applies.

Evidence

Incident on x11, 2026-11-20 15:09 to 15:15 EST. OpenAI's Codex backend returned intermittent 503 upstream connect error on both usable Codex accounts. The proxy returned 256 responses with status 503, and 181 of them took under 50ms without reaching OpenAI. The third Codex account was held back by its 20% reserve. Config: request-retry: 1, max-retry-interval: 10, transient-error-cooldown-seconds: 5.

  • Before (selection code from 4e348c7e, new tests):
    • TestExecuteWaitsForCooldownWhenRemainingCredentialIsReserved: FAIL, auth_unavailable: no auth available (last upstream error: upstream connect error), the same message as in production.
    • TestShouldRetryAfterErrorIgnoresReservedCredentials: FAIL in both cases, (0s, true), which means an immediate retry.
  • After (231cffaf): both tests PASS. The second request waits for the earliest usable account's recovery, then succeeds. The reserved account gets zero executor calls.

The Execute test uses testing/synctest, so it does not depend on wall-clock timing.

Merge Danger

Door: Two-way. Revert the commit to restore the old behavior. It has no state or config changes.

Blast Radius: Retry timing.

This applies only when routing.quota-reserves is configured. A request that arrives while every non-reserved account is cooling now waits up to max-retry-interval, instead of failing in about 1ms. When every eligible account is reserved, or the request is pinned to a reserved account, it still fails at once. Home mode rejects reserves, so it is unaffected.

Acceptance criteria

  1. "With a quota reserve active, a request that arrives while every non-reserved credential is in a transient cooldown waits for the cooldown (within max-retry-interval) and then retries, instead of failing at once."
    Verified by TestExecuteWaitsForCooldownWhenRemainingCredentialIsReserved: it fails before the fix and passes after.
  2. "A reserved credential never counts as available in the retry and wait checks."
    Verified by TestShouldRetryAfterErrorIgnoresReservedCredentials, with two cases: every credential reserved, and a pin to a reserved credential. It also checks the zero reserved-account calls in test 1.
  3. "A regression test covers this path."
    Covered by the two tests above, in sdk/cliproxy/auth/quota_reserve_retry_wait_test.go.
  4. "The fix is deployed to x11 and the deployed version is confirmed with a live request."
    Not yet. This follows the merge and is recorded on the ticket.

Gate results

Run on head 231cffaf on studio2:

Check Result
go build -o test-output ./cmd/server exit 0
go vet ./sdk/cliproxy/auth/ exit 0
go test -count=1 ./sdk/cliproxy/auth/... ok
go test -count=1 ./... 1 failure: sdk/cliproxy TestServiceHeartbeatFailoverDoesNotDrainBlockingScope (see below)
gofmt -l . lists internal/pluginhost/host.go (see below)
  • TestServiceHeartbeatFailoverDoesNotDrainBlockingScope fails 5 of 5 runs on main 4e348c7e too. Its message is "new subscription did not start while the old scope remained active". studio2's load average was about 30 at the time.
  • internal/pluginhost/host.go is listed by gofmt -l . on main too. This PR does not touch that file.

Not verified

  • The fix's behavior during a real upstream outage. That cannot be triggered on demand, so the unit tests are the proof.
  • The deployment to x11, which happens after the merge.

Linear: HOUSE-145

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

…s (HOUSE-145)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@unfret

unfret Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

✅ No blocking findings

Walkthrough

When quota reserves are configured, the retry and cooldown-wait checks now treat a reserved credential as unavailable, matching what selection already does. Before this change, a request that arrived while the usable accounts were in a short transient cooldown counted the reserved account as ready. It retried with no wait, selection excluded that account again, and the request failed at once with `503 auth_unavailable`. Now it waits up to `max-retry-interval` for a usable account to recover. `retryRoundAvailabilityForAuth` becomes a `Manager` method so it can check the reserve. The PR adds tests for the wait-then-succeed path (using `synctest`) and for not retrying when every credential is reserved or the request is pinned to a reserved one. The author reports that both tests fail before the fix and pass after it. Deployment to x11 is still pending.

Area What changes
Retry and cooldown wait with quota reserves Reserved credentials no longer count as available when deciding whether to retry or how long to wait. Requests now wait for a non-reserved account's cooldown instead of failing immediately. Requests still fail at once when every eligible account is reserved or the request is pinned to a reserved account.
Tests Adds `quota_reserve_retry_wait_test.go`, covering the cooldown wait (with zero executor calls on the reserved account) and the no-retry cases. Updates an existing contract test to call the new method form.
Details
Lane / stage Model Effort Outcome Duration
broad-gpt-5.6-sol-xhigh / review openai/gpt-5.6-sol → gpt-5.6-sol · primary xhigh succeeded 1 min 16 s
broad-opus-5.5-max / review anthropic/claude-opus-5-5 → claude-opus-5-5 · primary max succeeded 6 min 9 s
focal-gpt-5.6-sol-xhigh copy / review openai/gpt-6.1-sol → gpt-6.1-sol · primary xhigh succeeded 7 min 32 s
focal-gpt-5.6-sol-xhigh copy 2 / review openai/gpt-6.1-sol → gpt-6.1-sol · primary xhigh succeeded 8 min 26 s

First review of this pull request.
4 reviewers finished.
Commit 231cffa.
Check run.
Run: GET /unfret/run/panel:231cffaf67579dbf34097bdcfb5bd9ed6cc7050e:OwBrA8Hbbqq:MrA-wFXz244.

@unfret review · @unfret status · @unfret help

@unfret unfret Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No blocking findings
Check

@mergify

mergify Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

✅ Merged into main as bc94728 at 00:05 UTC

Less than a minute in the queue, with no time running CI.

So far

  • ✅ Entered the queue Sat 21 Nov, 00:05 · rule default · triggered by merge protections
  • ✅ Checks skipped · PR is already up-to-date
  • ✅ Merged 00:05 · bc94728
10 merge conditions met
  • github-review-approved · required by GitHub repository ruleset rule Mergify merge protections
  • -draft · required by Merge Protections rule Required checks
  • -label = hold · required by Merge Protections rule Required checks
  • check-success = Analyze (actions) · required by Merge Protections rule Required checks
  • check-success = Analyze (go) · required by Merge Protections rule Required checks
  • check-success = Analyze (javascript-typescript) · required by Merge Protections rule Required checks
  • check-success = e2e-hermetic · required by Merge Protections rule Required checks
  • check-success = go · required by Merge Protections rule Required checks
  • check-success = @unfret/Unfret · required by Merge Protections rule Unfret on risky paths
  • any of · required by GitHub repository ruleset rule Mergify merge protections:
    • check-success = @mergify/Mergify Merge Protections
    • check-neutral = @mergify/Mergify Merge Protections
    • check-skipped = @mergify/Mergify Merge Protections

Times in UTC · dashboard · how the merge queue works

@mergify
mergify Bot merged commit bc94728 into main Nov 21, 2026
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant