Skip to content

Restore Claude routing after an early quota reset - #31

Merged
mergify[bot] merged 2 commits into
mainfrom
HOUSE-129-lane
Nov 16, 2026
Merged

mergify[bot] merged 2 commits into
mainfrom
HOUSE-129-lane

Conversation

@ericlitman

@ericlitman ericlitman commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A Claude subscription reset left the account blocked until its previous reset time. Fresh usage reads now release the stale account-wide quota block and update routing and saved cooldown state.

five-minute usage read
  validate both shared windows (explicit idle null is allowed)
  reject stale reads after a new failure or re-registration
  release account quota, preserve independent model failures
  update registry, scheduler and cooldown store

Linear: HOUSE-129

Evidence

  • Before: the new service regressions fail on main 7b3b9fa with a 96-hour cooldown and a restored stale cooldown after restart.
    After: both regressions pass on this branch through scheduler and reserve-enabled selection, with sibling errors and the exhausted second account preserved.
  • Actual server proof: on Studio2, identical seeded stale cooldowns returned 429 on main (port 18317). The fixed server (port 18318) released on the next five-minute poll and returned 200 EARLY_RESET_OK. The saved account-wide block was removed. Recorded live comparison. Startup loading changed the account during the first read, so that stale read correctly refused release.

Merge Danger

Door: two-way

Revert the change or restore the previous binary. No credential or configuration migration.

Blast Radius: routing

Only confirmed Claude account-wide quota cooldowns are released. Existing observation-only behavior, model-specific cooldowns, Codex behavior and reserve rules remain. The exported QuotaUsageReader return type changes to include an explicit availability result; all repository implementations and callers are migrated. External SDK implementations must migrate.

Acceptance criteria

  • “Identify the reset Claude account using current upstream usage and runtime cooldown evidence. Clear its stale quota block and verify a real Claude request through the running proxy.” Done: eric@litman.org had 0% weekly usage with a November 19 cooldown. Its v8 reset returned 200, then real claude-opus-5-5 inference returned 200 PROXY_OK.
  • “Automatically release stale Claude quota cooldowns when authoritative fresh usage proves quota is available before the previously recorded reset.” Covered by actual server comparison and service regressions.
  • “Preserve disabled accounts, authentication failures, unrelated cooldowns, legitimate exhausted windows, partial or invalid usage evidence, configured reserves, and other accounts.” Negative parser/service tables and manager refusal/layer tests pass. Existing reserve tests pass unchanged. Per-model-only cooldowns stay because shared usage cannot prove their recovery.
  • “Cover early restoration, non-restoration, scheduler/registry state and persisted cooldown behavior with focused regression tests. Required compilation and applicable tests pass.” Service tests exercise both selectors, registry suspension, saved state and restart. New-read failure and re-registration guards are covered.
  • “Merge the durable fix through the repository workflow, deploy it to Studio2, and verify the running proxy.” Done: Mergify merged c7430f1. Released v8.0.44-mob.6 and deployed once to Studio2. Production /v1/messages returned HTTP 200 DEPLOY_OK.

Gate results

Parent-run on head 1ad939b: go test ./... exit 0; go build -o <scratch>/test-output ./cmd/server exit 0 (binary removed); git diff --check exit 0; gofmt check of changed Go files exit 0. Focused race tests also passed in the implementation lane. Validation record.

Not verified

Production deployment and inference passed; no synthetic stale block was added to production. Automatic recovery was proved on an isolated real proxy. An intermediate full run failed TestServiceHeartbeatFailoverDoesNotDrainBlockingScope and TestServiceShutdown_ViolentShutdownImmediatelyClosesServer; final rerun passed. The heartbeat failure reproduces on unchanged main in a 15-run comparison. Shutdown passed all 15 main runs and is not traced to this diff. Neither lifecycle test or implementation is changed. No claim is made that the shutdown failure is resolved.

Design observations

The existing Devin restoration guard and observation timestamps have unrelated limitations recorded in the design ledger. This change preserves Devin's decision rule. No additional work is proposed here.

Review disposition on head 1

Operator waiver for head 1ad939b, under the installed babysit-pr consequence and reachability rules.

  • v1:4cee3f676c03fb3bf9288d626da37ec4752f6012b804e8848d15c2161ad51c17 — pre-accepted. The confirmed sequence uses a direct SDK MarkResult with Model empty, following an active credential quota. That can erase a separate auth-level 404. Our Claude server execution, preparation and streaming paths attach stateModel/resultModel and RouteModel (sdk/cliproxy/auth/conductor_execution.go:550,637,1126). Independent model blocks are preserved in shadows. No normal deployed producer of the alleged model-less 404 was found. This accepts the SDK limitation; it does not claim it is fixed or unsupported.
  • v1:769c6899d82fd86a2c41d6c6391b594bb886dcb63866da53feb81ee47cb492a0 — unconfirmed after bounded investigation. The parser checks shared five_hour/seven_day, not seven_day_opus. The judge states that the alleged exhausted-Opus-window to CredentialScope causal path remains unproven. The classifier makes shared unified rejections account-wide and ordinary model 429s model-scoped (internal/runtime/executor/claude_executor_request.go:809-824; helps/claude_ratelimit.go:27-47); independent model blocks survive release. The live reset account response has its model-specific windows null. A future upstream combination could expose the alleged behavior; this waiver accepts the uncertainty, not a claim of disproof.
  • v1:b97f97068b18f7d317852845afb47da8e978eabf92d78a0b4cbc4a1be6a3557d — pre-accepted. Complete-Auth custom SDK Stores can reload the old block because release saves CooldownStateStore only. Our deployed FileTokenStore.Save writes Metadata only (sdk/auth/filestore.go:130-132), so it cannot reload runtime cooldown fields. All bundled alternate stores likewise marshal Metadata (internal/store/postgresstore.go:264, objectstore.go:213, gitstore.go:484). The actual file cooldown restart regression and real .cds proof pass. This accepts the custom-store limitation; it does not claim all SDK Stores are unaffected.

None of these allegations concerns security, secret exposure, data loss or gate/waiver integrity. No repair dispatch is warranted by actual deployed reachability. A three-round repair estimate is about 120 minutes and $39 at the skill's planning rate; it establishes no new waiver authority. No code changed after the verdict, so this remains reviewed head 1. One allegation about losing the reporting model's own cooldown was dismissed by the panel on exact-head code evidence.

Deployment evidence

Deployment, production request, and release. Merge tree equals the tested PR head. The first post-merge Go CI run hit TestResponsesInterruptStopsHTTPUpstream; its fake Codex executor has no usage poller and does not call the new release path. Failed job rerun is pending.

Eric A. Litman and others added 2 commits November 15, 2026 18:20
A Claude usage read that proves both shared windows available (each an
explicit null or a finite utilization below 100 with a reset after the read
started, at least one active) now ends a stale credential-wide quota
cooldown. The release refuses when the credential changed since the read
started (Generation or RegistrationEpoch), and keeps disabled accounts,
auth and grant failures, and each model's own cooldowns. It publishes to the
registry and scheduler and saves the cooldown state.

QuotaUsageReader now returns QuotaUsage{Headers, CredentialQuotaAvailable}.
ObserveQuotaHeaders stays observational. The release body is shared with the
Devin quota restoration, and the publish tail with ResetQuota.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@unfret

unfret Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

✅ Accepted risk

Accepted by ericlitman: HOUSE-129 (authority 48fa77b)

🟡 Medium · Quota release erases independent auth-level cooldowns

🟡 Medium · Exhausted model-specific windows still authorize release

🟡 Medium · Quota release is not saved to the backing auth Store

Details

Run: GET /unfret/run/panel:1ad939b606e2eafee39857c44500ffc6a40bfea7:s0DeZ9mJy9N:MqyWBVLIbIg.

@unfret review · @unfret status · @unfret help

@unfret unfret Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛑 3 blocking findings
🟡 Medium · Quota release erases independent auth-level cooldowns
🟡 Medium · Exhausted model-specific windows still authorize release
🟡 Medium · Quota release is not saved to the backing auth Store
Check

Comment thread sdk/cliproxy/auth/conductor_cooldown.go Resolved
Comment thread sdk/cliproxy/auth/conductor_cooldown.go Resolved
@ericlitman

Copy link
Copy Markdown
Contributor Author

Operator waiver for head 1ad939b, under the installed babysit-pr consequence and reachability rules.

  • v1:4cee3f676c03fb3bf9288d626da37ec4752f6012b804e8848d15c2161ad51c17 — pre-accepted. The confirmed sequence uses a direct SDK MarkResult with Model empty, following an active credential quota. That can erase a separate auth-level 404. Our Claude server execution, preparation and streaming paths attach stateModel/resultModel and RouteModel (sdk/cliproxy/auth/conductor_execution.go:550,637,1126). Independent model blocks are preserved in shadows. No normal deployed producer of the alleged model-less 404 was found. This accepts the SDK limitation; it does not claim it is fixed or unsupported.
  • v1:769c6899d82fd86a2c41d6c6391b594bb886dcb63866da53feb81ee47cb492a0 — unconfirmed after bounded investigation. The parser checks shared five_hour/seven_day, not seven_day_opus. The judge states that the alleged exhausted-Opus-window to CredentialScope causal path remains unproven. The classifier makes shared unified rejections account-wide and ordinary model 429s model-scoped (internal/runtime/executor/claude_executor_request.go:809-824; helps/claude_ratelimit.go:27-47); independent model blocks survive release. The live reset account response has its model-specific windows null. A future upstream combination could expose the alleged behavior; this waiver accepts the uncertainty, not a claim of disproof.
  • v1:b97f97068b18f7d317852845afb47da8e978eabf92d78a0b4cbc4a1be6a3557d — pre-accepted. Complete-Auth custom SDK Stores can reload the old block because release saves CooldownStateStore only. Our deployed FileTokenStore.Save writes Metadata only (sdk/auth/filestore.go:130-132), so it cannot reload runtime cooldown fields. All bundled alternate stores likewise marshal Metadata (internal/store/postgresstore.go:264, objectstore.go:213, gitstore.go:484). The actual file cooldown restart regression and real .cds proof pass. This accepts the custom-store limitation; it does not claim all SDK Stores are unaffected.

None of these allegations concerns security, secret exposure, data loss or gate/waiver integrity. No repair dispatch is warranted by actual deployed reachability. A three-round repair estimate is about 120 minutes and $39 at the skill's planning rate; it establishes no new waiver authority. No code changed after the verdict, so this remains reviewed head 1. One allegation about losing the reporting model's own cooldown was dismissed by the panel on exact-head code evidence.

@mergify
mergify Bot merged commit c7430f1 into main Nov 16, 2026
9 checks passed
@mergify

mergify Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

✅ Merged into main as c7430f1 at 00:00 UTC

Less than a minute in the queue, with no time running CI.

So far

  • ✅ Entered the queue Mon 16 Nov, 00:00 · rule default · triggered by merge protections
  • ✅ Checks skipped · PR is already up-to-date
  • ✅ Merged 00:00 · c7430f1
10 merge conditions met
  • github-review-approved · required by GitHub repository ruleset rule Mergify merge protections
  • -draft · required by Merge Protections rule Required checks
  • -label = hold · required by Merge Protections rule Required checks
  • check-success = Analyze (actions) · required by Merge Protections rule Required checks
  • check-success = Analyze (go) · required by Merge Protections rule Required checks
  • check-success = Analyze (javascript-typescript) · required by Merge Protections rule Required checks
  • check-success = e2e-hermetic · required by Merge Protections rule Required checks
  • check-success = go · required by Merge Protections rule Required checks
  • check-success = @unfret/Unfret · required by Merge Protections rule Unfret on risky paths
  • any of · required by GitHub repository ruleset rule Mergify merge protections:
    • check-success = @mergify/Mergify Merge Protections
    • check-neutral = @mergify/Mergify Merge Protections
    • check-skipped = @mergify/Mergify Merge Protections

Times in UTC · dashboard · how the merge queue works

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant