Repository navigation
fix(kv): recover fixed-key watches after Redis reconnect exhaustion - #2644
Conversation
|
[PHASE: PLAN] Actual baseline regression now reproduced on unchanged main product source: three real Redis probes fail. Stopping the server exhausts native subscriber retries; restarting it leaves the existing iterator silent despite PING readiness. Leaving it down yields no typed failure within35seconds. A native MULTI transaction kills exactly one pub/sub client and writes state before any resubscription can occur; a short reconnect also never reports that gap value. No fake adapter, emitted message or timing race hides a missing reread. See committed tests. Locked feature plan: keep the existing shared lease owner; explicitly restore live channels and wait for SUBSCRIBE acknowledgement before current-state snapshots. Recreate ended subscribers with existing retry policy under a30-second cancellable recovery budget, fence stale generations, preserve independent releases and promptly cancel when no leases remain. Queue failure wakes iteration with existing KvConnectionError/KvClosedError, exposed as documented public errors. Snapshots may repeat state and include missing keys as delete/null; no mutation-history promise. No Redis Streams, saga transport or scaffold/Aspire output changes. Static red-test lint passes. All test-owned containers are removed by immutable identity and unique label. Plan-Gate is selected before any production implementation. Finalized plan and file groups are in |
|
[PHASE: PLAN-EVAL] Separate-session PLAN-EVAL for issue #2639, evaluated head Route: requested feature PLAN-EVAL primary GLM 5.3 provider default → observed GLM 5.3 via OpenCode Go, provider default, options {}; matches the live matrix (fallback Fable 5.1 low not used). Generator Codex gpt-6.1-sol/xhigh (OpenAI family) — different vendor family, separate session. No substitution. Plan-Gate checklist — 8/8 PASS:
Acceptance coverage — 3/3 COVERED (all red on main, verified against the structured raw reports):
Feasibility checks: @std retry/delay AbortSignal support verified; primary-client reconnect timing handled (getMany retried under the recovery signal); ordering semantics precise (buffer through recovery, suppress only racing snapshot entries, never compare opaque versionstamps); sibling leases keep independent queues/resync/releases; terminal end cannot hang (queue failure wakes Debt: open Non-blocking IMPL-EVAL watch-points (recorded in Implementation may begin (S2). Feature-tier loop policy: cycle 1 of max 2, verdict PASS. |
|
[PHASE: IMPL] Contract slice f42ac48 follows the independent Plan-Gate PASS. Existing KvError, KvConnectionError and KvClosedError are now documented named exports from root and Redis entrypoints; the real outage regression imports its expected error through the public Redis entrypoint. No recovery implementation yet, so all three real baseline regressions remain the acceptance oracle. Structured package check59files passes; package doc lint zero; changed error-class lint/format pass. No new dependency or lock change. Next implement the approved bounded subscriber/lease recovery, queue failure and state rereads, then run all real lifecycle and sibling-lease tests. |
|
[PHASE: IMPL] Commitbfaa29cd6 implements the independently approved recovery contract. Existing live leases survive native reconnect exhaustion: the shared owner reconnects, explicitly restores subscriptions and rereads current state after acknowledgement. A30-second disconnect budget bounds all retry/restoration/reread work and preserves the last cause. Terminal failure/close wakes idle or debounced iteration with public KvConnectionError/KvClosedError; caller abort completes cleanly. Releasing one lease preserves siblings, and final release cancels pending recovery. A new watch works after terminal failure. No idle keyspace polling added to fixed-key watches. Genuine main regressions: actual server restart after observed subscriber end hangs; an unrecoverable outage never yields a typed error; atomic native subscriber kill plus state write is missed after short reconnect. All three now pass. Additional live tests cover sibling cancellation after end, final abort during a paused reread followed by a fresh watch/typed close, and prefix subscription restoration/typed close. Queue tests cover idle/debounce failure and abort precedence. Final full KV suite with isolated Redis and actual-server cases enabled:250passed/0failed/2existingignored. Regression test. Structured package check61files, scoped lint/fmt, quality:gate, release:preflight, docs:accuracy, doc lint zero and JSR dry-run/audit pass (one slow-type inspection warning). Fixed-key bookkeeping moved into a focused module so the main adapter shrinks747->704lines. README/guide document current-state/coalescing semantics and typed errors. All uniquely owned containers removed; foreign resources untouched. No locks/dependencies, Redis Streams or saga transport changes. Canonical consumer generation/freshness and final independent IMPL-EVAL next; full scaffold.runtime N/A because no scaffold/Aspire/executable wiring changes. |
|
[PHASE: IMPL] Final recovery sourceea85fb387:251tests pass with2existing opt-in/CI skips; all7new real-server cases execute. The additional red->green case returns a watch while its reread is paused and proves shared recovery/new sibling intake proceeds without waiting for the released observer. Each observer now owns cancellation for its in-flight reread. No new public option or signature. Canonical prose/corpus/barrel/publish consumers are current, and corpus/publish freshness rechecks pass after this private correction. Structured package check61files, lint/fmt, quality:gate, release:preflight, docs:accuracy, doc lint zero and JSR dry-run/audit pass. All exact-owned Redis containers removed; no foreign resource changes. Public semantics/docs match the independently approved plan. No Redis Streams or saga transport edits. Acceptance proof remains actual stopped-server/end/restart recovery, atomic native subscriber kill+storage write gap recovery, and terminal KvConnectionError within the30-second budget. Root and Redis entrypoints export the existing typed error identities. Independent feature IMPL-EVAL next; PR remains draft until PASS and final required-head/thread gates. Full scaffold.runtime N/A: no scaffold/Aspire/executable wiring changes. |
|
[PHASE: IMPL-EVAL] [VERDICT: PASS] Independent final IMPL-EVAL for issue #2639: scope complete, all required gates green. Route: requested feature IMPL-EVAL primary Muse Spark 1.3/xhigh via OpenCode Go → observed same; fallback Opus 5/xhigh unused. Generator Codex gpt-6.1-sol/xhigh (OpenAI family): separate session and separate vendor family, no substitution. No subagents used. Heads: evaluated Harness: PASS. PLAN-EVAL PASS before implementation confirmed; design checkpoint present; slices match plan with per-slice gates. Static/fitness: scoped check/lint/fmt 61 files PASS; doc lint 0 PASS; Runtime (independently re-run with owned fixtures): focused Consumer: Acceptance (issue #2639, 3/3): box 1 real stop/end/restart recovery PASS; box 2 gap-value reread after atomic subscriber kill plus restart deletion PASS; box 3 unrecoverable typed Findings: one low non-blocking note (unrelated site-tooling check, see Next: coordinator owns final required-head/review-thread checks and the intentionally open DoD box; evaluator makes no label/readiness/merge changes. Full verdict in PR #2644, head |
|
[PHASE: REVIEW] Ready for review atfcd970c8bc7a63f43843d29bb2809a6a4d2b9d3a: independent feature IMPL-EVAL PASS, full KV251/0/2 and all10focused live/queue cases independently reproduced, required head qualitySUCCESS, check-test/deps-report policy-skipped, review-thread gate0/0. Local close-gate PASS with all3issue boxes and complete PR DoD. Final commit retains verdict/run evidence only; source unchanged fromea85fb387. The remote close-gate result predates the final DoD update. Its rerun is queued as follow-up work once the still-running workflow can accept a rerun; both workflow/job rerun requests are currently refused while other shards run. No gate/status was waived or tampered with. No merge, ready-merge, auto-merge or publication. |
|
[PHASE: IMPL] Evaluated head: |
Summary
Restore existing Redis watches after subscriber reconnect exhaustion and reread current values after every acknowledged reconnect. Unrecoverable outages and adapter close now reach callers through public typed KV errors.
Scope
Closes #2639
Slices
Validation
Harness
.llm/runs/fix-redis-watch-recovery--issue-2639/.Drift / Debt
Native retry requires a finite attempt count; chosen minimum cumulative delay exceeds the30-second deadline. Existing native connection retry jitter retained. Disable subscriber automatic command replay alongside automatic resubscription to fence stale leases. Split fixed-key bookkeeping into a focused127-line module, shrinking the main adapter747->704lines; S2<=18files, within approved same-concern split allowance. Existing debt is reduced. No Redis Streams or saga transport edits.
Definition of Done
🤖 Generated with Codex