Skip to content

fix(kv): recover fixed-key watches after Redis reconnect exhaustion - #2644

Merged
rickylabs merged 10 commits into
mainfrom
fix/redis-watch-recovery
Nov 17, 2026
Merged

rickylabs merged 10 commits into
mainfrom
fix/redis-watch-recovery

Conversation

@rickylabs

@rickylabs rickylabs commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

Restore existing Redis watches after subscriber reconnect exhaustion and reread current values after every acknowledged reconnect. Unrecoverable outages and adapter close now reach callers through public typed KV errors.

Scope

  • Archetype3 / packages/kv Redis lifecycle.

Closes #2639

  • Decisions: one shared subscription owner, explicit SUBSCRIBE acknowledgement before snapshots,30-second total recovery budget from disconnect, at-least-once current-state observation including missing keys. Existing errors become public catchable exports. No new settings/dependencies.

Slices

  • S1 Three real red-on-main outage/gap tests and independent PLAN-EVAL PASS.
  • S2 Public error contract first, bounded recovery, green regressions and docs.
  • S3 Canonical consumer assets and freshness.
  • S3 Independent implementation evaluation and current head gates.

Validation

  • Main: actual server stop until subscriber end, restart and PING readiness leave the iterator silent; leaving Redis down yields no typed error; atomic subscriber kill plus SET is missed after short reconnect. All three now pass.
  • Final full KV suite with isolated Redis and actual-server cases enabled:251passed/0failed/2existingignored. Includes sibling release, abort during reread, fresh watch after failure, typed close, prefix restoration and idle/debounce error propagation.
  • Structured check61files, lint/fmt, quality:gate, release:preflight, docs:accuracy, doc lint zero, JSR dry-run/audit pass (one slow-type inspection warning). No lock churn. Exact owned container identities/unique labels fence cleanup, final scan empty.
  • Canonical consumer generation/freshness complete; Independent feature IMPL-EVAL PASS; required head quality SUCCESS and check-test/deps-report policy-skipped; review-thread gate0/0. Verdict-retention head changes only run artifacts. Returned leases also cancel their own in-flight rereads without holding shared recovery. Full scaffold.runtime N/A: no scaffold/Aspire/runtime-launch output change; direct actual-server lifecycle tests prove acceptance.
issue: 2639
entries:
  - box-index: 1
    evidence: "https://github.com/rickylabs/netscript/pull/2644#issuecomment-6928867800 — actual server stop until subscriber end, restart/PING and existing-iterator delivery; red on main, green at final source."
  - box-index: 2
    evidence: "https://github.com/rickylabs/netscript/pull/2644#issuecomment-6928867800 — native atomic subscriber kill plus SET before resubscribe, and restart deletion reread."
  - box-index: 3
    evidence: "https://github.com/rickylabs/netscript/pull/2644#issuecomment-6928867800 — unrecoverable actual-server outage raises public KvConnectionError within35seconds, with prompt abort/close and fresh watch after failure."

Harness

  • Run dir: .llm/runs/fix-redis-watch-recovery--issue-2639/.
  • Phase: impl-eval. Feature tier; independent PLAN-EVAL PASS before implementation. Final independent IMPL-EVAL mandatory.

Drift / Debt

Native retry requires a finite attempt count; chosen minimum cumulative delay exceeds the30-second deadline. Existing native connection retry jitter retained. Disable subscriber automatic command replay alongside automatic resubscription to fence stale leases. Split fixed-key bookkeeping into a focused127-line module, shrinking the main adapter747->704lines; S2<=18files, within approved same-concern split allowance. Existing debt is reduced. No Redis Streams or saga transport edits.

Definition of Done

  • Existing watches recover after subscriber end and real server restart.
  • Reconnect rereads keys changed during the gap.
  • Terminal outage rejects with catchable typed KV error, cleanup does not hang.
  • Consumer assets, independent evaluation, head checks and review threads complete.
  • Branch taxonomy and milestone0.0.10.

🤖 Generated with Codex

@rickylabs rickylabs added this to the 0.0.10 milestone Nov 16, 2026
@rickylabs rickylabs added type:fix ci:skip-e2e status:research Harness research phase priority:p2 Medium area:kv packages/kv ci:skip-scaffold Skip the scaffold-static (deno-only) scaffold gate labels Nov 16, 2026
@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: PLAN]

Actual baseline regression now reproduced on unchanged main product source: three real Redis probes fail. Stopping the server exhausts native subscriber retries; restarting it leaves the existing iterator silent despite PING readiness. Leaving it down yields no typed failure within35seconds. A native MULTI transaction kills exactly one pub/sub client and writes state before any resubscription can occur; a short reconnect also never reports that gap value. No fake adapter, emitted message or timing race hides a missing reread. See committed tests.

Locked feature plan: keep the existing shared lease owner; explicitly restore live channels and wait for SUBSCRIBE acknowledgement before current-state snapshots. Recreate ended subscribers with existing retry policy under a30-second cancellable recovery budget, fence stale generations, preserve independent releases and promptly cancel when no leases remain. Queue failure wakes iteration with existing KvConnectionError/KvClosedError, exposed as documented public errors. Snapshots may repeat state and include missing keys as delete/null; no mutation-history promise. No Redis Streams, saga transport or scaffold/Aspire output changes.

Static red-test lint passes. All test-owned containers are removed by immutable identity and unique label. Plan-Gate is selected before any production implementation. Finalized plan and file groups are in .llm/runs/fix-redis-watch-recovery--issue-2639/plan.md; separate feature PLAN-EVAL next.

@rickylabs rickylabs added status:plan-eval Awaiting plan evaluation and removed status:research Harness research phase labels Nov 16, 2026
@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: PLAN-EVAL]
[VERDICT: APPROVED]

Separate-session PLAN-EVAL for issue #2639, evaluated head 94a7bb380c92f44d028fb4d21788601d47131cd5 (branch fix/redis-watch-recovery, base main b0eb07957). Full verdict written to .llm/runs/fix-redis-watch-recovery--issue-2639/plan-eval.md (left uncommitted for the generator).

Route: requested feature PLAN-EVAL primary GLM 5.3 provider default → observed GLM 5.3 via OpenCode Go, provider default, options {}; matches the live matrix (fallback Fable 5.1 low not used). Generator Codex gpt-6.1-sol/xhigh (OpenAI family) — different vendor family, separate session. No substitution.

Plan-Gate checklist — 8/8 PASS:

  • PASS Research present and current — research re-baselined on fresh main; every load-bearing finding verified against the tree: ensureSubscriber() keeps a ended subscriber while ensureClient() resets it (packages/kv/adapters/redis/connection.ts:94-97 vs :127-152); retryStrategy returns null after 5 attempts (connection.ts:298-301); subscriber errors only logged (connection.ts:269-271); the watch loop and WatchBatchQueue have no failure path (packages/kv/adapters/redis.adapter.ts:530-535, packages/kv/adapters/redis/watch-batch-queue.ts).
  • PASS Decisions locked — single lease owner retained; autoResubscribe disabled only for the dedicated subscriber; explicit SUBSCRIBE acknowledgement before rereads (grounded: pinned ioredis 5.11.1 readyHandler sets status ready before resubscribing, verified in the installed source); 30-second cancellable recovery bound; generation fences; queue terminal failure; existing typed errors exported with JSDoc; no new settings/dependencies.
  • PASS Open-decision sweep — plan lists none; evaluator sweep found no rework-forcing decision (recovery triggers per case, second-outage invalidation, acquire-during-recovery, release-vs-resubscribe races, abort/close precedence, prefix leases riding the shared lifecycle all resolved by locked invariants).
  • PASS Commit slices — S1 7 files (landed, red suite committed); S2 ≤ 17 files enumerated; S3 ≤ 14 files; each names what it proves, its proving gate, and its files; all < 30.
  • PASS Risk register — seven risks with mitigations, including real-server baseline against fake-only proof.
  • PASS Gate set selected — ARCHETYPE-3 folding Integration-2: scoped static wrappers, quality:gate, real Redis outage + full KV suite (runtime gates required), consumer gates (exports change). Full scaffold.runtime N/A justified — no scaffold/Aspire/DB-wiring/publish-shape change; PR labels consistent.
  • PASS Deferred scope explicit — prefix discovery, Redis Streams, saga transports, new abstractions; no packages/plugin-sagas-core/src/transports/ file touched in the diff.
  • PASS jsr-audit surface scan — planned surface is three existing explicitly typed error classes re-exported from packages/kv/mod.ts + packages/kv/redis.ts; no slow types possible; JSDoc/doc-lint and publish dry-run assigned to S2/S3. Note: application/errors.ts module doc still says "Internal error helpers" — reword when it becomes public contract.

Acceptance coverage — 3/3 COVERED (all red on main, verified against the structured raw reports):

  1. Real stop/restart after asserted subscriber end must restore the existing iterator (deliver ['delete', null], then a live write) — currently times out.
  2. Gap value written atomically before any resubscription (CLIENT KILL TYPE pubsub + SET) must be emitted after reconnect — currently times out.
  3. Never-restarted outage must reject with KvConnectionError within 35 s — currently the deadline itself rejects (iterator hangs); 30 s recovery bound + native window fits the test bound.

Feasibility checks: @std retry/delay AbortSignal support verified; primary-client reconnect timing handled (getMany retried under the recovery signal); ordering semantics precise (buffer through recovery, suppress only racing snapshot entries, never compare opaque versionstamps); sibling leases keep independent queues/resync/releases; terminal end cannot hang (queue failure wakes next(), close rejects with KvClosedError before closing sockets, ended sockets disconnect rather than await QUIT).

Debt: open packages/kv AP-1 test-file debt not deepened (no bridge_test.ts growth); no new debt entry required.

Non-blocking IMPL-EVAL watch-points (recorded in plan-eval.md): release during failed/recovering subscription must not mask the typed error in the generator's finally; late generations must not re-subscribe released channels; snapshot bookkeeping consistency; prefix-watch behavior under the shared lifecycle; F-1 file-size as recovery logic lands in existing owner files.

Implementation may begin (S2). Feature-tier loop policy: cycle 1 of max 2, verdict PASS.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

Contract slice f42ac48 follows the independent Plan-Gate PASS. Existing KvError, KvConnectionError and KvClosedError are now documented named exports from root and Redis entrypoints; the real outage regression imports its expected error through the public Redis entrypoint. No recovery implementation yet, so all three real baseline regressions remain the acceptance oracle.

Structured package check59files passes; package doc lint zero; changed error-class lint/format pass. No new dependency or lock change. Next implement the approved bounded subscriber/lease recovery, queue failure and state rereads, then run all real lifecycle and sibling-lease tests.

@rickylabs rickylabs added status:impl and removed status:plan-eval Awaiting plan evaluation labels Nov 16, 2026
@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

Commitbfaa29cd6 implements the independently approved recovery contract. Existing live leases survive native reconnect exhaustion: the shared owner reconnects, explicitly restores subscriptions and rereads current state after acknowledgement. A30-second disconnect budget bounds all retry/restoration/reread work and preserves the last cause. Terminal failure/close wakes idle or debounced iteration with public KvConnectionError/KvClosedError; caller abort completes cleanly. Releasing one lease preserves siblings, and final release cancels pending recovery. A new watch works after terminal failure. No idle keyspace polling added to fixed-key watches.

Genuine main regressions: actual server restart after observed subscriber end hangs; an unrecoverable outage never yields a typed error; atomic native subscriber kill plus state write is missed after short reconnect. All three now pass. Additional live tests cover sibling cancellation after end, final abort during a paused reread followed by a fresh watch/typed close, and prefix subscription restoration/typed close. Queue tests cover idle/debounce failure and abort precedence. Final full KV suite with isolated Redis and actual-server cases enabled:250passed/0failed/2existingignored. Regression test.

Structured package check61files, scoped lint/fmt, quality:gate, release:preflight, docs:accuracy, doc lint zero and JSR dry-run/audit pass (one slow-type inspection warning). Fixed-key bookkeeping moved into a focused module so the main adapter shrinks747->704lines. README/guide document current-state/coalescing semantics and typed errors. All uniquely owned containers removed; foreign resources untouched. No locks/dependencies, Redis Streams or saga transport changes. Canonical consumer generation/freshness and final independent IMPL-EVAL next; full scaffold.runtime N/A because no scaffold/Aspire/executable wiring changes.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

Final recovery sourceea85fb387:251tests pass with2existing opt-in/CI skips; all7new real-server cases execute. The additional red->green case returns a watch while its reread is paused and proves shared recovery/new sibling intake proceeds without waiting for the released observer. Each observer now owns cancellation for its in-flight reread. No new public option or signature.

Canonical prose/corpus/barrel/publish consumers are current, and corpus/publish freshness rechecks pass after this private correction. Structured package check61files, lint/fmt, quality:gate, release:preflight, docs:accuracy, doc lint zero and JSR dry-run/audit pass. All exact-owned Redis containers removed; no foreign resource changes. Public semantics/docs match the independently approved plan. No Redis Streams or saga transport edits.

Acceptance proof remains actual stopped-server/end/restart recovery, atomic native subscriber kill+storage write gap recovery, and terminal KvConnectionError within the30-second budget. Root and Redis entrypoints export the existing typed error identities. Independent feature IMPL-EVAL next; PR remains draft until PASS and final required-head/thread gates. Full scaffold.runtime N/A: no scaffold/Aspire/executable wiring changes.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL-EVAL] [VERDICT: PASS]

Independent final IMPL-EVAL for issue #2639: scope complete, all required gates green.

Route: requested feature IMPL-EVAL primary Muse Spark 1.3/xhigh via OpenCode Go → observed same; fallback Opus 5/xhigh unused. Generator Codex gpt-6.1-sol/xhigh (OpenAI family): separate session and separate vendor family, no substitution. No subagents used.

Heads: evaluated 93bf1a6bce1b5b34fea6cf32a90cef795da42a3f (branch fix/redis-watch-recovery, base origin/main b0eb079575968e843b781579673c9fe2ae250e07, clean tree). Source ea85fb387d0e379b97c598aa4a9a14dea604a6dd is byte-identical across packages//plugins//docs/; delta to evaluated head is run-evidence only.

Harness: PASS. PLAN-EVAL PASS before implementation confirmed; design checkpoint present; slices match plan with per-slice gates.

Static/fitness: scoped check/lint/fmt 61 files PASS; doc lint 0 PASS; release:preflight PASS; docs:accuracy PASS; JSR dry-run PASS + audit PASS (one pre-existing slow-type warning); quality:gate exit 0 with packages/kv FAIL=0. F-1 improves: packages/kv/adapters/redis.adapter.ts 747→704 lines, new focused packages/kv/adapters/redis/watch-fixed-keys.ts 127 lines within approved split. No new public abstraction/settings/dependencies; no packages/plugin-sagas-core/src/transports/ edits; no lock churn.

Runtime (independently re-run with owned fixtures): focused packages/kv/tests/redis-watch-recovery_test.ts + packages/kv/tests/redis-watch-failure_test.ts → 10 passed/0 failed (7 live server cases + 3 queue cases, none skipped). Full packages/kv/tests/ with owned external Redis → 251 passed/0 failed/2 ignored (ignores are the pre-existing CI-only and old supplied-container tests; no acceptance case skipped). All six plan-eval watch-points plus the per-lease cancellation regression verified in code and live races: typed failure survives release, late generations fenced, snapshot suppression without versionstamp ordering, prefix lifecycle with typed failure/close, queue idle/debounce/abort precedence, returned leases cancel only their own reread. Owned fixtures removed by exact identity with final scan empty; foreign resources untouched. scaffold.runtime N/A per approved plan.

Consumer: check:mcp-export-corpus PASS; check:publish-assets PASS. (Broader check:agent-docs-prose chain fails in unrelated fresh-ui site tooling; the three carriers this run changes are fresh per the checks above.)

Acceptance (issue #2639, 3/3): box 1 real stop/end/restart recovery PASS; box 2 gap-value reread after atomic subscriber kill plus restart deletion PASS; box 3 unrecoverable typed KvConnectionError without hang plus abort/close/fresh-watch PASS. PR body Closes #2639 + one fenced acceptance-evidence block with 3 box-index entries verified against the final IMPL proof comment.

Findings: one low non-blocking note (unrelated site-tooling check, see evaluate.md); no high/medium findings; no new debt.

Next: coordinator owns final required-head/review-thread checks and the intentionally open DoD box; evaluator makes no label/readiness/merge changes. Full verdict in .llm/runs/fix-redis-watch-recovery--issue-2639/evaluate.md (uncommitted).

PR #2644, head 93bf1a6bce1b5b34fea6cf32a90cef795da42a3f, checks and reviews per current PR state (draft, status:impl-eval, 0 review threads).

@rickylabs
rickylabs marked this pull request as ready for review November 16, 2026 21:04
@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: REVIEW]

Ready for review atfcd970c8bc7a63f43843d29bb2809a6a4d2b9d3a: independent feature IMPL-EVAL PASS, full KV251/0/2 and all10focused live/queue cases independently reproduced, required head qualitySUCCESS, check-test/deps-report policy-skipped, review-thread gate0/0. Local close-gate PASS with all3issue boxes and complete PR DoD. Final commit retains verdict/run evidence only; source unchanged fromea85fb387.

The remote close-gate result predates the final DoD update. Its rerun is queued as follow-up work once the still-running workflow can accept a rerun; both workflow/job rerun requests are currently refused while other shards run. No gate/status was waived or tampered with. No merge, ready-merge, auto-merge or publication.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

Evaluated head: 93bf1a6bce1b5b34fea6cf32a90cef795da42a3f (approved tip fcd970c8bc7a63f43843d29bb2809a6a4d2b9d3a). New head: 791de9cb0d81da9cb0ebbbf7c9ec74dbe7411ef9, merging main ecf982ddc2d81555ee3aecf1ecb64777dd06314b. The delta is merge plus canonical regeneration only. The KV guide auto-merged cleanly: both main's persistence text and this PR's watch-recovery additions/removals are unchanged; the rest of the handwritten diff is byte-identical. No handwritten conflict resolution was needed. Generated-file freshness, docs source format/README compilation, release:preflight and docs:accuracy pass; labels and acceptance evidence are unchanged.

@rickylabs
rickylabs merged commit f59c185 into main Nov 17, 2026
31 checks passed
@github-actions github-actions Bot added the canary:0.0.10-canary.4 Published NetScript prerelease 0.0.10-canary.4 label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:kv packages/kv canary:0.0.10-canary.4 Published NetScript prerelease 0.0.10-canary.4 ci:skip-e2e ci:skip-scaffold Skip the scaffold-static (deno-only) scaffold gate priority:p2 Medium status:shipped type:fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[kv] Redis watch stops silently once the subscriber gives up reconnecting

1 participant