Skip to content

fix(sagas): retain ordered SSE events for independent readers - #2646

Merged
rickylabs merged 12 commits into
mainfrom
fix/saga-event-feed
Nov 17, 2026
Merged

rickylabs merged 12 commits into
mainfrom
fix/saga-event-feed

Conversation

@rickylabs

@rickylabs rickylabs commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

Summary

Replace destructive saga SSE polling with a retained KV journal. Every reader has its own ordered cursor; stable ids support reconnect replay, and slow or expired readers receive an explicit gap.

Scope

Closes #2637

The portable core adapter uses256event slots plus one head. One atomic head/slot CAS orders successful commits across API instances; fixed-head watch plus bounded known-slot reads replaces polling and deletion. Native oRPC Last-Event-ID and error frames preserve the existing route/schema. Default API publication remains acceptance first, observation second.

Slices

  • Fresh-main research and four genuine red HTTP regressions.
  • Independent feature PLAN-EVAL APPROVED.
  • Domain contract and portable bounded journal.
  • Native HTTP integration and memory/DenoKV/Redis/Garnet proof.
  • Public retention/replay docs and canonical generated consumers.
  • Independent IMPL-EVAL PASS and exact-owned resource cleanup.
  • Final required head checks and review-thread gate.

Validation

Main0/4HTTPreds become green. Final full core249pass/0fail/4existing optional ignores; plugin196pass/0fail/2existing optional ignores. New core journal/subscription44/44 and HTTP9/9 include separate native Redis/Garnet clients, concurrent writers, slow readers, native header replay, cursor/gap/error/cancel/overwrite races,257-key capacity and large per-record values. Native Redis commandstatsSCANdelta0 over startup+idle+close.

Scoped check/lint/fmt0; quality:gate/architecture, release:preflight/docs:accuracy, nativeJSR dry-runs/audits and canonical consumer freshness pass. Exact doc-lint baseline core7/plugin30 unchanged. Full scaffold.runtimeN/A: no scaffold/launch/DB/Aspire/CLI/publish-map behavior change; asset generation confirms no CLI scaffold output diff.

issue: 2637
entries:
  - box-index: 1
    evidence: "Real HTTP two-reader red on main, green with matching ids/events on nativeDenoKV/Redis/Garnet. https://github.com/rickylabs/netscript/pull/2646#issuecomment-6935399907"
  - box-index: 2
    evidence: "Atomic multi-client ordering and native Last-Event-ID replay, exact boundary/overrun/reset/future/foreign and error frames. https://github.com/rickylabs/netscript/pull/2646#issuecomment-6935399907"
  - box-index: 3
    evidence: "Idle native list0, fixed-key spies on four backends and real Redis INFO SCANdelta0 during startup/idle/close. https://github.com/rickylabs/netscript/pull/2646#issuecomment-6935399907"
  - box-index: 4
    evidence: "300publications retain257keys; core/plugin READMEs and saga guide document256global events, replay/gaps/upgrade/durability with fresh generated consumers. https://github.com/rickylabs/netscript/pull/2646#issuecomment-6935399907"

Independent evaluator re-ran core249/0/4existing and plugin196/0/2existing, targeted46/0/0 and HTTP9/0/0, all277source checks/lint/fmt0 and public/consumer gates. Exactly2owned containers/2unshared original volumes removed after maintained preview and native identity/mount/reference proof; zero survivors, foreign resources retained.

Post-evaluation documentation correction: the native README fence gate caught five undefined example names. The example now declares its host KV, signal and optional cursor as typed function parameters;115fences compile with0exemptions and exactly the existing5consumer-import diagnostics in4otherREADMEs. Framework behavior and public types are unchanged. Documentation preflight/accuracy and canonical consumer freshness pass; normal required new-head CI remains authoritative.

Harness

Run: .llm/runs/fix-saga-event-feed--issue-2637/. Feature tier, Archetype3 with KV integration2/plugin5. Separate GLM PLAN-EVAL approved; separate Muse IMPL-EVAL PASS. Plan and decisions, validation history.

Drift / Debt

No transport/Streams, runtime engine publisher, lifecycle registry or process change. Fixed global count retention and a versioned prefix deliberately leave legacy history unmigrated; upgrade drains old API instances and clears old cursors. Unknown append outcomes are not blindly retried. Four lock snapshots add only metadata for already locked std async, required by frozen workspace guards; no version/integrity change. Existing doctrine/doc-lint debt unchanged; audit helper miscounts the normal slow-type banner, native output has no slow-type finding.

Definition of Done

  • Two concurrent clients both receive every published event, with a genuine main regression.
  • Stable publish-order ids and Last-Event-ID replay within retention.
  • Idle feed issues no periodic keyspace scan.
  • Bounded retention and reconnect/gap semantics documented.
  • Independent evaluation, final cleanup, all required current-head checks and review threads complete.
  • Taxonomy and milestone0.0.10.

🤖 Generated with Codex

@rickylabs rickylabs added this to the 0.0.10 milestone Nov 17, 2026
@rickylabs rickylabs added type:fix ci:skip-e2e area:plugins plugins/* and plugin-core packages status:research Harness research phase priority:p2 Medium ci:skip-scaffold Skip the scaffold-static (deno-only) scaffold gate area:sagas labels Nov 17, 2026
@rickylabs rickylabs added status:plan-eval Awaiting plan evaluation and removed status:research Harness research phase labels Nov 17, 2026
@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: RESEARCH]

Issue2637 reproduced on unchanged main b0eb079. HTTP regressions:0pass/4fail/0ignored. Fast client sees6while concurrent paused reader misses; resident sees3while native Last-Event-ID reconnect misses consumed data; idle raw native KV list4calls/350ms;300HTTP200publications leave300keys. Real Hono/oRPC/native Deno KV and default production writer, with only the unrelated runtime acceptance receiver inert. No mocked journal/watch; all handles closed and no backend resources started.

Native public KV atomic/fixed-key watch and oRPC lastEventId confirmed by deno doc/source. A native wire probe establishes HTTP200 can precede standard SSE error frames, so gap handling must use the native protocol. Scoped test check/lint/fmt0. Exact doc-lint baseline core7/plugin30 measured, not waived. No product implementation.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: PLAN]

Locked feature plan: one core portable KV journal,256ring slots+head, atomic append order and stable epoch:sequence ids. Native fixed-key subscribe-first/ready/reread plus per-reader cursors and a coalescing notification pump replaces polling and deletion. Known-key reads stay bounded; slow/expired/reset readers receive explicit native error frames, never silent loss/reset. Canonical SSE type identity preserved; plugin remains thin HTTP/publish wiring.

Count retention, native-value sizing, CAS conflict bounds, wire error codes, startup/overwrite races, cancellation/unknown outcomes, global filtering and legacy cursor/key upgrade boundary are decided. Per-slice file counts<30 and all F1-F19/integration/plugin gates named. Public quick-start/type contract precedes implementation. Existing package/core KV/watch ports and already locked @std facilities reused; no generic lifecycle/bus or dependency-version change.

Runtime proof is real HTTP plus native memory/DenoKV/Redis/Garnet, full affected suites and public/JSR/docs consumers. Full scaffold runtime N/A under user cost policy because this scope changes no scaffold/process/DB/CLI/Aspire/publish-map wiring; rescope triggers it if that changes. No saga transport or Streams edits. Separate GLM5.3/provider-default Plan-Gate next, product source remains unchanged.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: PLAN-EVAL] [VERDICT: APPROVED]

Independent feature-tier PLAN-EVAL (cycle 1 of 2): the locked plan at .llm/runs/fix-saga-event-feed--issue-2637/plan.md passes all 8 Plan-Gate boxes. Implementation may begin at S1. Full evaluation: .llm/runs/fix-saga-event-feed--issue-2637/plan-eval.md (uncommitted run artifact of this branch).

Route and independence

Requested and observed route identical to the pinned feature PLAN-EVAL row: GLM 5.3 at provider default via OpenCode Go, fresh session; fallback Fable 5.1 low unused (no failure evidence). Generator: Codex gpt-6.1-sol/xhigh — separate session, different vendor family. Evaluated head 224e8370; product source verified identical to main b0eb07957 (branch diff = 7 run artifacts + 1 committed HTTP test only).

Evidence independently re-verified (not taken from the generator)

  • Red oracles re-executed through the structured test wrapper: plugins/sagas/tests/services/event-feed-http_test.ts → 0 pass / 4 fail / 0 ignored, each failure matching one issue behavior (paused-reader loss, Last-Event-ID replay loss, 4 idle keyspace scans /350 ms, 300 unbounded keys).
  • Doc-lint re-measured at unchanged main source: core combined 7, plugin combined 30 (private-type-ref only) — matches the plan's baselines; the zero-new-diagnostics bar is well-defined.
  • Source claims spot-checked in plugins/sagas/services/src/routers/v1-handlers.ts (destructive watchPrefix 100 ms + delete-after-yield, per-connection ids, random-key publish, acceptance-before-write ordering and writeEvent seam) and across @netscript/plugin-sagas-core / @netscript/kv (742-line contracts file, 17-line domain vocabulary file among 14, WatchableKv subscribe-first/ready semantics, Deno KV getMany 10-key batching, Redis WATCH/MULTI/EXEC with post-commit publish, shared per-channel watch subscriptions, memory atomic/watch with existing contract tests).
  • Native seams confirmed via deno doc: oRPC ProcedureHandlerOptions.lastEventId (plus optional signal) and ORPCError code/status/data — cursor/gap error mapping (400/409/503 as native SSE error frames after a committed HTTP 200) rests on native error classes, not HTTP statuses.
  • Scope safety: packages/plugin-sagas-core/src/transports/ untouched; deferred-scope cross-references verified real (issue [kv] Redis watch stops silently once the subscriber gives up reconnecting #2639, PR feat(workers): pause intake and drain without charging shutdown deliveries #2645); scaffold.runtime N/A keying on what the run changes is honest, with the plan's explicit drift clause if that ever diverges.

Findings (non-blocking; no Plan-Gate box failure)

  1. Closing keyword before ready-merge (owner: generator, when slices land): the PR body's ## Scope references "Issue [sagas] SSE event feed scans the keyspace every 100 ms per client and loses events #2637" without a closing keyword, so merging would not auto-close it. Add Closes #2637 to the body.
  2. Epoch-minting wording at S1/S2: pin that the first append onto an absent head mints the epoch and later appends inherit it via the head CAS — the only reading consistent with the plan's own multi-adapter-writer and slot-fencing tests.
  3. Error-class test posture: keep S2/S3 assertions on the native error frame and on keeping history-lost vs invalid-cursor separately observable, rather than on an HTTP status or a shared message.

Next

  • Proceed with S1 (contracts/quick-start, JSR closure) following the locked slice order; per-slice gates and slice review as specified.
  • Any second PLAN-EVAL round follows the feature row's cycle-2 in-flight repair rule; this runtime's fresh-delegate requirement is recorded as tool drift in the run's supervisor.md.
  • Status-label transition to status:impl remains owned by the supervisor/implementation lane, as evaluator scope here excludes label changes.

@rickylabs rickylabs added status:impl and removed status:plan-eval Awaiting plan evaluation labels Nov 17, 2026
@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

Independent PLAN-EVAL APPROVED, all8boxes. S1 preserves canonical SSE type identity at contracts/v1 while moving vocabulary into the existing domain message file; adds structured invalid-cursor/history-lost/unavailable errors and public quick-start. Epoch creation/inheritance pinned explicitly. Contract tests3/3; scoped check/lint/fmt0, exact doc-lint baseline unchanged, quality:gate/preflight/accuracy PASS. Original4HTTPreds remain pending S2/S3. No transport, dependency version, publish-map or process wiring change.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

S2/S3 completed: portable256-slot+head journal and native HTTP wiring. Original4HTTPregressions green; extended nativeDeno/memory/Redis/Garnet coverage40/40green,0ignored (journal contract41/41 before one extra post-ready watch-failure case). Real HTTP multi-client ids/replay, native400/409/503 error frames after HTTP200, filter/cancellation/heartbeat semantics verified. Redis INFO commandstatsSCANdelta0 over startup+idle+close. Exact doc-lint core7/plugin30 unchanged, static/quality/preflight/accuracy/nativeJSR green. Four lock snapshots add only the existing std async dependency metadata required by frozen guards; no version/integrity change. Public docs/consumer refresh and independent final evaluation remain.

@rickylabs

rickylabs commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

All four issue2637 acceptance criteria are now proven at source0b7ca700faeaecee8965dc42e9da7a86026b4e68. The original real HTTP/nativeDenoKV suite reproduced0pass/4fail on unchanged main and is now green. Extended HTTP9/9 (including realRedis/Garnet), core journal/subscription44/44, and full core249pass/0fail/4existing optional ignores plus plugin196pass/0fail/2existing optional ignores. No new ignore; all newly added backend cases execute with the owned endpoints.

  1. Independent fast/paused readers receive identical data ids/events; actual HTTP on nativeDenoKV/Redis/Garnet and two separate native Redis/Garnet clients share the same journal.
  2. Atomic commit order yields one epoch and contiguous stable sequence ids; native Last-Event-ID replays retained events after another reader consumes. Exact oldest predecessor boundary, expired/foreign/reset versus malformed/future cursors and native terminal error frames tested.
  3. Idle raw native list spy0; all backend fixed-key watch/list/watchPrefix spies0scans; native Redis INFO commandstats SCANdelta0 during startup+500msidle+close. No polling/SCAN in the feed.
    4.300successful publications retain257keys;multiple16KiB native records exceed one aggregate value-size limit without exceeding any per-record limit. Core/plugin READMEs and saga guide document global256-event retention, ready/live/replay/filter/error/cancellation/durability/uncertain-outcome and legacy upgrade semantics. No transport/Streams dependency.

Structured14-file check/lint/fmt0 plus amended test scope0; quality:gate/arch/preflight/accuracy, nativeJSR dry-runs/audits and canonical consumer freshness all pass. Core7/plugin30 exact doc-lint baseline unchanged, no new cardinality/slow types (audit helper's normal banner miscount disclosed). Four metadata-only lock snapshots reuse the existing std dependency; no version/integrity change. Actual canonical generation produces no CLI/scaffold launch diff, so full scaffold.runtime remains N/A under approved scope/cost policy.

Independent implementation evaluation is next. Draft final DoD remains unchecked pending that verdict, final owned-resource cleanup and required-head checks/review threads. Two exact owned backends retained for evaluator probes; no foreign resources touched. No merge or publication.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL-EVAL] [VERDICT: PASS]

Independent feature IMPL-EVAL passes for issue 2637 at head 3e5e6745760abac0867c5d7e87e35bb311263937. Approved scope complete with no unapproved design change.

Findings

  1. Process/plan fidelity PASS — Plan-Gate PASS at 224e8370 before implementation; Design checkpoint in worklog.md; slices S1-S5 match plan.md via 334f21485, 5545d55f0, 5175c6f1b, 41213d147, 0b7ca700f, 3e5e67457; packages/plugin-sagas-core/src/transports/ untouched; no Streams dependency.
  2. Static/fitness PASS — Structured wrappers: core 154 files check/lint/fmt 0, plugin 123 files check/lint/fmt 0; doc-lint core combined 7 / plugin combined 30 unchanged, 0 missing; deno publish --dry-run Success in packages/plugin-sagas-core; deno task quality:gate, arch:check, release:preflight, docs:accuracy, MCP prose/corpus/publish freshness PASS; JSR audit exit 0 with only preexisting cardinality plus known banner miscount.
  3. Runtime PASS — Targeted packages/plugin-sagas-core/tests/contracts/saga-event-feed_test.ts, tests/stores/saga-event-feed_test.ts, tests/stores/saga-event-subscription_test.ts 46/0/0 across memory/DenoKV/owned Redis/Garnet; plugins/sagas/tests/services/event-feed-http_test.ts 9/0/0 including real Redis/Garnet fanout/replay; full core 249/0/4 preexisting ignores and full plugin 196/0/2 preexisting ignores re-ran independently; idle list/watchPrefix 0 with fixed-head watch; 300 writes retain 257 keys; large per-record values avoid aggregate cap; native BAD_REQUEST/CONFLICT/SERVICE_UNAVAILABLE error frames and sagaName filter verified.
  4. Route/independence PASS — Requested/observed feature IMPL-EVAL primary Muse Spark 1.3 xhigh via OpenCode Go, fallback Opus 5 xhigh unused; generator Codex gpt-6.1-sol xhigh separate session/vendor family; feature cycle 1 of 5; live agentic:matrix --tier feature --impl-evaluator matches.
  5. Non-blocking only — Preexisting doc-lint cardinality and audit banner unchanged; draft DoD last box, owned-resource cleanup, and required-head CI remain parent-owned; review threads 0/0; path-filtered checks skipping as expected for draft status:impl-eval with ci:skip-e2e/ci:skip-scaffold.

Next

  • Parent to retain evaluate.md uncommitted, run required-head CI, complete owned cleanup, then transition readiness; no merge/publication from evaluator.
  • Full scaffold.runtime remains N/A: no scaffold/process/DB/CLI/publish-map changes, lock diffs metadata-only, asset generation shows zero CLI/scaffold diff.

Run: .llm/runs/fix-saga-event-feed--issue-2637/evaluate.md. PR: #2646

@rickylabs rickylabs removed ci:skip-e2e ci:skip-scaffold Skip the scaffold-static (deno-only) scaffold gate labels Nov 17, 2026
@rickylabs
rickylabs marked this pull request as ready for review November 17, 2026 02:06
@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

Independent feature IMPL-EVAL PASS retained unchanged; no product delta. Evaluator independently confirmed full core249/0/4existing, plugin196/0/2existing, targeted46/0/0 and HTTP9/0/0, all277source gates and public/doc/consumer fitness.

Exact2owned backend containers and2original unshared volumes removed after maintained read-only leak/teardown preview plus native immutable identity/label/image/mount and survivor-reference proof. Final owned/volume scans0; post-cleanup maintained reporter0; all foreign/unproven objects retained. Sanitized final evidence committed.

Moved out of draft and removed draft skip labels so normal required head checks run. Final checks/threads DoD remains unchecked until those head results are verified; no local-green substitution. Full scaffold.runtime remains honestly N/A under independently approved actual scope. No merge, ready-merge or publication.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

Required quality found five unbound names in the new README example. Native docs:readme-fences reproduced10typeerrors/5failingREADMEs, then returns PASS at exact baseline5consumer-owned import diagnostics/4otherREADMEs after explicit typed host-context parameters.115checked/0exempt; no gate/ratchet/exemption change. No framework/API/runtime delta after independent PASS.

release:preflight fourPASS, docs:accuracyPASS, canonical prose/corpus/publish freshness PASS with no generated diff. Existing independent runtime evidence and zero-owned-resource proof retained. Correction pushed; required head CI and final DoD remain pending their actual results.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

Old head: 0fd41d417a801324d838f53d1c15a9ada3b2f5a7. New head: e14aab33a0e177a15bb79977f9be37c3f6fe1090. Merged main: 20b7180e62dc88834ae96a3a486d45fc9dd04270. The delta is merge plus canonical regeneration only: the handwritten PR diff against main is byte-identical before and after. No handwritten files conflicted; the publish-assets conflict was resolved by the full generated-carrier chain.

Local gates pass: structured check/lint/fmt across 277 saga files; 60 focused tests with zero failures or skips, including real Redis/Garnet fanout and replay; quality/architecture; all carrier freshness checks; release:preflight; docs:accuracy; README fences. Review threads: 0 unanswered. Both owned backend containers and their two original unshared volumes were removed. Labels, PR body and issue checkboxes are unchanged.

@rickylabs
rickylabs merged commit 8fe1080 into main Nov 17, 2026
35 checks passed
@github-actions github-actions Bot added the canary:0.0.10-canary.4 Published NetScript prerelease 0.0.10-canary.4 label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:plugins plugins/* and plugin-core packages area:sagas canary:0.0.10-canary.4 Published NetScript prerelease 0.0.10-canary.4 priority:p2 Medium status:shipped type:fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[sagas] SSE event feed scans the keyspace every 100 ms per client and loses events

1 participant