Skip to content

fix(runtime): close owned resources and bound process shutdown - #2641

Merged
rickylabs merged 20 commits into
mainfrom
fix/runtime-child-drain
Nov 17, 2026
Merged

rickylabs merged 20 commits into
mainfrom
fix/runtime-child-drain

Conversation

@rickylabs

@rickylabs rickylabs commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Summary

Background children and services now finish signal shutdown with explicit exit codes after bounded work and resource cleanup. Child health binds loopback by default and accepts a native hostname override.

Scope

Closes #2630
Closes #2631

  • Decisions: exit 0 clean, 1 failed, 2 timed out; child 35 s total (30 s work + 5 s cleanup), service 30 s total (25 s work + 5 s cleanup). Manual stop/external abort retain host KV/DB ownership. Explicit host disposal disables duplicate raw-builder DB disposal.

Slices

  • S1 Main-baseline real Redis subprocess reproduction and approved design.
  • S2 Bounded lifecycle, signal-owned KV/Prisma disposal, loopback binding and OTel exit ownership.
  • S3 Generated acquired-resource disposal, public docs and regenerated consumers.
  • S4 Supported-runner full runtime verdict.
  • S4 Independent implementation evaluation and final current-head gates.

Validation

  • Real Redis child/service clean and timeout regressions: four watchdog failures on base; 4/4 pass after implementation.
  • Child bind and OTel ordering regressions fail before and pass after.
  • Full structured service/plugin suite: 820 passed, 0 failed, 4 optional Redis cases skipped; those four passed separately with real Redis.
  • Generated owner/installed runtime tests: 37/37 pass; acquired/unused and duplicate client cases covered.
  • Structured changed-source check, lint, format; quality:gate; docs:accuracy; release:preflight; generated-carrier freshness: pass.
  • Service doc lint: clean. Plugin 18 and telemetry 7 diagnostics match measured base distributions. Plugin JSR audit reports three missing module tags in untouched entrypoints; its dry-run succeeds. Service/telemetry/CLI JSR audits pass with existing warnings.
  • Full local scaffold.runtime attempted twice: remote Docker prevented loopback readiness, then plugin doctor explicitly rejected that topology. A temporary owned-only bridge proved infrastructure and affected service/worker health; this is partial evidence, not full smoke PASS. No guard disabled. Supported workflow43951994678 SUCCESS at sourced6ef1fa71: Postgres124/124, SQLite116/116 plus showcase8/8, scaffold-static/native and cleanup all pass.
  • Temporary bridge and exact diagnostic AppHost stopped; teardown preview/apply removed one owned diagnostic PostgreSQL container. Final identity scans show no owned mount/bridge container survivors; foreign resources untouched.
issue: 2630
entries:
  - box-index: 1
    evidence: "https://github.com/rickylabs/netscript/pull/2641#issuecomment-6929449167 — real Redis subprocess child clean/timeout red on main, green after bounded explicit exit."
  - box-index: 2
    evidence: "https://github.com/rickylabs/netscript/pull/2641#issuecomment-6929449167 — real Redis service subprocess clean0/timeout2; fatal1 preserved."
  - box-index: 3
    evidence: "https://github.com/rickylabs/netscript/pull/2641#issuecomment-6929449167 — singleton KV/Redis and identity-deduplicated acquired Prisma disposal tests; owner/runtime37/37 and supported full cleanup."
  - box-index: 4
    evidence: "https://github.com/rickylabs/netscript/pull/2641#issuecomment-6929449167 — plugin/service/telemetry docs and source-derived child-health docs describe exit0/1/2 and cleanup budgets."
issue: 2631
entries:
  - box-index: 1
    evidence: "https://github.com/rickylabs/netscript/pull/2641#issuecomment-6929449167 — default loopback regression fails main; generated controls also red on defaults before fix."
  - box-index: 2
    evidence: "https://github.com/rickylabs/netscript/pull/2641#issuecomment-6929449167 — native hostname option and generated environment forwarding;6/6real HTTP bind controls including non-loopback interface."
  - box-index: 3
    evidence: "https://github.com/rickylabs/netscript/pull/2641#issuecomment-6929449167 — generated non-loopback HTTP override controls, installed artifact31/31, supported Aspire generated role health/readiness200 and full runtime124/124."

Harness

  • Run dir: .llm/runs/fix-runtime-child-drain--issues-2630-2631/.
  • Feature tier; PLAN-EVAL PASS before implementation; separate-session IMPL-EVAL cycle2 PASS. Current draft required checks policy-skipped; thread gate0/0. Final verdict-retention commit is evidence only; current ready-head checks will be confirmed after transition.
  • Detailed validation and environment evidence: worklog.

Drift / Debt

  • Current services use explicit native hostname options; there is no hostname environment fallback on the base. Child follows that option convention.
  • Standalone OTel flushes within a bound and leaves termination to its composing host; documented.
  • Generated workers/sagas/triggers forward NETSCRIPT_CHILD_HEALTH_HOSTNAME to the native option; explicit override serves real HTTP over a non-loopback interface,6/6tests pass. Native restart error-path diagnostic preserves exit/detail (2red->green,14/14adjacent tests); no retry or gate bypass added.
  • Existing measured doc/JSR diagnostics remain disclosed above. No saga transport changes or Redis Streams dependency.

Definition of Done

  • Exit/bind regressions proved failing before and passing after.
  • Owned resource disposal, embedding semantics and exit codes implemented and documented.
  • Generated launches pass full scaffold.runtime on a supported runner.
  • Independent implementation evaluation, required checks and review threads pass.

🤖 Generated with Codex

@rickylabs rickylabs added this to the 0.0.10 milestone Nov 16, 2026
@rickylabs rickylabs added type:fix area:plugins plugins/* and plugin-core packages status:plan-eval Awaiting plan evaluation priority:p1 High area:service packages/service labels Nov 16, 2026
@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: PLAN]

Both reported process-exit failures reproduce on current main with a real Redis socket. The structured subprocess tests reach readiness, receive SIGTERM, and each exceeds the exit watchdog: child clean/timed-out drain 0/2 pass; service clean/timed-out drain 0/2 pass. Repeated on the pinned Deno 2.9.7 with the same four failures. Deno KV alone is a negative control and exits naturally.

Commit 2913e2a locks process ownership, defined 0/1/2 exit semantics, explicit loopback hostname override, and cleanup deadlines within the total shutdown budget. Plan includes generated role ownership and one full scaffold.runtime smoke. Separate-session PLAN-EVAL is required before product implementation.

The isolated Redis resource was removed after the teardown preview and exact ownership verification. Foreign/unknown-owner resources were left alone. No operator environment values enter public evidence.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: PLAN-EVAL] [VERDICT: CHANGES_REQUESTED]

Gate result: FAIL_PLAN — the plan's scope, root cause, deadline accounting, ownership model, and role coverage are sound, but four unstated decisions and a slice-contract gap must be locked in plan.md before implementation. All required fixes are plan-text edits; no re-planning. Verdict artifact: .llm/runs/fix-runtime-child-drain--issues-2630-2631/plan-eval.md.

Evaluated head 2913e2a51c160dfe191e0ec524f921cd349d9321 (base origin/main b0eb079575968e843b781579673c9fe2ae250e07). Separate-session feature-tier PLAN-EVAL per the pinned matrix (glm_5_3/provider_default, OpenCode Go transport); generator is the Codex Sol xhigh session — different session and vendor family, no substitution.

Findings

  1. Clean-path termination not locked. Exit 2 can only terminate explicitly (a hung run promise retains the event loop), but the plan does not state whether the clean path exits explicitly or relies on natural event-loop drain after cleanup. Natural drain is nondeterministic under foreign retainers — including an OTel periodic metric reader that this PR's own OTel fix may leave running when it stops exiting early. Fix: lock explicit termination with the computed exit code after the drain/cleanup budget completes or expires, for both child and service.
  2. Prisma disposal point ambiguous; the committed S1 test pins the other reading. Plan item 4 says "implement at composition roots" (reads as the defineService/createPluginService presets), but the S1 service test pins the raw builder: createService(...).withDatabase({ first: client, alias: client }) expecting exactly-once $disconnect. Preset-only disposal leaves that test permanently red. Fix: implement the identity-deduplicated $disconnect in the service builder's database registration (withDatabase, inherited by both presets), running in the reserved cleanup phase; keep the S1 test as written.
  3. Shared-KV disposal route not named. closeKv() (packages/kv/application/shared.ts) is singleton-aware and a safe no-op when never initialized; the lazy PluginServiceContext KV wrapper's close() resolves the underlying adapter on first call — closing an unused context KV would open a Redis connection merely to close it (violating the plan's own "never resolve an unused client" principle) and leaves the shared singleton pointing at a closed adapter for embedded manual-stop hosts. Fix: name closeKv() as the executable-role cleanup path; make the lazy wrapper's close() a no-op when unresolved or document it as not a disposal path.
  4. Standalone-OTel behavior change not named. After removing the ahead-of-drain Deno.exit(0) (packages/telemetry/src/adapters/otel/otel-sdk.ts), a process using the opt-in SDK provider without a NetScript runtime no longer exits from OTel's own SIGTERM handler. Acceptable — process ownership belongs to the composing runtime — but it must be stated and covered by the S3 docs slice. Fix: state it in plan item 5 (bounded flush + listener removal retained, exit dropped, embedders compose a runtime or own signals) and add it to docs:accuracy scope.
  5. Slice contract incomplete (Plan-Gate box). S1–S4 are enumerated, ordered, and name what each proves and its proving gate, but per-slice file groups are not listed, so the <30-file bound is asserted rather than shown. Fix: add file groups per slice (S1 subprocess/bind tests; S2 packages/plugin child edge, packages/service builder/coordinator/listener, packages/telemetry OTel adapter; S3 the three plugin glue stubs, CLI service-context template, first-party service entrypoints, docs; S4 evidence only).

Confirmed

  • Research is current and every load-bearing finding spot-checked against the tree, including the [plugins] Child-health listener binds 0.0.0.0 #2631 drift correction (no hostname env fallback exists in current main; the native explicit hostname option is the right convention to reuse).
  • The Redis-backed red baseline (child clean/timeout 2/2, service 2/2, KV-alone negative control, watchdog-tolerant harness, Docker-network harness failure excluded) is valid evidence; the container was already removed.
  • All seven acceptance boxes across [service] Processes do not exit after a drain, and nothing closes KV or Prisma #2630/[plugins] Child-health listener binds 0.0.0.0 #2631 are meetable after the fixes; no missing roles (workers/sagas/triggers children; workers/sagas/auth API services; sagas runner bootstrap-context DB; streams exits via the generic service semantics; scaffold app services via defineService). No generic lifecycle registry, no saga transport changes, no Redis Streams dependency. Manual stop and external abort remain embeddable; disposal stays signal-path scoped.
  • Risk register, gate set (incl. required full scaffold.runtime smoke for generated-glue changes), deferred scope, and the jsr-audit surface scan all pass.
  • Advisory: the service drainTimeoutMs semantic narrows to "total budget including reserved cleanup" (30 s → 25 s work + 5 s cleanup) — deliberate; document it. The conditional hostname env fallback stays deferred to the S4 smoke decision.

Next

  • Owner/generator: apply the five repairs to .llm/runs/fix-runtime-child-drain--issues-2630-2631/plan.md (or re-steer this evaluator to edit them on cycle two per the feature-tier policy), then re-run PLAN-EVAL. No implementation slice may be committed before a PASS.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: PLAN]

Applied all five cycle-one PLAN-EVAL repairs in commit f86b889: explicit Deno.exit on clean and failed process-owned paths; exactly-once Prisma disposal registered by raw withDatabase; closeKv() as the singleton-aware process route; OTel bounded flush without exiting, with standalone behavior documented; and per-slice file groups, each below 30 files. No production implementation has started. Manual stop and external cancellation remain embeddable.

The accepted real Redis reproduction remains the baseline. Cycle-two PLAN-EVAL is next, before implementation.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: PLAN-EVAL] [VERDICT: APPROVED]

Gate result: PASS — every Plan-Gate box is satisfied at head f86b8891; implementation may begin. Cycle two of the feature-tier maximum of two: all five cycle-one required plan repairs were verified applied, and no further plan edit was needed. Verdict artifact: .llm/runs/fix-runtime-child-drain--issues-2630-2631/plan-eval.md (cycle-one history preserved in the artifact).

Evaluated head f86b8891b52953569c5070a49bbc382c5aea38fd (base origin/main b0eb079575968e843b781579673c9fe2ae250e07). The head advances the cycle-one evaluated head with run-dir documentation only, so production sources equal main. Separate-session feature-tier PLAN-EVAL per the pinned matrix (glm_5_3/provider_default, OpenCode Go transport, fresh cycle-two session); generator is the Codex Sol xhigh session — different session and vendor family, no substitution.

Cycle-one repairs verified

  1. Final termination locked. Explicit Deno.exit with the computed code on clean, fatal and timed-out process-owned paths, for both child and service; never natural event-loop drain. Exit 0 clean / 1 fatal or cleanup failure / 2 drain timeout; the one-second fatal-health grace precedes the bound; the deadline is measured monotonically from shutdown admission.
  2. Prisma disposal point named to match the S1 test. Identity-deduplicated $disconnect registered at the raw builder withDatabase boundary, inherited by defineService and createPluginService, run in the reserved signal-path-only cleanup phase; covers every acquired client; the redundant preset-only health hook is removed. Aliases disconnect exactly once, as the committed raw-builder subprocess test pins.
  3. Shared-KV disposal route named. Singleton-aware closeKv() (no-op when unused) on executable-role signal shutdown, including its Redis client/subscriber; the lazy PluginServiceContext KV wrapper close() is explicitly excluded from process disposal (resolve-on-close; singleton poisoning for embedded manual-stop hosts).
  4. Standalone-OTel behavior change named. The SIGTERM handler keeps bounded flush and listener removal and drops Deno.exit; standalone embedders compose a runtime or own signal termination; stated in the plan and carried into the S3 telemetry docs and docs:accuracy scope.
  5. Slice contract complete. Per-slice file groups enumerated and counted: S1 = 8, S2 = 12, S3a = 16, S3b = 15, S4 = 4 run files — all below 30, ordered, each naming what it proves, its proving gate, and its files.

Checklist results

All eight gates/plan-gate.md boxes PASS:

  • Research present and current — re-baselined against main b0eb0795; the [plugins] Child-health listener binds 0.0.0.0 #2631 drift correction (current main exposes the explicit native hostname option, not an environment convention) is recorded in drift.md.
  • Decisions locked; open-decision sweep clean — only the conditional hostname environment delivery remains open, explicitly safe to defer to the S4 scaffold.runtime decision, with the explicit native hostname override locked.
  • Commit slices below 30 files each, with gate + files enumerated.
  • Risk register with mitigations (embedded shared resources, hanging cleanup, fatal-grace retention, in-flight work).
  • Gate set matching Archetype 3 plus the release-gate class for generated runtime glue (full one-pass scaffold.runtime smoke in S4).
  • Deferred scope explicit (workers pause/delivery-budget, saga SSE log, other milestone fixes; no saga-transport changes; no Redis Streams).
  • jsr-audit surface scan for the planned public surface (additive optional fields, documented, no new dependencies; slow-type risk named).

The evaluator re-verified the load-bearing findings against the tree at the head: child health binds without hostname and awaits completion unbounded with only the fatal path setting an exit code; service shutdown bounds the 30 s drain with no process exit anywhere in packages/service; the only runtime Deno.exit is the OTel SDK handler exiting 0 ahead of drain; defineService disconnects only the health-check client; closeKv() is the singleton-aware route. Both untracked S1 research tests were re-read: they pin drainTimeoutMs and bounded onShutdown on the child, exit 0/2, PROBE_CLOSED, raw-builder exactly-once PROBE_DB_CLOSED under aliases, and a hanging drain hook that cannot prevent remaining disposal — all consistent with the repaired plan. The recorded Redis-backed red baseline stands (child clean/timeout 2/2 and service clean/timeout 2/2 fail the 5 s exit watchdog after readiness on an isolated real Redis endpoint; Deno KV alone exits naturally; the corrected Docker-network harness failure is excluded; the container was removed).

Acceptance coverage

All seven acceptance boxes across #2630 and #2631 are meetable: child SIGTERM exit within its drain budget (S1 red subprocess tests → S2); service exits after clean and timed-out drains with distinct codes (S1 → S2, exit vocabulary 0/1/2); role-level KV/Prisma/Redis closure (S2 core closeKv() + raw-builder disposal; S3a generated glue cleanup callbacks, host-bootstrap acquired-only disposal, sagas runner DB disposal); exit codes documented (S3b docs under docs:accuracy); loopback-by-default child health (S1 bind tests → S2 default); explicit override reusing the plugin-service native hostname convention (S2); generated container/Aspire launches still pass health checks (S4 full smoke including hostname overrides). No missing roles (workers/sagas/triggers children; workers/sagas/auth API services; sagas runner bootstrap-context DB; streams via generic service semantics; scaffold app services via defineService); the web frontend and Aspire host are out of scope. No generic lifecycle registry, no plugin-sagas-core transport changes, no Redis Streams dependency. Manual stop and external abort remain embeddable; disposal stays signal-path scoped.

Plan-only repairs

None required on cycle two — the committed plan satisfies every box without further edit.

Next

  • Implementation may begin, one slice at a time with per-slice gates and per-slice PR comments: S1 tests (red baseline) → S2 core → S3a generated glue → S3b docs/assets → S4 one-pass full smoke.
  • Independent IMPL-EVAL (separate session, different vendor family) remains mandatory before any readiness or merge consideration; no merge or publication before that gate.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

Committed the S1 regressions after PLAN-EVAL PASS. In addition to the four real Redis subprocess failures already recorded, unchanged main source fails the loopback bind regression and exits 0 during the OTel/runtime ownership test instead of preserving the runtime's drain and exit 7. Baseline focused suite: 9 pass, 2 fail. Signal subprocess cases skip Windows because the acceptance uses SIGTERM.

@rickylabs rickylabs added status:impl and removed status:plan-eval Awaiting plan evaluation labels Nov 16, 2026
@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

S2 commit 029b9e6 makes the lifecycle regressions pass. Child health defaults to loopback with an explicit hostname override. Process-owned child/service shutdown uses bounded drain/cleanup deadlines and explicit 0/1/2 exits; raw withDatabase registration deduplicates client identities for reserved signal cleanup, and KV cleanup uses closeKv(). OTel retains bounded shutdown without taking over process termination.

Real Redis subprocess tests now pass 4/4 (formerly four watchdog failures); focused child/coordinator/OTel tests pass 15/15; expanded resource/coordinator tests pass 8/8; existing listener/builder/runtime tests pass 34/34. Structured type-check/lint/fmt and quality:gate pass. Two embedded signal tests explicitly retain process ownership and preserve all assertions. Owned Redis resource was removed after read-only teardown preview and identity proof.

Generated role disposal, documentation/assets, final scaffold.runtime smoke and IMPL-EVAL remain pending.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

S3a commit eab4bc2 adds owner-supplied idempotent context disposal and generated DB acquisition tracking. Standard child roles close shared KV and invoke the host bootstrap disposer; workers/sagas/auth service signal cleanup invokes the owner callback. The host callback closes only acquired DB identities; KV remains singleton-aware process-edge cleanup. When a host callback owns DB disposal, disposeDatabaseOnSignal:false avoids duplicate automatic disconnects.

Before refreshing the embedded template, two generated-owner tests failed with missing disposal; after canonical gen:assets-barrel, the focused suite passes 37/37. It covers all three generated runtime artifacts, unused/acquired DB cases, disposal idempotency and exactly-once host DB cleanup under a real signal. Structured check/lint/fmt pass, using the explicit CLI quality config for files excluded by the root config. No lock changes.

Documentation, generated documentation/publish carriers, one-pass scaffold.runtime smoke and IMPL-EVAL remain.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

S3b documents shutdown ownership, loopback override, total/reserved budgets and 0/1/2 exits, and refreshes source-derived child health, shipped prose, export-surface and publish-asset carriers. Documentation generator tests pass 2/2; freshness, docs:accuracy, release:preflight and quality:gate pass. Reference formatting-only churn was removed from the final diff.

Service doc lint is clean. Plugin doc lint (18 diagnostics) and telemetry doc lint (7) exactly match the base command/distribution. Plugin JSR dry-run succeeds; its audit retains three missing @module tags in untouched entrypoints, proven unchanged by git diff. Service, telemetry and CLI JSR audits exit 0 with existing warnings. These baseline limits remain visible in the run evidence.

Started the required single one-pass command on pinned Deno 2.9.7:
deno task e2e:cli run scaffold.runtime --cleanup --format pretty

Runtime verdict and independent IMPL-EVAL are pending.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

S2-S3 and runtime validation follow-up are committed through 26e945e. Explicit process termination, bounded cleanup, deduplicated database ownership, singleton KV disposal, child loopback override and OTel composition are implemented. The service/plugin structured suite passes 820/820, and all four separately executed real Redis subprocess regressions pass. Static/publication/docs/carrier gates pass apart from the measured pre-existing doc/JSR diagnostics disclosed in the PR body.

The local full runtime command ran twice and cannot provide a complete verdict on this environment: native PostgreSQL readiness failed on remote Docker; a temporary bridge repaired reachability but plugin doctor correctly rejects remote Docker. Infrastructure and affected service/worker health were observed healthy. No guard was disabled. A supported-runner workflow is requested for the full runtime verdict.

Temporary bridge/AppHost cleanup is complete. Preview/apply removed one owned diagnostic container; final scans show zero owned mount/bridge container survivors. The temporary bridge's interference with persistent-network cleanup and already-removed-proxy cleanup error remain recorded in the run worklog. Foreign and unknown-owner resources were untouched.

Next: separate feature-tier IMPL-EVAL, supported CI runtime evidence, then issue acceptance/body reconciliation and readiness. No merge or release requested.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL-EVAL] [VERDICT: CHANGES_REQUESTED]

Independent feature-tier IMPL-EVAL returns FAIL_FIX. Lifecycle and ownership match the approved plan; release-gate evidence is missing.

Evaluator: separate session from the Codex generator. Requested muse_spark_1_3/xhigh, observed opencode-go/muse-spark-1.3-contributor. Different session and vendor family. Fallback not used. Head 26e945e, base origin/main b0eb079.

Findings

  1. F1 high — no supported-runner full smoke. Local scaffold.runtime stopped at remote-Docker loopback readiness, then at the maintained plugin-doctor remote-Docker rejection. Bridge evidence is partial only. Supported-runner e2e-cli dispatch is in progress with no terminal verdict. Do not claim full PASS.
  2. F2 high — generated non-loopback coverage open. Child defaults to 127.0.0.1 with an explicit native hostname option, but workers/sagas/triggers glue stubs forward only onShutdown plus restart history. No hostname forwarding or environment delivery, and the deferred S4 hostname decision is unrecorded. Issue 2631 third acceptance needs supported-runner probe evidence.
  3. Close-gate correctly red. Both issues and the PR Definition of Done leave smoke plus evaluation unchecked. No closing keywords yet.

Independent checks

  • Structured service check/lint/fmt PASS; child plus OTel focused tests 11/11 PASS; resources plus coordinator 9/9 PASS; quality:gate and arch:check exit 0; service doc-lint 0 diagnostics.
  • Hard restrictions honored: no plugin-sagas-core transport edits, no Redis Streams dependency.
  • No resources started; no foreign processes or containers touched.

Next

  • Generator: wire generated hostname override or prove loopback suffices on the supported runner, record the hostname decision in drift.md, then link the green one-pass smoke on this exact head.
  • Then re-run IMPL-EVAL currency checks before any ready-merge move. Artifact: .llm/runs/fix-runtime-child-drain--issues-2630-2631/evaluate.md (uncommitted).

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

Addressed IMPL-EVAL F2 in 0cea585: generated workers/sagas/triggers forward NETSCRIPT_CHILD_HEALTH_HOSTNAME to the existing native hostname option. Unset preserves loopback, including Aspire executable probes; container launches can explicitly set 0.0.0.0. The S4 decision and scope are recorded in drift.md.

The new subprocess regression executes all three emitted entrypoints and observes native bound addresses plus real HTTP health responses. On base b0eb079, 3/6 fail because default binds all interfaces. Before this repair, 3/6 fail because the override is ignored and non-loopback HTTP is refused. After repair, 6/6 pass; with installed artifact tests, 31/31 pass. Structured changed-source check/lint/fmt, quality:gate, release:preflight and docs:accuracy pass.

F1 remains pending supported-runner full runtime evidence. The prior full dispatch evaluates 26e945e; a final-head dispatch and evaluator currency pass follow generated consumer refresh. No local full smoke is repeated on the explicitly unsupported remote-Docker topology.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

S4 native failure diagnosis atd6ef1fa71: the supported Postgres smoke stops at generated-web resource restart after55passing gates; cleanup passes. Main SQLite comparison has the same symptom while main/Garnet Postgres passes, so no product cause is inferred yet. The CLI discarded Aspire's native restart exit code and diagnostic, preventing a decisive diagnosis.

Two genuine regression tests fail on the unchanged-main CLI error path (stderr and stdout fallback). Minimal repair retains native exit code/detail and the scoped manual remedy. Adjacent refresh suite14/14, scoped typecheck, quality:cli, release:preflight and docs:accuracy pass. Failure observability only: no retry, target, cleanup assertion, drain contract or readiness bypass. New supported final-head full smoke will provide the native error or a complete pass. No unsupported-host local rerun.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

S4 supported full-runtime verdict is now green atd6ef1fa71: workflow43951994678 SUCCESS, Postgres124/124, SQLite116/116 plus showcase8/8, scaffold-static and desktop-native all succeed. Cleanup runs intact. No guard bypass, weaker assertion or automated retry was added. Earlier native web-restart failure remains diagnosed as an intermittent supported-runner symptom also seen on main; the small error-path repair retains native exit/detail for future failures.

Both cycle-one findings are addressed: F1 complete supported smoke now passes; F2 all workers/sagas/triggers generated launchers forward NETSCRIPT_CHILD_HEALTH_HOSTNAME to the native option. Main baseline fails3/6on defaults; pre-repair branch fails3/6on ignored override; fixed generated bind tests6/6pass, serving real HTTP on a non-loopback interface for the explicit override. Combined installed artifact suite31/31. Full smoke confirms actual generated role health/readiness200 in Aspire.

Acceptance #2630: four real Redis subprocess regressions failed via watchdog on main and pass after repair (child/service clean and timeout; distinct0/2exitcodes), fatalexit1 preserved, all role-owned singletonKV/Redis and identity-deduped acquiredPrisma disposal covered by builder/host/installed tests, embedding retains external ownership, exitcodes documented. Acceptance #2631: native loopback default/override red->green and generated forwarding/health evidence above. Full service/plugin820pass/4optionalRedis separately4pass; owner/runtime37/37; all scoped static/quality/preflight/accuracy/carrier gates pass with measured unchanged doc/JSR baseline warnings disclosed in body/run.

Final independent IMPL-EVAL follow-up next. No merge, auto-merge, ready-merge or publication. Run .llm/runs/fix-runtime-child-drain--issues-2630-2631/.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL-EVAL] [VERDICT: PASS]

Independent feature-tier IMPL-EVAL cycle two. Fresh evaluator session; prior cycle-one FAIL_FIX history preserved in .llm/runs/fix-runtime-child-drain--issues-2630-2631/evaluate.md.

  • Exact head evaluated: 582372c4a3f160ea6b944c1ace574ab9a5c6ba67, base origin/main b0eb079575968e843b781579673c9fe2ae250e07
  • Green-smoke source: d6ef1fa71 (workflow 43951994678 SUCCESS); production/tests/docs identical to current head (diff exits 0, only 2 run files differ)
  • Route: requested muse_spark_1_3/xhigh, observed opencode-go/muse-spark-1.3-contributor; generator owner-selected Codex gpt-6.1-sol/xhigh; different session and vendor family; no substitution; fallback opus_5/xhigh unused

F1 complete supported smoke — repaired and verified:

  • Workflow 43951994678 SUCCESS at d6ef1fa71: Postgres 124/124, SQLite 116/116 plus showcase 8/8, scaffold-static and desktop-native success, cleanup intact
  • No guard bypass, retry, target, cleanup or lifecycle-gate change; no new unproven runtime root-cause fix

F2 generated hostname forwarding — repaired and independently verified:

  • All 3 actual workers/sagas/triggers entrypoints forward NETSCRIPT_CHILD_HEALTH_HOSTNAME into native hostname option (plugins/workers| sagas|triggers/src/adapter/resources/glue/runtime.stub.ts)
  • Default remains 127.0.0.1; explicit override serves real HTTP on a non-loopback interface
  • Independent: packages/plugin/tests/health/generated-hostname_test.ts 6/6 pass (structured wrapper exit 0); combined installed suite 31/31 reported; full smoke confirms Aspire generated health/readiness 200

Lifecycle (spot-check preserved):

  • Child/service process-owned paths exit 0 clean / 1 fatal-or-cleanup-failure / 2 timeout after monotonic work plus reserved cleanup; 35 total (30 work + 5 cleanup) child, 30 total (25 work + 5 cleanup) service; 1s fatal-observation grace precedes budget; loopback default, native explicit hostname, opt-out exitOnShutdown:false embeds
  • Service signal-only termination; manual stop / external abort / exitOnSignal:false preserve host ownership; raw withDatabase identity-deduped Prisma disposal inherited by presets; explicit host disposal option avoids double-disconnect; closeKv singleton-aware process path; generated host tracks acquired DB only with idempotent dispose; OTel bounded flush with listener removal, no Deno.exit
  • No generic lifecycle registry, no saga transports change, no Redis Streams

Independent gates at current head:

  • Isolated real Redis subprocesses: packages/plugin/src/presentation/run-child-health-process_exit_test.ts + packages/service/tests/signal-exit_test.ts 4/4 pass (structured wrapper exit 0); owned container used unique label, verified by immutable ID before removal, zero own survivors, foreign untouched
  • Structured check: service 131 files 0 diagnostics; child presentation 4 files 0; lint/fmt 131 files 0 findings
  • quality:gate exit 0, arch:check exit 0, docs:accuracy PASS, release:preflight PASS
  • Service doc-lint 0; plugin 18 / telemetry 7 match measured base distributions; changed health/SDK entrypoints clean; plugin JSR 3 missing module tags in untouched entrypoints with dry-run success (pre-existing, disclosed)
  • No expensive full suite rerun: no generated behavior change after green source head

Acceptance (7 boxes, all checked with linked proof #2641 (comment)):

Finding: low only — draft-head CI shows expected skipping apart from docs/build lanes; decisive green is the identical-source supported workflow above. No product/test/doc edits, commits, pushes, labels, merges, or watching in this lane. Run dir updated with cycle-two verdict, left uncommitted.

@rickylabs
rickylabs marked this pull request as ready for review November 16, 2026 21:30
@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

Ready-transition CI found one new public type diagnostic: ServeOptions.onCleanup reached the work-hook type through the workers service facade. The existing public doc-lint test reproduced red locally. Cleanup always belongs to the signal disposal phase, so its new contract now accepts zero-argument disposers; reason-aware work hooks remain on onShutdown. This preserves the separately bounded native type export inventories without widening them.

Validation at source 212a8d394: worker public contracts plus service resource ownership 7/7 pass, scoped check/lint/fmt zero, release:preflight and docs:accuracy pass. Canonical prose/export/assets generation produced no changes; export/publish freshness pass. Executable behavior is unchanged from the supported full runtime smoke and independent cycle-two PASS. Current head CI is running; no merge or readiness label change.

@rickylabs

rickylabs commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

Ready for independent review at 212a8d394: the cycle-two evaluator PASS and full supported scaffold smoke remain the executable evidence. The subsequent zero-argument disposer type closure repair is covered by the existing public doc-lint red-to-green test and resource ownership tests7/7; repair evidence. Current head checks22SUCCESS/13SKIPPED, required checks green/policy-skipped, review-thread gate0/0. All seven issue boxes and PR DoD/evidence mappings complete. No merge, auto-merge or ready-merge label.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL-EVAL] [VERDICT: PASS]

Narrow ready-transition follow-up at 212a8d39. Cycle-two PASS remains authoritative for all unchanged runtime behavior; this verdict covers only 582372c4..212a8d39.

Scope

  • Production delta is packages/service/src/types.ts only: ServeOptions.onCleanup narrowed from readonly ShutdownHook[] to readonly (() => Promise<void> | void)[], plus JSDoc directing reason-aware work hooks to onShutdown. No facade exports added, no bounded worker type-closure growth, no control-flow/launch/budget change.
  • Run-only notes in .llm/runs/fix-runtime-child-drain--issues-2630-2631/ (drift.md, worklog.md, context-pack.md); independent evidence appended to evaluate.md and left uncommitted.

Route

  • Feature-tier IMPL-EVAL primary muse_spark_1_3/xhigh via OpenCode Go, observed opencode-go/muse-spark-1.3-contributor; generator is the separate Codex gpt-6.1-sol/xhigh session (different vendor family, no substitution, no subagents).

Independent evidence

  • Focused suite plugins/workers/tests/public/type-refs_test.ts + packages/service/tests/service-resources_test.ts: 7/7 pass (structured wrapper, pinned Deno 2.9.7).
  • Scoped check/lint/fmt over packages/service/src (74 files): 0 diagnostics / 0 findings.
  • Service doc lint combined 0; workers ./service and ./services entrypoints 0 (previously failing facade now clean; remaining 2 contract diagnostics are the excluded pre-existing scope).
  • docs:accuracy PASS; release:preflight PASS on all four checks; no carrier diff beyond types.ts.
  • All onCleanup producers verified zero-arg (ctx.dispose, () => db.$disconnect(), () => closeKv()); internal shutdown path unchanged.
  • Current head: gh pr checks 22 pass / 13 skipping / 0 fail; check:review-threads 0/0; issues [service] Processes do not exit after a drain, and nothing closes KV or Prisma #2630 (4) + [plugins] Child-health listener binds 0.0.0.0 #2631 (3) boxes checked with fenced acceptance evidence in the PR body.

Findings

  • None blocking. Justified new-option contract change; no new doc/consumer/type errors.

No source/docs edits, commits, pushes, labels, watching, merge, or publication in this lane.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

Final evidence head 02711c97ff88c79d408e21f780d7c0b5530c8c2c contains only the retained final independent type-closure PASS and worklog. Production source equals the evaluated 212a8d394. Current head22SUCCESS/13SKIPPED, required quality/check-test pass and dependency report policy-skips; review-thread gate0/0. PR ready for review, all seven acceptance boxes/body DoD checked. No merge/auto-merge/ready-merge or publication.

@rickylabs

Copy link
Copy Markdown
Owner Author

[PHASE: IMPL]

Old head: 02711c97ff88c79d408e21f780d7c0b5530c8c2c. New head: 3dc8957b1160303c3f415c4cf8e01728552dd58a. Merged main: f59c185920edb40823d0695280c62772d01cb6d6. The delta is merge plus canonical regeneration only: the handwritten PR diff against main is byte-identical before and after, with no handwritten conflicts.

Local gates pass: structured check/lint/fmt for 443 files (including explicit CLI/docs-helper configurations), 118 focused tests with zero failures or skips (real Redis signal exits, generated hostname/ownership/crash-reload, service resources, OTel and public type closure), quality/architecture, carrier freshness, release:preflight and docs:accuracy. The exact owned Redis container and original volume were removed; labels, PR body and issue checkboxes are unchanged.

@rickylabs
rickylabs merged commit 20b7180 into main Nov 17, 2026
35 checks passed
@github-actions github-actions Bot added the canary:0.0.10-canary.4 Published NetScript prerelease 0.0.10-canary.4 label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:cli area:plugins plugins/* and plugin-core packages area:service packages/service canary:0.0.10-canary.4 Published NetScript prerelease 0.0.10-canary.4 priority:p1 High status:shipped type:fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[plugins] Child-health listener binds 0.0.0.0 [service] Processes do not exit after a drain, and nothing closes KV or Prisma

1 participant