Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
65fd947
docs(harness): scope worker pause drain and delivery-attempt contracts
rickylabs Nov 16, 2026
144011a
test(workers): reproduce pause, drain and shutdown delivery accounting
rickylabs Nov 16, 2026
89effd6
docs(harness): pin derived worker cleanup defaults
rickylabs Nov 16, 2026
7989476
docs(harness): retain approved worker lifecycle plan
rickylabs Nov 16, 2026
261ffcc
feat(workers): define bounded drain and fenced shutdown handback
rickylabs Nov 16, 2026
014730e
feat(queue): preserve native and polling shutdown handback
rickylabs Nov 16, 2026
031332f
docs(harness): retain listener drain ownership approval
rickylabs Nov 16, 2026
0fbed1b
feat(queue): allow bounded owners to retain admitted deliveries
rickylabs Nov 16, 2026
0483eee
fix(queue): confirm AMQP handback before acknowledging deliveries
rickylabs Nov 16, 2026
1802d73
feat(workers): pause intake and drain admitted work before abort
rickylabs Nov 16, 2026
2625555
test(workers): pin schema budgets and real runtime grace forwarding
rickylabs Nov 16, 2026
fefc5e3
feat(workers): configure cleanup budgets in standard runtime
rickylabs Nov 16, 2026
667ffa8
docs(workers): describe pause drain budgets and shutdown handback
rickylabs Nov 16, 2026
764271b
chore(workers): refresh published docs and API catalog
rickylabs Nov 16, 2026
2d7aa10
fix(workers): own listener startup before later registry waits
rickylabs Nov 17, 2026
33c034c
docs(harness): record worker final gates and runtime dispatch
rickylabs Nov 17, 2026
79f3582
docs(harness): map worker acceptance before independent evaluation
rickylabs Nov 17, 2026
9147e69
docs(harness): retain worker evaluation and exact resource cleanup
rickylabs Nov 17, 2026
4ed91e2
docs(harness): record worker runtime completion
rickylabs Nov 17, 2026
4450021
fix(workers): preserve the in-process runner startup log
rickylabs Nov 17, 2026
b5512b1
Merge origin/main into feat/worker-pause-drain
rickylabs Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .llm/harness/debt/arch-debt.md
Original file line number Diff line number Diff line change
Expand Up @@ -3206,7 +3206,11 @@ match the merged exemplars). IMPL-EVAL must not FAIL a slice for retaining eithe
Folding these into redis.adapter.ts would deepen its existing orchestration size and mix
independently reviewed failure boundaries. Issue #2351 adds one private admission/drain module
to keep the adapter below the 500-line gate while preserving independent receipt lifetimes.
Direct children now total 20.
Direct children now total 20. The #2635 pause/shutdown handback fix groups polling message/key
and receipt settlement into one private `_kv-polling/` concern directory (21 direct children).
This reduces the touched polling adapter from811 to727lines and enables CAS/fenced handback
without deepening its existing F-1 error. No new public export, dependency or scanner exemption.
The same cardinality closing gate applies; no further directory growth is accepted by analogy.
- **Owner:** NetScript queue maintainers.
- **Target:** Next queue adapter organization pass after the 0.0.9 recovery fix.
- **Linked plan:** .llm/runs/orch-divybot-6536806908--2395/plan.md; issue #2395.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Context pack

Issues2633/2635, PR2645, branch feat/worker-pause-drain, base b0eb079575968e843b781579673c9fe2ae250e07. Read plan.md/research.md/worklog.md/drift.md/supervisor.md and the two live issues in full. Plan-Gate required before product source changes. Four new tests are executable against unchanged main and include genuine red behavior plus failure controls. See research for exact results and excluded harness errors. Research resources cleaned: all five container identities including an earlier startup replacement, and five associated anonymous volumes proven by mount events. Current implementation retains four positively owned Redis/Garnet/Postgres/AMQP backends for implementation, with immutable IDs/mounts privately recorded; foreign resources untouched.

Primary paths: plugins/workers/worker/{worker,worker-options,listener-supervisor,queue-consumer,job-dispatcher,delivery-lease,job-runner-pool}.ts; plugins/workers/bin/runtime.ts; plugins/workers/src/cli/load-workers-config.ts; core config/workers-config.ts and runtime/in-process-job-runner.ts; existing core src/shutdown/ subpath. Queue ports/message-queue.ts, Redis lease/script, Postgres, KV polling, native Deno KV, AMQP, memory. Actual public types inspected with deno doc first.

Use Deno2.9.7 through mise. rtk/rg absent; plain commands. Scoped structured wrappers, --unstable-kv for checks. agentic matrix valid; current aliases maint:leak-check, maint:teardown, check:review-threads. Native Aspire guard rejects this local remote-Docker topology; full end-of-PR supported-runner smoke required, never bypass/bridge. User no helper agents; independent evaluator session/family separation mandatory. No merge, status:ready-merge, force push/rebase, publication, lock/cache deletion/reload, root fmt or saga transport/Streams edits.

Public artifacts/comments/bodies must contain no operator absolute paths, hostnames/ports, session/thread IDs or downstream names. Keep raw logs private, public evidence repo-relative. Evaluator writes plan-eval.md and phase comment, leaves artifact uncommitted. No source changes, commits, pushes, labels/readiness changes or PR watching by evaluator.

Current checkpoint: S2/S3 portable/native handback, S4 lifecycle/drain, S5 configuration/default forwarding and S6 public docs/generated consumers committed. Original and listener-owner addendum Plan-Gates approved. Real runtime/schema/runner12/12, full core112/0/1 existing ignore, full queue215/0/3 existing optional-backend ignores, original Worker/core-cause113/0/0; all seven adapters now have actual eight-Worker-replacement coverage. Final startup pause before/after allocation repair25/25 and corrected fixture13/13; final static/quality/corpus gates pass, public doc diagnostics unchanged. Final full plugin266/0/0green. Supported one-pass scaffold.runtime workflow43984548275 was dispatched once at source2d7aa10448700b46cb1c2dbb4665044eafb60dfa; it is still pending, not green evidence. Independent IMPL-EVAL, final acceptance/head checks/threads and cleanup remain. All four implementation backend containers and four recorded anonymous volumes are now cleaned with exact identity/mount proof; no survivors.

Completion phase comment6933353414. Draft status:impl-eval, all9acceptance boxes checked and both closing lines/mappings present; final DoD pending. Thread gate0/0. Evaluator PASS retained; all four owned implementation fixtures and their four volumes are now removed. Do not reuse stale connection receipts.

External first attempt: SQLite116+showcase8/static/native green; Postgres failed during documented-scaffold db generate dashboard refresh (same symptom baseline43936107382). Same-source failed-job-only retry requested, gate still mandatory. Independent verdict comment6933679178; product source unchanged.

Latest checkpoint: independent IMPL-EVAL PASS and supported workflow43984548275 attempt2 SUCCESS (Postgres124, SQLite116+showcase8, static/native/cleanup) at unchanged source2d7aa104. Four owned containers/four volumes gone. PR2645 ready for review; required non-draft head CI pending at transition, not yet a final green-head claim. No further implementation rework identified.

Ready head CI found a genuine accidentally omitted startup-log guard; exact main message restored, unchanged guard/intake/supervisor27/27 and source gates green. Only three console.log lines change after independent runtime PASS. Required new-head CI still mandatory; earlier head had17success/13skip/2failure.
21 changes: 21 additions & 0 deletions .llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Drift

None yet. Initial findings are code/docs spot-checks, not runtime reproduction. Existing owner-selected implementation session does its own planning under the user no-helper-agent instruction; separate independent evaluator sessions remain mandatory. T3 requires a fresh delegate task for each review/recheck, overriding same-evaluator-session steering in local policy; carry prior context into each brief.


Research corrections: task callbacks do not use activeJobs, so drain waits per-listener generation barriers rather than that map. Native Deno KV explicit nack is a no-op, requiring portable enqueue-before-ack handback. AMQP pinned provider defaults native retry off and double-acks async handlers; native upstream cancel/confirmed handback is necessary for the feature, with baseline transport failure distinguished from accounting evidence. Initial Postgres fixture schema race was a harness failure, corrected before recording product reproduction. The new Worker budget is25s total, with19s work/5s abort cleanup/1s settlement, preserving5s parent work headroom without stacking PR2641.


Plan-evaluator corrections: actual docs path tune-worker-runtime.md, not a directory index; exact generated carriers named. A committed claim racing pause is an ordinary charged requeue; draining handback preserves the attempt. The latter follows the already locked shutdown-only opt-in and is pinned by an S4 race test.


S3a stays within approved adapter scope. Its handback storage-failure test exposed an unjoined callback rejection; red fault test precedes the error-propagation fix, which keeps the original processing entry durable and fails the listener. Native listener post-intake expiry can resolve a pause generation before handlers finish;3real/scaled owner cases red with3default controls green. That rework-forcing contract discovery is stopped at a separate plan checkpoint, with exact proposal/files/gates in plan-intake-drain-addendum.md. No new-option implementation yet.


Follow-up PLAN-EVAL cycle2 settlement: addendum risk text now pins the shared name with the already approved Worker-total stop option (Listener JSDoc/S6 docs must cross-reference the per-listener post-intake cutoff against the unchanged25s Worker total). Wording only, settled in place under the cycle2 authorization; no product/tests or design change. Original Plan-Gate PASS history preserved; follow-up verdict PASS appended to plan-eval.md at014730e1.


S4 test contract alignment: older immediate-abort fixtures now assert the work window before ShutdownError. The held progress sink permits only the documented bounded TimeoutError result in the shutdown case; cause, once-only state/cleanup and claim-release assertions are retained. Relevant lifecycle fixtures moved into their own small file, reducing the pre-existing oversized dispatch test. Redis delivery fixture uses500ms total to include the native adapter's unchanged100ms disconnect grace, retaining all accounting assertions. Core retry catch formerly replaced an observed failure with a later abort; red first-cause cases precede its narrow preservation fix. No budget or acceptance contract changed.


Final-gate scope repairs: actual Worker replacement coverage extended to the five remaining adapters (test-only supplement,175lines). Native AMQP lifecycle made its old Fedify-only abort fixture obsolete; replaced its upstream channel model and added cleanup-order assertions. Full plugin validation exposed a pre-existing seed-fixture stale mkdir-event race; require current directory existence without changing the watcher. The approved startup/pause risk received a red held-registry test and immediate per-listener intake-state application, fixing the gap while keeping Worker<500. No architecture/budget re-plan, no unrelated production change.
Loading
Loading