From 65fd9470a027d16a5aa6ee0dff32e1198dd23147 Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Mon, 16 Nov 2026 22:18:07 +0100 Subject: [PATCH 01/20] docs(harness): scope worker pause drain and delivery-attempt contracts --- .../context-pack.md | 3 ++ .../drift.md | 3 ++ .../plan.md | 30 +++++++++++++++++++ .../research.md | 7 +++++ .../supervisor.md | 3 ++ .../worklog.md | 7 +++++ 6 files changed, 53 insertions(+) create mode 100644 .llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md create mode 100644 .llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md create mode 100644 .llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md create mode 100644 .llm/runs/feat-worker-pause-drain--issues-2633-2635/research.md create mode 100644 .llm/runs/feat-worker-pause-drain--issues-2633-2635/supervisor.md create mode 100644 .llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md new file mode 100644 index 0000000000..71f4557f42 --- /dev/null +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md @@ -0,0 +1,3 @@ +# Context pack + +Combined issues2633+2635, branch feat/worker-pause-drain from fresh main b0eb07957. Bootstrap research only; source untouched. Need reproduce all required behavior before repair, lock named worker/intake/execution lifecycles and queue refund contract, feature PLAN-EVAL before production. Core runner default1s, Worker.stop immediately aborts jobs before its later30s wait, Worker.start lifetime tied to listener completion. Queue API context has deliveryCount/nack; exact backend inventory/config forwarding next. Full CLI smoke at end for generated runtime wiring. No stacks/Streams/saga transports/merge/publication. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md new file mode 100644 index 0000000000..0cbbfc9083 --- /dev/null +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md @@ -0,0 +1,3 @@ +# Drift + +None yet. Initial findings are code/docs spot-checks, not runtime reproduction. Existing owner-selected implementation session does its own planning under the user no-helper-agent instruction; separate independent evaluator sessions remain mandatory. T3 requires a fresh delegate task for each review/recheck, overriding same-evaluator-session steering in local policy; carry prior context into each brief. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md new file mode 100644 index 0000000000..c5f0dbd09c --- /dev/null +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md @@ -0,0 +1,30 @@ +# Plan under research + +## SKILL + +netscript-doctrine, netscript-harness, netscript-pr, netscript-tools, netscript-deno-toolchain, netscript-cli, jsr-audit. + +## Required public behavior + +Worker pause/resume stop/restart intake while admitted jobs continue. A drain stops intake, waits for admitted work within a configurable budget, then aborts leftovers and awaits bounded handler cleanup. Grace is configurable through worker config and forwarded by standard/generated runtimes, with a default derived/documented against drain ownership. Shutdown handback refunds the current delivery attempt on every adapter that tracks attempts; actual handler failures still charge. Retain fencing/idempotency and durable DLQ semantics. Docs explain all fields/defaults/semantics. No Redis Streams or saga transport edits. + +## Must resolve before PLAN-EVAL + +- Exact listener/worker lifetime split: pause cannot settle start(), close queues or destroy runner; resume cannot exceed configured admission or mask listener failures. +- Signal scope: stop intake without aborting jobs/tasks/lease heartbeats; later abort only after the work budget. +- One total/default budget and runner cleanup grace, config schema/types/JSDoc, standard and generated forwarding, no dependency on the separate shutdown PR. +- Inventory every delivery-count backend and its atomic/fenced refund boundary; distinguish worker shutdown from handler failure/caller cancel/deadline; retain default nack behavior for existing consumers. +- Real failing tests for pause/no claims, within-budget ack, repeated shutdown beyond maxDeliveries, failure charging and >1second cleanup through configured runner path. +- Concrete ordered file groups<30each and package/public-surface gate set, debt/JSR risks. + +## Intended slices + +S1 research/reproductions/locked contracts and separate feature PLAN-EVAL; run artifacts plus focused tests, no production. S2 queue contract then adapter implementations and refund tests. S3 worker lifecycle/budget contract then implementation, pool/config/generated forwarding and tests. S4 docs/generated consumers, static/full runtime validation, final independent evaluation and ready/head/thread gates. Refine exact file groups after inventory. One direct-to-main PR, no stacking with runtime shutdown PR. + +## Validation selection + +Archetype3 runtime/behavior folding queue integration2 and plugin5; all required static, fitness, backend lifecycle and consumer gates. Structured check/test/lint/fmt, quality:gate, doc lint/baseline and JSR audits/dry-run, release:preflight/docs:accuracy, generated assets freshness. Real backend cases for changed accounting adapters, preserve existing fencing/expiry/renewal/outage tests. Full scaffold.runtime once at the end because standard/generated worker runtime wiring changes; supported runner if local Docker topology is unsupported. All containers/resources fenced by positive ownership; no foreign cleanup. + +## Deferred scope + +No saga feed/transport/Streams work, no generic lifecycle registry, no unrelated dependency or cache/lock changes. Resolve routine choices autonomously and record them; independent plan evaluation is mandatory before implementation. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/research.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/research.md new file mode 100644 index 0000000000..27df9f8246 --- /dev/null +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/research.md @@ -0,0 +1,7 @@ +# Research + +Both issues read in full. All nine acceptance boxes are the contract. Repository guidance/doctrine/runtime/CLI skills apply. Public Worker docs expose start/cancel/stop but no pause/resume. Current stop aborts its shared controller and all active executions immediately, then calls workerPool.shutdown and listener.stop before the existing30second waitForActiveJobs. That later wait cannot provide the requested graceful work interval. Start currently awaits the job-listener completion, so intentionally pausing listeners must not accidentally end the worker's process-owned start lifetime. + +Core README confirms runner abortGracePeriodMs default1000milliseconds. Standard config/runtime forwarding needs research. Queue public MessageContext exposes deliveryCount/ack/nack/extendVisibility; ListenOptions.signal stops listening and concurrency bounds admission. Published providers Redis/RabbitMQ/DenoKV/Postgres plus memory/KV-polling implementations need an exact accounting inventory before locking shutdown refund behavior. No running reproduction claimed yet. + +MCP guidance points to tune-worker-runtime and direct retry/DLQ docs; public API inspected with deno doc, then focused Worker source. Main unchanged. Next inspect queue adapter settlement and worker contexts/pool/supervisor/config, write failing functional oracles, then lock the feature plan. No source implementation before selected Plan-Gate PASS. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/supervisor.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/supervisor.md new file mode 100644 index 0000000000..1618e0579a --- /dev/null +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/supervisor.md @@ -0,0 +1,3 @@ +# Implementation lane + +Issues2633/2635 share branch feat/worker-pause-drain because abort grace and drain ownership overlap. Fresh origin/main b0eb079575968e843b781579673c9fe2ae250e07 confirmed before creating this worktree. Owner-requested implementation lane Codex gpt-6.1-sol/xhigh owns research/design/implementation; no helper/planning agents. Feature tier, separate GLM5.3/provider-default PLAN-EVAL before production, MuseSpark1.3/xhigh final IMPL-EVAL (matrix lookup before dispatch), different session/vendor family. No merge, ready-merge, rebase/force-push or publication. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md new file mode 100644 index 0000000000..32a6a24063 --- /dev/null +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -0,0 +1,7 @@ +# Worklog + +## Design + +Research is active; required behavior and must-resolve lifecycle/budget/accounting choices are in plan.md. Existing Worker lifetime currently conflates listener cancellation and active job abort; queue/runner ownership boundaries must be named before a repair. Feature Plan-Gate selected. No production changes or reproduction claimed. + +Fresh main b0eb07957 fetched before branch. Both full issues and relevant public docs/APIs read. Initial source spot-check confirms missing pause/resume, immediate abort at stop and ineffective later30second work wait. Core runner docs confirm1second grace. Next complete backend inventory and functional red tests. From 144011af3850ac77727694446abb7f83e3aec78d Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Mon, 16 Nov 2026 22:59:58 +0100 Subject: [PATCH 02/20] test(workers): reproduce pause, drain and shutdown delivery accounting --- .../context-pack.md | 8 +- .../drift.md | 3 + .../leak-report.md | 3 + .../plan.md | 65 ++++++-- .../research.md | 49 +++++- .../supervisor.md | 2 + .../worklog.md | 9 + .../runtime/abort-cleanup-budget_test.ts | 41 +++++ packages/queue/tests/shutdown-requeue_test.ts | 154 ++++++++++++++++++ .../workers/worker/worker-pause-drain_test.ts | 98 +++++++++++ .../worker/worker-shutdown-deliveries_test.ts | 89 ++++++++++ 11 files changed, 498 insertions(+), 23 deletions(-) create mode 100644 .llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md create mode 100644 packages/plugin-workers-core/tests/runtime/abort-cleanup-budget_test.ts create mode 100644 packages/queue/tests/shutdown-requeue_test.ts create mode 100644 plugins/workers/worker/worker-pause-drain_test.ts create mode 100644 plugins/workers/worker/worker-shutdown-deliveries_test.ts diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md index 71f4557f42..5ebc08e867 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md @@ -1,3 +1,9 @@ # Context pack -Combined issues2633+2635, branch feat/worker-pause-drain from fresh main b0eb07957. Bootstrap research only; source untouched. Need reproduce all required behavior before repair, lock named worker/intake/execution lifecycles and queue refund contract, feature PLAN-EVAL before production. Core runner default1s, Worker.stop immediately aborts jobs before its later30s wait, Worker.start lifetime tied to listener completion. Queue API context has deliveryCount/nack; exact backend inventory/config forwarding next. Full CLI smoke at end for generated runtime wiring. No stacks/Streams/saga transports/merge/publication. +Issues2633/2635, PR2645, branch feat/worker-pause-drain, base b0eb079575968e843b781579673c9fe2ae250e07. Read plan.md/research.md/worklog.md/drift.md/supervisor.md and the two live issues in full. Plan-Gate required before product source changes. Four new tests are executable against unchanged main and include genuine red behavior plus failure controls. See research for exact results and excluded harness errors. No current run-owned resources: all five container identities including an earlier startup replacement, and five associated anonymous volumes proven by mount events, cleaned; foreign resources untouched. + +Primary paths: plugins/workers/worker/{worker,worker-options,listener-supervisor,queue-consumer,job-dispatcher,delivery-lease,job-runner-pool}.ts; plugins/workers/bin/runtime.ts; plugins/workers/src/cli/load-workers-config.ts; core config/workers-config.ts and runtime/in-process-job-runner.ts; existing core src/shutdown/ subpath. Queue ports/message-queue.ts, Redis lease/script, Postgres, KV polling, native Deno KV, AMQP, memory. Actual public types inspected with deno doc first. + +Use Deno2.9.7 through mise. rtk/rg absent; plain commands. Scoped structured wrappers, --unstable-kv for checks. agentic matrix valid; current aliases maint:leak-check, maint:teardown, check:review-threads. Native Aspire guard rejects this local remote-Docker topology; full end-of-PR supported-runner smoke required, never bypass/bridge. User no helper agents; independent evaluator session/family separation mandatory. No merge, status:ready-merge, force push/rebase, publication, lock/cache deletion/reload, root fmt or saga transport/Streams edits. + +Public artifacts/comments/bodies must contain no operator absolute paths, hostnames/ports, session/thread IDs or downstream names. Keep raw logs private, public evidence repo-relative. Evaluator writes plan-eval.md and phase comment, leaves artifact uncommitted. No source changes, commits, pushes, labels/readiness changes or PR watching by evaluator. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md index 0cbbfc9083..43b83fcbe7 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md @@ -1,3 +1,6 @@ # Drift None yet. Initial findings are code/docs spot-checks, not runtime reproduction. Existing owner-selected implementation session does its own planning under the user no-helper-agent instruction; separate independent evaluator sessions remain mandatory. T3 requires a fresh delegate task for each review/recheck, overriding same-evaluator-session steering in local policy; carry prior context into each brief. + + +Research corrections: task callbacks do not use activeJobs, so drain waits per-listener generation barriers rather than that map. Native Deno KV explicit nack is a no-op, requiring portable enqueue-before-ack handback. AMQP pinned provider defaults native retry off and double-acks async handlers; native upstream cancel/confirmed handback is necessary for the feature, with baseline transport failure distinguished from accounting evidence. Initial Postgres fixture schema race was a harness failure, corrected before recording product reproduction. The new Worker budget is25s total, with19s work/5s abort cleanup/1s settlement, preserving5s parent work headroom without stacking PR2641. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md new file mode 100644 index 0000000000..e1e1a45928 --- /dev/null +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md @@ -0,0 +1,3 @@ +# Research resource hygiene + +Commands: maint:leak-check and maint:teardown preview against this run and worktree. Reporter raw output retained privately. Four active research backend containers (plus one earlier owned Garnet startup replacement) positively identified by recorded immutable IDs and matching test-owner label; all removed by exact ID after preview. Subsequent scan of that exact owner set: zero containers. All five anonymous volumes removed only after native mount events proved association with the five run-created container identities and a reference scan proved no other container used them. The earlier Garnet startup replacement was included. No owned volume remains. Foreign/unknown-owner resources untouched. No process or container pattern kill. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md index c5f0dbd09c..5a765b70de 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md @@ -1,30 +1,59 @@ -# Plan under research +# Locked plan: worker intake, bounded drain, and shutdown handback -## SKILL +## SKILL and baseline -netscript-doctrine, netscript-harness, netscript-pr, netscript-tools, netscript-deno-toolchain, netscript-cli, jsr-audit. +netscript-doctrine, netscript-harness, netscript-pr, netscript-tools, netscript-deno-toolchain, netscript-cli, jsr-audit. Archetype3 runtime/behavior, folding queue integration2 and plugin5. Feature tier. User-selected Sol implementation lane owns its own research/design and implementation; only independent evaluators delegated. Both issues read in full. Fresh fetch/main b0eb079575968e843b781579673c9fe2ae250e07 before branch creation; no product changes before Plan-Gate PASS. PR2645 targets main directly, independent of PR2641. No Redis Streams or saga transport edits. -## Required public behavior +## Decisions -Worker pause/resume stop/restart intake while admitted jobs continue. A drain stops intake, waits for admitted work within a configurable budget, then aborts leftovers and awaits bounded handler cleanup. Grace is configurable through worker config and forwarded by standard/generated runtimes, with a default derived/documented against drain ownership. Shutdown handback refunds the current delivery attempt on every adapter that tracks attempts; actual handler failures still charge. Retain fencing/idempotency and durable DLQ semantics. Docs explain all fields/defaults/semantics. No Redis Streams or saga transport edits. +1. **Lifecycle and intake.** Core-owned named WorkerLifecycleState is idle/running/paused/draining/stopped. Worker exposes pause()/resume(), isPaused, and its health snapshot lifecycle. Pause requests intake cancellation synchronously, leaves all admitted jobs/tasks/trigger jobs and lease renewal running, and keeps start() pending. Each listener supervisor owns a separate attempt AbortController and a pause barrier. Its long-lived run loop parks after the old queue.listen generation drains; resume starts the next generation only after that barrier, preserving concurrency and queue reuse. A committed claim racing pause/drain is handed back without running a new user handler. Pause is intentional healthy idleness, not a crash/restart: preserve restart counters and failure history; failed supervisors cannot be revived by resume. Worker.stop is idempotent, concurrent callers share completion, stop is terminal for that Worker instance; replacement uses a new instance as the runner is already terminal on current main. Pause before start/after stop and resume while draining/stopped are harmless no-ops; repeated pause/resume are idempotent. No queue closes and no runner shutdown during pause. -## Must resolve before PLAN-EVAL +2. **Two-phase bounded stop.** drainTimeoutMs is the TOTAL Worker.stop budget, default25,000ms. abortGracePeriodMs defaults5,000ms (one fifth of the default drain budget), with explicit finite nonnegative safe-integer validation; total drain must be positive and grace cannot exceed it. Reserve min(1,000ms, one twenty-fifth of total) for queue/listener settlement and close. Work may finish without cancellation until total minus grace minus that reserve (default19s); abort leftovers only then, await runner cleanup/listener generation barriers within the remaining grace, then stop owned queues within the reserve. Use one monotonic deadline, never restart each timeout. Work that completes in the first phase follows normal success/idempotency/ack. Task listeners are included through their generation barriers, including executor and registry/state work before any activeJobs entry. Cancellation/individual deadlines remain immediate with the configured grace. Core runner standalone default5s is documented against this25s worker total and the queue's30s listener drain; the process-level parent budget has at least5s headroom. Larger configured worker budgets require a correspondingly larger process drain budget and are documented. A hung executor/storage/close makes stop reject TimeoutError within the total bound; in-process JavaScript cannot be physically killed and late results cannot mark success after abort. Existing executable parent shutdown owns eventual process termination. -- Exact listener/worker lifetime split: pause cannot settle start(), close queues or destroy runner; resume cannot exceed configured admission or mask listener failures. -- Signal scope: stop intake without aborting jobs/tasks/lease heartbeats; later abort only after the work budget. -- One total/default budget and runner cleanup grace, config schema/types/JSDoc, standard and generated forwarding, no dependency on the separate shutdown PR. -- Inventory every delivery-count backend and its atomic/fenced refund boundary; distinguish worker shutdown from handler failure/caller cancel/deadline; retain default nack behavior for existing consumers. -- Real failing tests for pause/no claims, within-budget ack, repeated shutdown beyond maxDeliveries, failure charging and >1second cleanup through configured runner path. -- Concrete ordered file groups<30each and package/public-surface gate set, debt/JSR risks. +3. **Clock contract.** The core-owned WorkerDrainClock port has monotonic now() and disposable schedule(delayMs,callback), structurally reusing the scheduler method from JobRunnerClock without changing that port's epoch-time meaning. Web Platform default uses performance.now and owned/cleared setTimeout. Worker accepts this clock as an optional direct-composition seam; standard config stays data-only. Core owns the duration/default resolver and bounded drain lifecycle primitives; the plugin wires listeners, runner and queues. Use the existing core ./shutdown public subpath for the new lifecycle/budget primitives and types, avoiding extra runtime/root barrel growth. The existing generic ShutdownManager lacks the two-phase work/abort/settlement contract and is left unchanged. No generic lifecycle registry or dependency added. Tests use manual advancement for boundary/race cases, plus real-time cleanup>1s and actual broker oracles. -## Intended slices +4. **Configuration and runtime.** Add drainTimeoutMs/abortGracePeriodMs to WorkersConfigData, authoring input, shared schema and direct WorkerOptions. Defaults come from the same core policy. Load the authored workers section through the existing loadConfig plus WorkersConfigSchema path before runtime dependencies. Move the existing CLI loadWorkersConfig implementation into src/runtime/workers-config.ts and retain the CLI file as an identity forwarding shim; bin/runtime.ts must not import a CLI implementation. startWorkerProcess and startCombinedProcess both accept explicit budget overrides, otherwise use validated project config, otherwise defaults. Forward grace through Worker into the actual WorkerPool/InProcessJobRunner even when no static registry exists. Explicit direct WorkerOptions.abortGracePeriodMs wins over the legacy workerPoolOptions value; legacy pool-only customization remains supported when the top-level value is absent and is validated against total budget. Generated glue delegates to the same startCombinedProcess and requires no second resolver. A subprocess using a real project config and real handler proves standard startup/forwarding, not a string search or fake runner. -S1 research/reproductions/locked contracts and separate feature PLAN-EVAL; run artifacts plus focused tests, no production. S2 queue contract then adapter implementations and refund tests. S3 worker lifecycle/budget contract then implementation, pool/config/generated forwarding and tests. S4 docs/generated consumers, static/full runtime validation, final independent evaluation and ready/head/thread gates. Refine exact file groups after inventory. One direct-to-main PR, no stacking with runtime shutdown PR. +5. **Queue handback contract.** Add optional NackOptions.preserveDeliveryCount, false by default and effective only for requeue:true. It refunds exactly this live delivery, never arbitrary historical attempts; next delivery sees the same count. It bypasses max-deliveries exhaustion and failure backoff for this handback only, retaining message identity, original enqueue time, payload, headers, queue and ordering/delay metadata. requeue:false ignores preservation and retains terminal policy. Existing ordinary explicit/implicit failures and crash/expiry reclaim still charge. Duplicate and stale settlement must not refund a newer claim or enqueue duplicate copies. Worker requests preservation only for its controlled shutdown cause (first cause ShutdownError), never merely because a stop happened concurrently with a business error, timeout, lease loss, duplicate claim, state/DLQ outage or operator cancel. Preserve existing error propagation/retry behavior for those paths. -## Validation selection +6. **Adapters.** Redis/Garnet: existing atomic receipt Lua transition fences ownership/expiry, selects refunded encoded envelope only for explicit handback, removes the receipt once, and never changes expiry/reclaim/DLQ/outbox behavior. No new commands; declare all existing script keys. Postgres: one fenced UPDATE decrements the current delivery and unlocks it atomically before any max-retry decision. Generate an opaque per-claim locked_by receipt (existing column) and carry it through context settlement; guard against expired/replaced receipt and repeated local calls. Normal requeue remains counted. KV polling: extract internal message/key and settlement concerns into _kv-polling/message.ts and settlement.ts, keeping the currently811-line adapter below800 and not deepening the warning unnecessarily. CAS checks the stored processing entry and a fresh per-claim identity, then atomically moves to pending with attempts minus1 for handback; preserve failure delay/count/DLQ behavior. Successful explicit settlement must suppress the existing unconditional automatic ack/nack. Recovery retains its charged attempt and must not settle a replacement. Prefer existing kv.atomic; retain and document the existing single-consumer best-effort fallback when a caller supplies a KV without atomic support (no stronger concurrency guarantee is claimed there). Memory: per-delivery settlement guard, refund once, pending identity stable. Deno KV: explicit requeue and a thrown handler failure must actually enqueue the same addressed envelope before returning so the native dispatcher can acknowledge the old delivery; count is previous/current according to preservation. An explicit settlement prevents another automatic settlement; propagate failed persistence to the native retry path without acknowledging. Current nack is a no-op, so remove the worker's extendVisibility requirement only once portable handback works. A transport failure before the retry metadata can be committed retains the native unacknowledged recovery path; ordinary user-handler failures use the counted enqueue-before-ack path. Document crash/outage limitations honestly. AMQP: existing wrapper leaves Fedify native retry disabled (upstream double-acks Promise handlers) and does not pass queueName. Set native retry and queue name for publishing, and consume through the existing upstream amqplib channel so intake cancellation uses basic.cancel while admitted handlers settle before channel close. Use native prefetch for concurrency and one terminal ack/nack path, with same-envelope explicit requeue/count preservation. On handler failure publish counted retry before acknowledgement; a publish failure retains native unacked/redelivery behavior. No new transport abstraction or dependency; stop closes only the adapter-owned channel/connection. This is necessary for portable worker pause/handback, not a new AMQP feature set. Durability remains at-least-once. The pinned Fedify publisher does not wait for broker confirmation, so handback/counting retries use the already installed amqplib createConfirmChannel/sendToQueue callback and acknowledge the old delivery only after broker confirmation. Publish failure nacks the old live delivery; disconnect closes/requeues the unacked claim. Keep the existing public delayed-enqueue path; no new publisher abstraction. -Archetype3 runtime/behavior folding queue integration2 and plugin5; all required static, fitness, backend lifecycle and consumer gates. Structured check/test/lint/fmt, quality:gate, doc lint/baseline and JSR audits/dry-run, release:preflight/docs:accuracy, generated assets freshness. Real backend cases for changed accounting adapters, preserve existing fencing/expiry/renewal/outage tests. Full scaffold.runtime once at the end because standard/generated worker runtime wiring changes; supported runner if local Docker topology is unsupported. All containers/resources fenced by positive ownership; no foreign cleanup. +## Reproductions before implementation -## Deferred scope +See research.md and worklog.md. Initial combined run:2pass/8fail/6unconfigured-backend ignores; failures include config dropped, default dispatch settling before1.25s cleanup, missing pause, immediate drain abort, memory/KV-poll counts1->2, Deno KV explicit nack losing the handback. Live Redis+Garnet each preserve test red(count2), each ordinary failure control green. Corrected real Postgres test red(count2), ordinary failure control green; initial concurrent enqueue/schema harness race excluded. Real Worker instances on Redis and Garnet both red on first replacement(count2). AMQP cannot reach accounting oracle on main: upstream async double-ack yields broker406 unknown delivery tag; native options/cancel/settlement repair is prerequisite, with no false accounting claim. Test eight clean restarts beyond maxDeliveries5 then successful completion, all adapters, failure charging, final queue/DLQ depths and identity. -No saga feed/transport/Streams work, no generic lifecycle registry, no unrelated dependency or cache/lock changes. Resolve routine choices autonomously and record them; independent plan evaluation is mandatory before implementation. +## Acceptance evidence to produce + +2633 boxes1/4: schema/type config and real standard/generated runtime forwarding plus docs. Box2: default policy/JSDoc exact budget arithmetic and validation tests. Box3: real handler cleanup1.25s awaited with default/configured allowed grace; cancellation still fails closed. +2635 box1: pause no intake, admitted job/task/trigger completion, pending start lifetime, resumed intake, repeated/racing calls and concurrency ceiling. Box2: within-budget success+ack plus work/grace/total boundary/noncooperative tests. Box3: real Worker replacement8times on Redis/Garnet plus adapter-contract8restarts on memory, KV polling, Deno KV, Postgres, AMQP, all retaining metadata and empty final ready/processing/DLQ. Box4: ordinary nack/actual handler throw counting and terminal max-delivery controls, including stale/double refund race tests. Box5: workers README, core README and tune-worker-runtime docs. + +## Ordered slices (every slice includes scoped run evidence; all file groups below30) + +S1 research/red tests/plan only (11files):4 test files abort-cleanup-budget_test.ts, shutdown-requeue_test.ts, worker-pause-drain_test.ts, worker-shutdown-deliveries_test.ts;7 scoped run docs. Proves current-main failures and explicit design; gate genuine runtime reports + independent PLAN-EVAL PASS before production. No product contract implementation yet. +S2 core/queue contracts and lease backends (<=18files): core src/shutdown/worker-lifecycle.ts, src/shutdown/worker-drain.ts, src/shutdown/mod.ts; queue ports/message-queue.ts, _redis-lease.ts, _redis-lease-script.ts, postgres.adapter.ts; queue tests/shutdown-requeue_test.ts plus focused shutdown-refund-races_test.ts; core tests/runtime/worker-drain-budget_test.ts; up to7run artifacts. Contract first then implementations. Proves defaults, total bounds and atomic receipt refunds on real Redis/Garnet/Postgres; existing lease fencing/outage/DLQ tests retained. +S3 remaining adapters (<=16files): kv-polling.adapter.ts, _kv-polling/message.ts, _kv-polling/settlement.ts, deno-kv.adapter.ts, amqp.adapter.ts, testing/memory-queue.ts; queue shutdown-requeue_test.ts and shutdown-refund-races_test.ts plus focused polling/AMQP tests; up to7run artifacts. Proves all counting backends hand back without count, failure still charges, no duplicate/stale refund, native resources close. Scoped check/test/lint/fmt and backend suites before push. +S4 worker supervisor and drain wiring (<=18files): worker/mod.ts, worker.ts, worker-options.ts, listener-supervisor.ts, queue-consumer.ts, job-dispatcher.ts, job-runner-pool.ts; worker-pause-drain_test.ts, worker-shutdown-deliveries_test.ts, listener-supervisor_test.ts plus focused worker-drain-budget_test.ts; up to7run artifacts. Core named lifecycle/clock contract reused; keep Worker under500 by moving bounded-budget orchestration into core-owned worker-drain primitive. Proves pause/resume start lifetime, task/trigger health/concurrency, successful drain ack, later abort and real8replacement delivery oracle. Adjacent worker cancellation/recovery/lease/idempotency tests retained. +S5 config/standard runtime (<=17files): core config/workers-config.ts, runtime/in-process-job-runner.ts and options JSDoc if needed; core config/runtime tests abort-cleanup-budget_test.ts and job-cancellation_test.ts; plugin src/runtime/workers-config.ts, src/cli/load-workers-config.ts, bin/runtime.ts; runtime budget forwarding subprocess test and private fixture in tests/runtime; up to7run artifacts. Proves schema defaults/validation and actual startup, including no-registry and custom-registry paths; runner cleanup>1s executed. No CLI->runtime layering reversal. +S6 docs/consumer freshness (<=15files): core README, plugin README, docs/site/background-processing/how-to/tune-worker-runtime/index.md; generated docs-prose.generated.ts, export-surface.generated.ts, CLI assets/publish carrier if changed; up to7run artifacts. Proves public option defaults, pause/drain/count/durability/ownership limits; canonical prose/corpus/assets refresh + docs accuracy/preflight/publish freshness. +S7 final gates/evaluation (<=7run files): all required local static/fitness/publication gates, full scaffold.runtime one pass on supported runtime, resource cleanup, separate MuseSpark implementation evaluation, acceptance checkboxes/body mappings, ready transition and current head required CI/thread gate. No merge/ready-merge/publication. + +## Risk register and mitigations + +- Queue stop can wait on handlers while pause must return: attempt pause barrier is separate from supervisor lifetime; public pause requests cancellation without joining admitted handlers. Resume waits old generation before listening again. Test no over-admission and start lifetime. +- Task dispatch lacks activeJobs bookkeeping: generation barriers include whole task callback, so work drain covers tasks without introducing a fake job map entry. +- Concurrent initial startup/stop/pause: own controllers/state before awaits, drain allocations exactly once; tests stop during initialization and queued last claim. Start errors must release owned resources; no swallowed fatal listener errors. +- Shutdown attribution race: first cause and typed reason gate preservation; negative business failure/deadline/cancel/lease-loss controls. Idempotency key remains stable because refunded delivery count stays stable; release interrupted claim before replacement processing. +- Atomic refund abuse/races: opt-in only, fenced receipt/claim token, local settlement once, atomic count+move, duplicates/stale receipts no effect; preserve terminal nacks. CAS/publish failures must leave durable ownership/retry, not report successful handback. +- AMQP baseline is broken before oracle: native upstream channel lifecycle and publisher confirmation inspected/tested with owned RabbitMQ, include failure/deadline/disconnect/pause cases; no claim that raw main passes accounting. Keep change within handback/intake prerequisite. +- Default inner/outer timing:25s total leaves5s within main child30s work; future combined runtime PR35total/30work keeps same headroom. Work19s, cleanup5s, settle1s. One monotonic bound. Customization docs require parentbudget >= worker total + external cleanup. +- F-1/F-16: baseline Worker463, KV polling811, Postgres581. Extract affected polling settlement (no new module at adapters root except one cohesive _kv-polling directory); reduce big touched files, no broad unrelated refactor or numeric exemption. Existing adapters cardinality debt records the one new concern directory with rationale and closing gate. Test files<480. Existing public type closure debt not deepened. +- JSR/slow types: explicit owned interface/data types and return annotations; no exported Zod implementation, broad any/casts, new dependency, lock change, or root/sub-barrel mirrors. Runtime/core additions exported only from appropriate existing subpath, no new package export. Run doc-lint all affected subpaths and compare existing baseline diagnoses, publish dry-run/audit and consumer type contracts; any new diagnostic is a failure. Core/plugin flat surface baseline recorded as debt if encountered, no waiver by analogy. +- Expensive runtime resources: full scaffold smoke only at end; local remote Docker is unsupported by native Aspire guard, so use supported runner rather than bypass/bridge. Positive ownership IDs for backend containers, read-only leak report and teardown preview before cleanup; foreign resources untouched. + +## Gates selected + +Archetype3 all F1-F19 plus integration/plugin overlays. Structured scoped source check(--unstable-kv), lint/fmt and test wrappers; manual/small public consumer/runtime-state/permissions/layering review for rules without script. quality:gate and arch:check; doc-lint changed entrypoints and all public plugin entries with baseline comparison; JSR audit and package dry-runs for queue/core/plugin. No root mutating fmt, lock reload or test weakening. release:preflight and docs:accuracy locally for source/docs changes; corpus/publish assets freshness. Full worker/core/queue suites + real Redis/Garnet/Postgres/AMQP contracts and generated-project full runtime suite. Independent PLAN-EVAL then IMPL-EVAL, no unanswered review threads and required head checks green/policy-skipped before ready completion. + +## Open decisions / deferred scope + +No must-resolve design choice remains. Exact private fixture/test filenames may change without altering contracts (safe to defer). Crash/outage recovery before a counted handback can be committed remains provider-limited and explicitly documented; ordinary handler failures must count on all adapters. General queue transaction/outbox redesign, broad Postgres/DenoKV transport changes, process lifecycle fixes owned by PR2641, saga SSE2637, all saga transports/Streams, dependency bumps and unrelated architecture migration are deferred. Any discovered contract/budget/fencing divergence is recorded and evaluated before implementing that change. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/research.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/research.md index 27df9f8246..9a2536013d 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/research.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/research.md @@ -1,7 +1,48 @@ -# Research +# Research at main b0eb079575968e843b781579673c9fe2ae250e07 -Both issues read in full. All nine acceptance boxes are the contract. Repository guidance/doctrine/runtime/CLI skills apply. Public Worker docs expose start/cancel/stop but no pause/resume. Current stop aborts its shared controller and all active executions immediately, then calls workerPool.shutdown and listener.stop before the existing30second waitForActiveJobs. That later wait cannot provide the requested graceful work interval. Start currently awaits the job-listener completion, so intentionally pausing listeners must not accidentally end the worker's process-owned start lifetime. +Both issues2633/2635 read in full; all nine acceptance boxes map to plan.md. Source remains unchanged from main; only tests/run docs are new. Public API inspected with deno doc, MCP guidance consulted, then focused source and pinned upstream implementations. Relevant doctrine3/runtime/plugin profiles, lifecycle/clock/failure rules, gate matrix and JSR rubric applied. -Core README confirms runner abortGracePeriodMs default1000milliseconds. Standard config/runtime forwarding needs research. Queue public MessageContext exposes deliveryCount/ack/nack/extendVisibility; ListenOptions.signal stops listening and concurrency bounds admission. Published providers Redis/RabbitMQ/DenoKV/Postgres plus memory/KV-polling implementations need an exact accounting inventory before locking shutdown refund behavior. No running reproduction claimed yet. +## Findings verified in code -MCP guidance points to tune-worker-runtime and direct retry/DLQ docs; public API inspected with deno doc, then focused Worker source. Main unchanged. Next inspect queue adapter settlement and worker contexts/pool/supervisor/config, write failing functional oracles, then lock the feature plan. No source implementation before selected Plan-Gate PASS. +Worker.start waits the job-supervisor lifetime. stop immediately aborts its shared execution/intake controller and active job controllers, awaits runner shutdown/listeners, then reaches a30s poll that cannot provide a graceful work interval. Tasks do not enter activeJobs. Each queue listen signal cancels admission while its callback/settlement lifetime may continue. Supervisor attempt pause barriers can preserve long-lived start and cap generation concurrency without creating duplicate trigger queues or shutting down the terminal runner. Lease renewal belongs to dispatch shutdown signal, not intake. + +Core runner default1000ms applies after all abort causes; explicit grace already works but no workers config field or standard runtime forwarding exists. Both runtime constructors pass only a static registry to WorkerPool. Existing CLI loadWorkersConfig wraps native loadConfig + shared WorkersConfigSchema; move this loader to runtime and keep the CLI forwarding identity, avoiding a CLI dependency in background startup. + +## Runtime reproductions (new tests against unchanged product source) + +| Oracle | Main result | Interpretation | +| --- | --- | --- | +| Config retains drain/grace | FAIL, value omitted | Shared schema strips both unknown fields | +| Default handler cleanup1250ms | FAIL, cleanup not finished when dispatch settles | Real >1s abort cleanup is cut short | +| Worker pause | FAIL, method missing | Functional paused/resume assertions remain in test | +| Worker graceful drain | FAIL, handler signal already aborted at stop entry | Finishing inside budget cannot succeed | +| Memory shutdown / ordinary requeue | FAIL(count2) / PASS(count2,3) | Preservation missing, failure control intact | +| KV polling shutdown / ordinary requeue | FAIL(count2) / PASS(count2,3) | Same | +| Deno KV explicit shutdown / ordinary requeue | Both FAIL, bounded next-delivery timeout | Explicit nack currently returns without enqueuing | +| Real Redis shutdown / ordinary requeue | FAIL(count2) / PASS(count2,3) | Native receipt counts shutdown | +| Real Garnet2.2.1 shutdown / ordinary requeue | FAIL(count2) / PASS(count2,3) | Same, Lua enabled, no Streams | +| Real Postgres17 shutdown / ordinary requeue | FAIL(count2) / PASS(count2,3) | Corrected harness; native SQL counts shutdown | +| Real Worker replacement on Redis/Garnet | Both FAIL on first replacement(count2) | Actual Worker.stop/dispatch handback consumes attempt | +| Real RabbitMQ AMQP wrapper | Broker406 unknown delivery tag before accounting | Upstream async double ack; no accounting claim | + +Initial combined test report2pass/8fail/6unconfigured-backend ignores. Live Redis/Garnet tests execute with no ignored cases. The first Postgres harness started listen and enqueue concurrently and hit a pre-existing client/schema initialization race; corrected harness enqueues before listen. Exclude that harness failure: corrected Postgres1pass/1fail is the accounting evidence. All reports retained privately; no raw paths/endpoints/logs copied here. + +AMQP inspected pinned @fedify/amqp2.4.2 from the actual Deno graph. Constructor defaults nativeRetrial=false; NetScript passes no options but advertises nativeRetrial=true. The upstream non-native async handler path acks immediately and in finally, then again when its Promise settles; real broker rejects duplicate tag. Current nack requeue is also no-op. For this feature use existing amqplib basic.cancel/native channel consumption to stop intake without closing live handlers; counted handback uses an owned confirm channel before old ack. Existing Fedify enqueue has no confirm wait, so do not use its returned Promise as durability proof. Queue name must be forwarded. No new dependency/transport abstraction. + +## Adapter accounting and fencing + +Redis/Garnet increments at claim; existing Lua receipt fences ownership/expiry and maxDeliveries5. Refund must change encoded envelope and maximum-limit decision in the same fenced transition, never expiry recovery. Postgres increments at UPDATE claim, currently unguarded unlock/delete; use existing locked_by as per-claim identity for refund, with same live-receipt/one-settlement invariant. KV polling increments at claim, currently deletes processing before requeue and auto-acks even explicit nack; scoped extraction provides CAS move and settlement guard. Memory needs per-delivery settlement guard and one refund. Deno KV/AMQP expose envelope counts but native implicit provider retries do not update persistent envelope counters; explicit nack will count by re-enqueuing current count, or preserve by re-enqueuing prior count. Document implicit retry limit honestly. + +## Source shape and risks + +Worker463lines, job-dispatcher414, KV polling811, Postgres581 at main. Plan keeps Worker<500 through core-owned bounded drain policy and reduces KV polling via focused message/settlement extraction. Existing queue adapters cardinality debt and public service/type closure debt reviewed; no new service exports. Core schema and data-only configuration require explicit annotations, native public types and doc-lint/JSR gates. No Streams, saga transport, lock or dependency change. Exact core lifecycle/clock/default/public options and delivery semantics are locked in plan.md before implementation. + +## Resource hygiene + +Four active research backend containers (plus one earlier owned Garnet startup replacement) (Redis, Garnet, Postgres, RabbitMQ) used recorded immutable identities plus an owner label. Read-only leak report and teardown preview ran before cleanup; all four active containers removed only after verifying exact identity/owner. Final owned-identity scan empty; foreign resources untouched. Raw reporter output retained privately and this run carries only the public-safe summary. + +## Measured public documentation baseline + +Structured doc-lint with Deno2.9.7 against unchanged main sources: combined core2 private-type-ref, queue1, plugin2. Per-entrypoint diagnostics retained privately for exact delta comparison; config0, Redis0, ports0, worker public entrypoint0, services0; runtime/core2, plugin runtime2; KV polling3, Postgres2, Deno KV2, AMQP1. Combined totals deduplicate diagnostics across entrypoints. These are pre-existing debt, not new waivers; new contracts must add zero diagnostics. Core already owns ./shutdown (5public names) with a generic ShutdownManager; use this small existing subpath for lifecycle/clock/bounded drain, without extending or depending on its unrelated single-phase resource registry. + +Actual thrown-handler failure control added before implementation: memory and KV polling2/2PASS (counts1,2,3); native Deno KVFAIL because its retry returns count1 instead of2. Four external backend variants unconfigured in that control run only; previously recorded native Redis/Garnet/Postgres explicit failure controls executed. Plan now requires counted enqueue-before-ack for ordinary native Deno KV user-handler errors, with durable persistence failure delegated to native unacked recovery. All four new test files pass scoped lint/fmt; no product source changed. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/supervisor.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/supervisor.md index 1618e0579a..717d517d3d 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/supervisor.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/supervisor.md @@ -1,3 +1,5 @@ # Implementation lane Issues2633/2635 share branch feat/worker-pause-drain because abort grace and drain ownership overlap. Fresh origin/main b0eb079575968e843b781579673c9fe2ae250e07 confirmed before creating this worktree. Owner-requested implementation lane Codex gpt-6.1-sol/xhigh owns research/design/implementation; no helper/planning agents. Feature tier, separate GLM5.3/provider-default PLAN-EVAL before production, MuseSpark1.3/xhigh final IMPL-EVAL (matrix lookup before dispatch), different session/vendor family. No merge, ready-merge, rebase/force-push or publication. + +Live matrix lookup confirms feature PLAN-EVAL primary glm_5_3/provider_default, fallback fable_5_1/low. Requested primary via OpenCode Go, provider options{}; generator OpenAI/Codex gpt-6.1-sol/xhigh. No route substitution permitted. Requested/observed identity and exact evaluated head must be recorded in the verdict. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index 32a6a24063..9600a1553c 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -5,3 +5,12 @@ Research is active; required behavior and must-resolve lifecycle/budget/accounting choices are in plan.md. Existing Worker lifetime currently conflates listener cancellation and active job abort; queue/runner ownership boundaries must be named before a repair. Feature Plan-Gate selected. No production changes or reproduction claimed. Fresh main b0eb07957 fetched before branch. Both full issues and relevant public docs/APIs read. Initial source spot-check confirms missing pause/resume, immediate abort at stop and ineffective later30second work wait. Core runner docs confirm1second grace. Next complete backend inventory and functional red tests. + + +## Research and red contracts complete + +All running findings reproduced as listed in research.md; corrected Postgres harness excludes the concurrent schema race. Actual Worker replacements on Redis/Garnet fail at count2 on first restart; ordinary requeue controls pass. AMQP baseline upstream double acknowledgement blocks its accounting oracle and is explicitly scoped as prerequisite, with confirmed-publish and native-cancel plan locked. Four active owned backend containers and the earlier Garnet startup replacement removed by immutable identity after leak/teardown preview; no foreign cleanup. Deno info introduced one exact-version specifier alias in deno.lock; removed only that incidental line, preserving the original lock. No product changes. + +## Design + +Core owns named idle/running/paused/draining/stopped lifecycle, monotonic disposable clock, total25s budget, default5s abort cleanup and1s settlement reservation; plugin wires existing queues/supervisors/runner. Per-attempt intake pause barrier preserves execution signals and lease renewal, start lifetime, task coverage and resumed concurrency. Explicit preserveDeliveryCount nack defaults false; shutdown first-cause only, fenced/idempotent one-refund handback. Config/schema/native loader and both standard runtime constructors forward data-only budgets into the real runner. No helper agent, no dependency on the separate process-shutdown PR. plan.md enumerates ordered <30-file slices, runtime/failure/concurrency/durability cases and all publication/fitness/runtime/consumer gates. Separate feature PLAN-EVAL required before production. diff --git a/packages/plugin-workers-core/tests/runtime/abort-cleanup-budget_test.ts b/packages/plugin-workers-core/tests/runtime/abort-cleanup-budget_test.ts new file mode 100644 index 0000000000..cbea05a7a1 --- /dev/null +++ b/packages/plugin-workers-core/tests/runtime/abort-cleanup-budget_test.ts @@ -0,0 +1,41 @@ +import { assertEquals } from '@std/assert'; +import { InProcessJobRunner, type JobDefinition } from '../../src/runtime/mod.ts'; +import { WorkersConfigSchema } from '../../src/config/mod.ts'; + +Deno.test('workers configuration retains the drain and abort cleanup budgets', () => { + const config = WorkersConfigSchema.parse({ drainTimeoutMs: 8_000, abortGracePeriodMs: 2_500 }); + assertEquals(config && Reflect.get(config, 'drainTimeoutMs'), 8_000); + assertEquals(config && Reflect.get(config, 'abortGracePeriodMs'), 2_500); +}); + +Deno.test('default runner awaits cleanup lasting more than one second after abort', async () => { + const entered = Promise.withResolvers(); + const cleaned = Promise.withResolvers(); + let cleanupFinished = false; + const job: JobDefinition = { + id: 'slow-abort-cleanup', + handler: async ({ signal }) => { + entered.resolve(); + await new Promise((resolve) => { + signal.addEventListener('abort', () => resolve(), { once: true }); + }); + await new Promise((resolve) => setTimeout(resolve, 1_250)); + cleanupFinished = true; + cleaned.resolve(); + return { success: true }; + }, + }; + const runner = new InProcessJobRunner(); + const controller = new AbortController(); + const result = runner.dispatch(job, { id: job.id, job, payload: {}, signal: controller.signal }) + .catch((error: unknown) => error); + try { + await entered.promise; + controller.abort(new DOMException('Controlled shutdown.', 'ShutdownError')); + await result; + assertEquals(cleanupFinished, true, 'Dispatch settled before the handler completed cleanup.'); + } finally { + await cleaned.promise; + await runner.stop(); + } +}); diff --git a/packages/queue/tests/shutdown-requeue_test.ts b/packages/queue/tests/shutdown-requeue_test.ts new file mode 100644 index 0000000000..6f0cf8202b --- /dev/null +++ b/packages/queue/tests/shutdown-requeue_test.ts @@ -0,0 +1,154 @@ +import { assertEquals } from '@std/assert'; +import { MemoryKvAdapter } from '@netscript/kv'; +import { KvPollingAdapter } from '../adapters/kv-polling.adapter.ts'; +import { DenoKvAdapter } from '../adapters/deno-kv.adapter.ts'; +import { RedisAdapter } from '../adapters/redis.adapter.ts'; +import { PostgresAdapter } from '../adapters/postgres.adapter.ts'; +import { AmqpAdapter } from '../adapters/amqp.adapter.ts'; +import { Pool } from 'npm:pg@^8.23.1'; +import { MemoryDeadLetterStore, MemoryQueueAdapter } from '../testing/memory-queue.ts'; +import type { MessageQueue } from '../ports/mod.ts'; +import { bounded, leaseBackends } from './_fixtures/redis-lease-harness.ts'; + +async function repeatedHandback( + create: () => MessageQueue<{ id: string }>, + preserve: boolean, + rounds: number, + throwFailure = false, +): Promise { + let identity: string | undefined; + for (let round = 0; round < rounds; round++) { + const queue = create(); + const stop = new AbortController(); + const receipt = Promise.withResolvers(); + if (round === 0) { + await queue.enqueue({ id: 'healthy' }, { headers: { purpose: 'shutdown-requeue' } }); + } + let expectedFailure = false; + const done = queue.listen(async (_message, delivery) => { + try { + assertEquals(delivery.deliveryCount, preserve ? 1 : round + 1); + identity ??= delivery.messageId; + assertEquals(delivery.messageId, identity); + assertEquals(delivery.headers, { purpose: 'shutdown-requeue' }); + stop.abort(); + if (round === rounds - 1) await delivery.ack(); + else if (throwFailure) { + expectedFailure = true; + receipt.resolve(); + throw new Error('Expected handler failure for counting contract.'); + } else { + // A variable keeps this behavioral oracle executable against the old port contract. + const options = { requeue: true, preserveDeliveryCount: preserve }; + await delivery.nack(options); + } + receipt.resolve(); + } catch (error) { + stop.abort(); + if (!expectedFailure) receipt.reject(error); + throw error; + } + }, { signal: stop.signal, pollIntervalMs: 10, maxDeliveries: 3 }); + void done.catch(() => {}); + try { + await bounded(receipt.promise); + } finally { + stop.abort(); + try { + await bounded(done).catch((error: unknown) => { + if (!expectedFailure) throw error; + }); + } finally { + await queue.stop(); + } + } + } +} + +for (const mode of ['shutdown', 'explicit', 'throw'] as const) { + const preserve = mode === 'shutdown'; + const throwFailure = mode === 'throw'; + const reason = preserve + ? 'shutdown handback preserves' + : throwFailure + ? 'thrown handler failure charges' + : 'handler requeue charges'; + const rounds = preserve ? 8 : 3; + Deno.test(`memory ${reason} delivery attempts across listener restarts`, async () => { + const queue = new MemoryQueueAdapter<{ id: string }>(); + await repeatedHandback(() => queue, preserve, rounds, throwFailure); + }); + Deno.test(`KV polling ${reason} delivery attempts across adapter restarts`, async () => { + await using kv = new MemoryKvAdapter(); + const deadLetters = new MemoryDeadLetterStore(); + const queueName = crypto.randomUUID(); + await repeatedHandback( + () => + new KvPollingAdapter({ + kv, + queueName, + pollInterval: 1, + maxRetries: 2, + retryBaseDelay: 0, + deadLetterStore: deadLetters, + }), + preserve, + rounds, + throwFailure, + ); + assertEquals(await deadLetters.depth(), 0); + }); + Deno.test(`Deno KV ${reason} delivery attempts across adapter restarts`, async () => { + await using kv = await Deno.openKv(':memory:'); + const name = crypto.randomUUID(); + await repeatedHandback(() => DenoKvAdapter.withKv(kv, name), preserve, rounds, throwFailure); + }); + for (const [backend, url] of leaseBackends) { + Deno.test({ + name: `${backend} ${reason} delivery attempts across adapter restarts`, + ignore: !url, + fn: async () => { + const name = `shutdown-${crypto.randomUUID()}`; + await repeatedHandback(() => new RedisAdapter(url!, name), preserve, rounds, throwFailure); + }, + }); + } + const postgres = Deno.env.get('NETSCRIPT_TEST_POSTGRES_URL'); + Deno.test({ + name: `Postgres ${reason} delivery attempts across adapter restarts`, + ignore: !postgres, + fn: async () => { + const tableName = `shutdown_${crypto.randomUUID().replaceAll('-', '')}`; + const client = new Pool({ connectionString: postgres }); + const deadLetterStore = new MemoryDeadLetterStore(); + try { + await repeatedHandback( + () => + new PostgresAdapter({ + client, + tableName, + maxRetries: 2, + pollInterval: 1, + deadLetterStore, + }), + preserve, + rounds, + throwFailure, + ); + assertEquals(await deadLetterStore.depth(), 0); + } finally { + await client.query(`DROP TABLE IF EXISTS "${tableName}"`); + await client.end(); + } + }, + }); + const amqp = Deno.env.get('NETSCRIPT_TEST_AMQP_URL'); + Deno.test({ + name: `AMQP ${reason} delivery attempts across adapter restarts`, + ignore: !amqp, + fn: async () => { + const name = `shutdown-${crypto.randomUUID()}`; + await repeatedHandback(() => new AmqpAdapter(amqp!, name), preserve, rounds, throwFailure); + }, + }); +} diff --git a/plugins/workers/worker/worker-pause-drain_test.ts b/plugins/workers/worker/worker-pause-drain_test.ts new file mode 100644 index 0000000000..f2192a22b7 --- /dev/null +++ b/plugins/workers/worker/worker-pause-drain_test.ts @@ -0,0 +1,98 @@ +import { assert, assertEquals } from '@std/assert'; +import { delay } from '@std/async'; +import { MemoryKvAdapter } from '@netscript/kv'; +import { KvWorkerIdempotencyStore } from '@netscript/plugin-workers-core/stores'; +import type { JobMessage, TaskMessage } from '@netscript/plugin-workers-core/runtime'; +import { MemoryQueueAdapter } from '@netscript/queue/testing'; +import { bounded, until } from '../../../packages/queue/tests/_fixtures/redis-lease-harness.ts'; +import { recoveryContext } from './test_utils/redis-worker-context.ts'; +import { Worker } from './worker.ts'; + +for (const mode of ['pause', 'drain'] as const) { + Deno.test(`Worker ${mode} preserves admitted work and acknowledges a successful job`, async () => { + await using kv = new MemoryKvAdapter(); + const entered = Promise.withResolvers(); + const release = Promise.withResolvers(); + let executions = 0; + let completed = 0; + let acknowledged = 0; + const queue = new MemoryQueueAdapter(); + const dispatch = recoveryContext(new KvWorkerIdempotencyStore({ kv }), { + execute: async (_definition, options) => { + assert(options.signal instanceof AbortSignal); + executions++; + entered.resolve(options.signal); + await release.promise; + return { success: true, duration: 1 }; + }, + }); + const workerOptions = { + ...dispatch, + drainTimeoutMs: 2_000, + abortGracePeriodMs: 500, + queue: { + nativeRetrial: queue.nativeRetrial, + enqueue: (message: JobMessage) => queue.enqueue(message), + stop: () => queue.stop(), + listen: ( + handler: Parameters[0], + options?: Parameters[1], + ) => + queue.listen(async (message, delivery) => { + await handler(message, delivery); + await delivery.ack(); + acknowledged++; + }, options), + }, + taskQueue: new MemoryQueueAdapter(), + executionState: { + ...dispatch.executionState, + complete: (id: string, result: { status: string }) => { + if (result.status === 'completed') completed++; + return Promise.resolve({ id }); + }, + }, + }; + const worker = new Worker(workerOptions); + if (mode === 'pause') { + assert('pause' in worker && typeof worker.pause === 'function', 'Worker.pause is missing.'); + assert( + 'resume' in worker && typeof worker.resume === 'function', + 'Worker.resume is missing.', + ); + } + await queue.enqueue({ jobId: 'first', topic: 'jobs', triggeredBy: 'queue' }); + let startSettled = false; + const started = worker.start().finally(() => startSettled = true); + void started.catch(() => {}); + let stopping: Promise | undefined; + try { + const signal = await bounded(entered.promise); + if (mode === 'pause') { + assert('pause' in worker && typeof worker.pause === 'function'); + await worker.pause(); + } else { + stopping = worker.stop(); + } + assertEquals(signal.aborted, false, 'Stopping intake must leave the handler running.'); + await queue.enqueue({ jobId: 'second', topic: 'jobs', triggeredBy: 'queue' }); + release.resolve(); + await until(() => Promise.resolve(completed === 1)); + await delay(30); + assertEquals(executions, 1, 'Paused/draining worker admitted another job.'); + assertEquals(acknowledged, 1); + if (mode === 'pause') { + assertEquals(startSettled, false, 'Pause must retain the process start lifetime.'); + assert('resume' in worker && typeof worker.resume === 'function'); + await worker.resume(); + await until(() => Promise.resolve(completed === 2)); + } else { + await bounded(stopping!); + } + } finally { + release.resolve(); + await bounded(stopping ?? worker.stop()); + await bounded(started); + } + }); +} diff --git a/plugins/workers/worker/worker-shutdown-deliveries_test.ts b/plugins/workers/worker/worker-shutdown-deliveries_test.ts new file mode 100644 index 0000000000..c63e593286 --- /dev/null +++ b/plugins/workers/worker/worker-shutdown-deliveries_test.ts @@ -0,0 +1,89 @@ +import { assert, assertEquals } from '@std/assert'; +import { MemoryKvAdapter } from '@netscript/kv'; +import { KvWorkerIdempotencyStore } from '@netscript/plugin-workers-core/stores'; +import type { JobMessage, TaskMessage } from '@netscript/plugin-workers-core/runtime'; +import { RedisAdapter } from '@netscript/queue/adapters/redis'; +import { MemoryQueueAdapter } from '@netscript/queue/testing'; +import { + bounded, + leaseBackends, + LeaseHarness, +} from '../../../packages/queue/tests/_fixtures/redis-lease-harness.ts'; +import { recoveryContext } from './test_utils/redis-worker-context.ts'; +import { Worker } from './worker.ts'; + +for (const [backend, url] of leaseBackends) { + Deno.test({ + name: `${backend} eight clean Worker restarts preserve a job beyond the five-delivery limit`, + ignore: !url, + async fn() { + assert(url); + await using harness = new LeaseHarness(url); + await using kv = new MemoryKvAdapter(); + const idempotency = new KvWorkerIdempotencyStore({ kv }); + const inspector = harness.queue(); + let identity: string | undefined; + for (let restart = 0; restart <= 8; restart++) { + const entered = Promise.withResolvers(); + const processed = Promise.withResolvers(); + let deliveryCount = 0; + const jobs = new RedisAdapter(url, harness.name); + const dispatch = recoveryContext(idempotency, { + execute: async (_definition, options) => { + assert(options.signal instanceof AbortSignal); + const signal = options.signal; + entered.resolve(); + if (restart < 8) { + await new Promise((resolve) => { + signal.addEventListener('abort', () => resolve(), { once: true }); + }); + return { success: false, duration: 1 }; + } + return { success: true, duration: 1 }; + }, + }); + const workerOptions = { + ...dispatch, + drainTimeoutMs: 100, + abortGracePeriodMs: 20, + taskQueue: new MemoryQueueAdapter(), + queue: { + nativeRetrial: jobs.nativeRetrial, + enqueue: (message: JobMessage) => jobs.enqueue(message), + stop: () => jobs.stop(), + listen: ( + handler: Parameters[0], + options?: Parameters[1], + ) => + jobs.listen(async (message, delivery) => { + deliveryCount = delivery.deliveryCount; + identity ??= delivery.messageId; + assertEquals(delivery.messageId, identity); + await handler(message, delivery); + processed.resolve(); + }, options), + }, + }; + if (restart === 0) { + await jobs.enqueue({ jobId: 'healthy', topic: 'jobs', triggeredBy: 'queue' }); + } + const worker = new Worker(workerOptions); + const started = worker.start(); + void started.catch(() => {}); + try { + await bounded(entered.promise); + assertEquals(deliveryCount, 1, `Shutdown ${restart} consumed a delivery attempt.`); + if (restart === 8) await bounded(processed.promise); + } finally { + await bounded(worker.stop()); + await bounded(started); + } + assertEquals((await inspector.getHealth()).deadLetterDepth, 0); + } + const health = await inspector.getHealth(); + assertEquals(health.ready, 0); + assertEquals(health.processing, 0); + assertEquals(health.deadLetterDepth, 0); + }, + }); +} From 89effd610f04deba89f08149f5d36ed822c6eb27 Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Mon, 16 Nov 2026 23:02:29 +0100 Subject: [PATCH 03/20] docs(harness): pin derived worker cleanup defaults --- .llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md index 5a765b70de..335411ecc0 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md @@ -8,7 +8,7 @@ netscript-doctrine, netscript-harness, netscript-pr, netscript-tools, netscript- 1. **Lifecycle and intake.** Core-owned named WorkerLifecycleState is idle/running/paused/draining/stopped. Worker exposes pause()/resume(), isPaused, and its health snapshot lifecycle. Pause requests intake cancellation synchronously, leaves all admitted jobs/tasks/trigger jobs and lease renewal running, and keeps start() pending. Each listener supervisor owns a separate attempt AbortController and a pause barrier. Its long-lived run loop parks after the old queue.listen generation drains; resume starts the next generation only after that barrier, preserving concurrency and queue reuse. A committed claim racing pause/drain is handed back without running a new user handler. Pause is intentional healthy idleness, not a crash/restart: preserve restart counters and failure history; failed supervisors cannot be revived by resume. Worker.stop is idempotent, concurrent callers share completion, stop is terminal for that Worker instance; replacement uses a new instance as the runner is already terminal on current main. Pause before start/after stop and resume while draining/stopped are harmless no-ops; repeated pause/resume are idempotent. No queue closes and no runner shutdown during pause. -2. **Two-phase bounded stop.** drainTimeoutMs is the TOTAL Worker.stop budget, default25,000ms. abortGracePeriodMs defaults5,000ms (one fifth of the default drain budget), with explicit finite nonnegative safe-integer validation; total drain must be positive and grace cannot exceed it. Reserve min(1,000ms, one twenty-fifth of total) for queue/listener settlement and close. Work may finish without cancellation until total minus grace minus that reserve (default19s); abort leftovers only then, await runner cleanup/listener generation barriers within the remaining grace, then stop owned queues within the reserve. Use one monotonic deadline, never restart each timeout. Work that completes in the first phase follows normal success/idempotency/ack. Task listeners are included through their generation barriers, including executor and registry/state work before any activeJobs entry. Cancellation/individual deadlines remain immediate with the configured grace. Core runner standalone default5s is documented against this25s worker total and the queue's30s listener drain; the process-level parent budget has at least5s headroom. Larger configured worker budgets require a correspondingly larger process drain budget and are documented. A hung executor/storage/close makes stop reject TimeoutError within the total bound; in-process JavaScript cannot be physically killed and late results cannot mark success after abort. Existing executable parent shutdown owns eventual process termination. +2. **Two-phase bounded stop.** drainTimeoutMs is the TOTAL Worker.stop budget, default25,000ms. abortGracePeriodMs defaults min(5,000ms, floor(drainTimeoutMs/5)), therefore5,000ms for the default total; standalone core runner uses that same default5s. Validate finite safe integer milliseconds, total1..2,147,483,647, grace0..total. Reserve min(1,000ms, max(1ms, floor(total/25))) for queue/listener settlement and close. Work may finish without cancellation until total minus grace minus that reserve (default19s); abort leftovers only then, await runner cleanup/listener generation barriers within the remaining grace, then stop owned queues within the reserve. Use one monotonic deadline, never restart each timeout. Work that completes in the first phase follows normal success/idempotency/ack. Task listeners are included through their generation barriers, including executor and registry/state work before any activeJobs entry. Cancellation/individual deadlines remain immediate with the configured grace. Core runner standalone default5s is documented against this25s worker total and the queue's30s listener drain; the process-level parent budget has at least5s headroom. Larger configured worker budgets require a correspondingly larger process drain budget and are documented. A hung executor/storage/close makes stop reject TimeoutError within the total bound; in-process JavaScript cannot be physically killed and late results cannot mark success after abort. Existing executable parent shutdown owns eventual process termination. 3. **Clock contract.** The core-owned WorkerDrainClock port has monotonic now() and disposable schedule(delayMs,callback), structurally reusing the scheduler method from JobRunnerClock without changing that port's epoch-time meaning. Web Platform default uses performance.now and owned/cleared setTimeout. Worker accepts this clock as an optional direct-composition seam; standard config stays data-only. Core owns the duration/default resolver and bounded drain lifecycle primitives; the plugin wires listeners, runner and queues. Use the existing core ./shutdown public subpath for the new lifecycle/budget primitives and types, avoiding extra runtime/root barrel growth. The existing generic ShutdownManager lacks the two-phase work/abort/settlement contract and is left unchanged. No generic lifecycle registry or dependency added. Tests use manual advancement for boundary/race cases, plus real-time cleanup>1s and actual broker oracles. From 7989476af3b6ff0924ae0e6c4bec2b040de8b428 Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Mon, 16 Nov 2026 23:21:15 +0100 Subject: [PATCH 04/20] docs(harness): retain approved worker lifecycle plan --- .../drift.md | 3 + .../plan-eval.md | 68 +++++++++++++++++++ .../plan.md | 4 +- .../worklog.md | 5 ++ 4 files changed, 78 insertions(+), 2 deletions(-) create mode 100644 .llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-eval.md diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md index 43b83fcbe7..452f7ec099 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md @@ -4,3 +4,6 @@ None yet. Initial findings are code/docs spot-checks, not runtime reproduction. Research corrections: task callbacks do not use activeJobs, so drain waits per-listener generation barriers rather than that map. Native Deno KV explicit nack is a no-op, requiring portable enqueue-before-ack handback. AMQP pinned provider defaults native retry off and double-acks async handlers; native upstream cancel/confirmed handback is necessary for the feature, with baseline transport failure distinguished from accounting evidence. Initial Postgres fixture schema race was a harness failure, corrected before recording product reproduction. The new Worker budget is25s total, with19s work/5s abort cleanup/1s settlement, preserving5s parent work headroom without stacking PR2641. + + +Plan-evaluator corrections: actual docs path tune-worker-runtime.md, not a directory index; exact generated carriers named. A committed claim racing pause is an ordinary charged requeue; draining handback preserves the attempt. The latter follows the already locked shutdown-only opt-in and is pinned by an S4 race test. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-eval.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-eval.md new file mode 100644 index 0000000000..bf7207ee5b --- /dev/null +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-eval.md @@ -0,0 +1,68 @@ +# PLAN-EVAL — feat-worker-pause-drain--issues-2633-2635 + +- Plan evaluator session: independent GLM 5.3 provider-default (OpenCode) session, 2026-11-16 +- Run: feat-worker-pause-drain--issues-2633-2635 +- Surface / archetype: Archetype 3 runtime/behavior, folding queue integration (Archetype 2) and workers plugin (Archetype 5) overlays +- Scope overlays: queue integration (packages/queue), workers plugin (plugins/workers), core contracts (packages/plugin-workers-core) +- Evaluated head: 89effd610f04deba89f08149f5d36ed822c6eb27 on feat/worker-pause-drain (base b0eb079575968e843b781579673c9fe2ae250e07) +- Route attestation: requested pinned feature primary `glm_5_3/provider_default` via OpenCode with empty options — observed exactly that route in this fresh session; live matrix lookup (feature tier, plan evaluator) confirms it as the feature primary and the fallback was not used. Generator: Codex gpt-6.1-sol/xhigh (OpenAI vendor family), separate session and separate vendor family. No substitution, no unasked fallback. Feature tier, cycle 1 of 2. + +## Checklist results + +| Plan-Gate item | Result | Evidence / location | +| --------------------------------------- | ------ | ------------------- | +| Research present and current | PASS | research.md baselined at the exact base and re-verified at head; both issues read in full; load-bearing findings independently spot-checked in tree (see below) | +| Decisions locked | PASS | plan.md Decisions 1–6 lock lifecycle/intake semantics, two-phase total-budget arithmetic, the clock port, the config precedence chain, the preserved-count handback contract, per-adapter mechanics, stop terminality, and PR2641 independence | +| Open-decision sweep | PASS | plan.md "Open decisions / deferred scope" lists no must-resolve choice; evaluator-run sweep found no unflagged rework-forcing decision (A2 below is a clarity residual, not rework-forcing) | +| Commit slices (< 30, gate + files each) | PASS | S1–S7 ordered, sized 11/≤18/≤16/≤18/≤17/≤15/≤7 files, each naming what it proves, the proving gate, and its files; one required path correction in S6 (C1 below, no design impact) | +| Risk register | PASS | plan.md risk register covers pause/stop barrier separation, the task bookkeeping gap, concurrent start/stop/pause allocation, shutdown attribution races, refund abuse/stale/duplicate fencing, AMQP broken-baseline honesty, default timing headroom, F-1/F-16 sizing, JSR slow types, and resource hygiene | +| Gate set selected | PASS | Archetype 3 (all F-1…F-19) plus integration/plugin overlays per gates/archetype-gate-matrix.md; scoped check (--unstable-kv)/lint/fmt/test wrappers, arch:check and quality:gate, doc-lint against the measured baseline, JSR audit plus package dry-runs, release:preflight + docs:accuracy + corpus/publish freshness, full backend suites, end-of-run scaffold.runtime on a supported runner, PLAN-EVAL → IMPL-EVAL, review-thread/head gates in S7 | +| Deferred scope explicit | PASS | plan.md final section: PR2641 process lifecycle, saga SSE2637, all saga transports/Streams, transaction/outbox redesign, broad transport changes, dependency bumps; divergence-record rule stated | +| jsr-audit surface scan (pkg/plugin) | PASS | Rubric applied to the planned public surface before slicing: explicit owned types/annotations, no exported Zod implementation, no broad any/casts, no new dependency/lock change, no root or sub-barrel mirrors, new types only via existing subpaths (core ./shutdown, core config, queue ports), no new package export, zero-new-diagnostics gate against the measured baseline (combined core2/queue1/plugin2), publish dry-run/audit and consumer type contracts in S7 | + +## Independent spot-checks (load-bearing research verified in tree) + +- `packages/plugin-workers-core/src/runtime/in-process-job-runner.ts:42` — `options.abortGracePeriodMs ?? 1_000` default confirmed; grace applies to every abort cause. +- `plugins/workers/bin/runtime.ts:181` — standard runtime passes only `{ registry }` as workerPoolOptions, so the default always applies today. +- `plugins/workers/worker/worker.ts` — `drain()` aborts the shared execution/intake controller and every `activeJobs` entry at drain start (:290–:296), with the later `30_000` wait (:378) that cannot be a graceful work interval; no pause/resume API exists. +- Task dispatch never enters `activeJobs` (checked in job-dispatcher: the map is touched only by the job branch), so the drift correction to per-listener generation barriers is real and required. +- `plugins/workers/worker/job-dispatcher.ts:220–228` — aborted claims are requeued via `nack({ requeue: true })` (also :376). +- Redis/Garnet lease: receipt-keyed atomic transition fencing ownership/expiry and dead-lettering at deliveryCount ≥ maxDeliveries confirmed in `_redis-lease.ts`; per-claim increment semantics consistent with the described refund design. +- Postgres `locked_by` receipt column already exists (claim/release paths confirmed), so the opaque per-claim receipt refund is feasible without a schema change. +- KV polling: processing entry deleted before requeue, unconditional automatic settlement around explicit handles, `kv.atomic` preferred with a documented non-atomic fallback — matches the plan's extraction/CAS/settlement-guard need. +- Deno KV: explicit requeue nack is a no-op (returns without requeue); thrown-handler native retry undercounts — matches the recorded reds and the enqueue-before-ack repair. +- AMQP wrapper advertises `nativeRetrial = true` while constructing the upstream queue with no options (native retry consequently left off upstream) and without forwarding the queue name — matches the prerequisite analysis and honest handling. +- Core `./shutdown` subpath exists with exactly ShutdownManager plus its four option/report/resource/state types; `JobRunnerClock` already provides `now()`/`schedule()`, so the drain clock reuses an existing seam without changing that port's meaning. +- `WorkersConfigObjectSchema` is a plain z.object, so unknown budget fields are stripped today — the config red test is genuinely red; `WorkerPoolOptions.abortGracePeriodMs` legacy seam exists for the locked precedence rules. +- Baselines match: Worker 463, dispatcher 414, KV polling 811, Postgres 581; queue adapters folder currently has 20 children, so the `_kv-polling` extraction must update the open F-16 redis-lease-cardinality debt entry exactly as the plan commits. +- All other slice-named files exist at main; the red tests' shared fixtures (`packages/queue/tests/_fixtures/redis-lease-harness.ts`, `plugins/workers/worker/test_utils/redis-worker-context.ts`) pre-exist and are not part of this branch. +- The four new test files are executable against unchanged main contracts (nack options go through an untyped variable; pause/resume access is `in`-narrowed), so the reproduction-before-fix evidence is executable red tests plus native backend controls, not a code-only claim. Diff at head: exactly four red test files under packages/plugins/docs plus run-dir documents — no product source changed. + +## Open-decision sweep (evaluator-run) + +Examined: pause racing-claim handback counting; deadline arithmetic at boundary totals (safe-integer bounds, grace 0..total, floored settlement reserve keeps the work window non-negative — min(5000, floor(T/5)) and min(1000, max(1, floor(T/25))) are consistent with 25s total = 19s work + 5s abort cleanup + 1s settlement); stop/replace instance policy; the config precedence chain; the preserved-refund first-cause window; Deno KV counted enqueue-before-ack; the AMQP confirm-channel primitive; the KV polling extraction boundary. None is rework-forcing when deferred; none is left without a locked default. + +## Acceptance coverage + +All nine acceptance boxes (#2633 boxes 1–4, #2635 boxes 1–5) are mapped to concrete planned evidence (plan.md "Acceptance evidence to produce") and are meetable under the locked decisions. #2635's preservation box is covered per adapter that tracks deliveries (memory, KV polling, Deno KV, Redis/Garnet, Postgres) plus the AMQP prerequisite handled honestly through S3; #2635 box3's beyond-limit oracle (8 clean restarts vs limit 5) is in place with real Worker instances and adapter-contract restarts plus final depth controls. + +## Verdict + +`PASS` + +### Required correction before S6 (plan bookkeeping, no design change) + +1. S6 names `docs/site/background-processing/how-to/tune-worker-runtime/index.md`, which does not exist. The actual file is `docs/site/background-processing/how-to/tune-worker-runtime.md`. Correct the S6 file name (keeping its box mappings for #2633 box4 / #2635 box5) and record the fix in drift.md. + +### Advisories (not blocking) + +- A1: In S6, name the generated carriers exactly (the agent-docs-prose carrier and the export-surface corpus — `gen:agent-docs-prose` and `gen:mcp-export-corpus` outputs — plus the publish assets carrier) so S7's freshness checks have unambiguous targets. +- A2: Decision 1 hands back a committed claim racing pause/drain without running a new user handler, while Decision 5 gates preserved-count refunds to the controlled-shutdown first cause. Add one sentence (or a draining-only precondition) stating that a claim raced by pause is an ordinary charged requeue, and have S4 tests pin it. + +## Notes + +- Feature tier cycle 1/2. On this PASS the generator may begin S2; cycle 2, if ever needed, is a fresh evaluator session and may edit a fixable plan per tier policy. +- PR state at evaluation: draft, `status:plan-eval`, milestone 0.0.10; required CI policy-skipped under the draft skip labels with `command-postgres` green; no reviews yet. Nothing here waives S7, IMPL-EVAL, or the current-head CI/review-thread gates. +- No resources were launched for this evaluation; this run's research backends and volumes were cleaned by proven identity per leak-report.md. Raw reproduction reports and diagnostics remain private to the operator; only repo-relative public evidence is used here. + +PR https://github.com/rickylabs/netscript/pull/2645, head 89effd610f04deba89f08149f5d36ed822c6eb27, checks and reviews: required CI policy-skipped (draft, command-postgres green), no reviews yet. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md index 335411ecc0..054afc9521 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md @@ -6,7 +6,7 @@ netscript-doctrine, netscript-harness, netscript-pr, netscript-tools, netscript- ## Decisions -1. **Lifecycle and intake.** Core-owned named WorkerLifecycleState is idle/running/paused/draining/stopped. Worker exposes pause()/resume(), isPaused, and its health snapshot lifecycle. Pause requests intake cancellation synchronously, leaves all admitted jobs/tasks/trigger jobs and lease renewal running, and keeps start() pending. Each listener supervisor owns a separate attempt AbortController and a pause barrier. Its long-lived run loop parks after the old queue.listen generation drains; resume starts the next generation only after that barrier, preserving concurrency and queue reuse. A committed claim racing pause/drain is handed back without running a new user handler. Pause is intentional healthy idleness, not a crash/restart: preserve restart counters and failure history; failed supervisors cannot be revived by resume. Worker.stop is idempotent, concurrent callers share completion, stop is terminal for that Worker instance; replacement uses a new instance as the runner is already terminal on current main. Pause before start/after stop and resume while draining/stopped are harmless no-ops; repeated pause/resume are idempotent. No queue closes and no runner shutdown during pause. +1. **Lifecycle and intake.** Core-owned named WorkerLifecycleState is idle/running/paused/draining/stopped. Worker exposes pause()/resume(), isPaused, and its health snapshot lifecycle. Pause requests intake cancellation synchronously, leaves all admitted jobs/tasks/trigger jobs and lease renewal running, and keeps start() pending. Each listener supervisor owns a separate attempt AbortController and a pause barrier. Its long-lived run loop parks after the old queue.listen generation drains; resume starts the next generation only after that barrier, preserving concurrency and queue reuse. A committed claim racing pause/drain is handed back without running a new user handler. A claim already committed before pause is handed back as an ordinary charged requeue; only draining/shutdown handback preserves its attempt, pinned by an S4 raced-claim test. Pause is intentional healthy idleness, not a crash/restart: preserve restart counters and failure history; failed supervisors cannot be revived by resume. Worker.stop is idempotent, concurrent callers share completion, stop is terminal for that Worker instance; replacement uses a new instance as the runner is already terminal on current main. Pause before start/after stop and resume while draining/stopped are harmless no-ops; repeated pause/resume are idempotent. No queue closes and no runner shutdown during pause. 2. **Two-phase bounded stop.** drainTimeoutMs is the TOTAL Worker.stop budget, default25,000ms. abortGracePeriodMs defaults min(5,000ms, floor(drainTimeoutMs/5)), therefore5,000ms for the default total; standalone core runner uses that same default5s. Validate finite safe integer milliseconds, total1..2,147,483,647, grace0..total. Reserve min(1,000ms, max(1ms, floor(total/25))) for queue/listener settlement and close. Work may finish without cancellation until total minus grace minus that reserve (default19s); abort leftovers only then, await runner cleanup/listener generation barriers within the remaining grace, then stop owned queues within the reserve. Use one monotonic deadline, never restart each timeout. Work that completes in the first phase follows normal success/idempotency/ack. Task listeners are included through their generation barriers, including executor and registry/state work before any activeJobs entry. Cancellation/individual deadlines remain immediate with the configured grace. Core runner standalone default5s is documented against this25s worker total and the queue's30s listener drain; the process-level parent budget has at least5s headroom. Larger configured worker budgets require a correspondingly larger process drain budget and are documented. A hung executor/storage/close makes stop reject TimeoutError within the total bound; in-process JavaScript cannot be physically killed and late results cannot mark success after abort. Existing executable parent shutdown owns eventual process termination. @@ -34,7 +34,7 @@ S2 core/queue contracts and lease backends (<=18files): core src/shutdown/worker S3 remaining adapters (<=16files): kv-polling.adapter.ts, _kv-polling/message.ts, _kv-polling/settlement.ts, deno-kv.adapter.ts, amqp.adapter.ts, testing/memory-queue.ts; queue shutdown-requeue_test.ts and shutdown-refund-races_test.ts plus focused polling/AMQP tests; up to7run artifacts. Proves all counting backends hand back without count, failure still charges, no duplicate/stale refund, native resources close. Scoped check/test/lint/fmt and backend suites before push. S4 worker supervisor and drain wiring (<=18files): worker/mod.ts, worker.ts, worker-options.ts, listener-supervisor.ts, queue-consumer.ts, job-dispatcher.ts, job-runner-pool.ts; worker-pause-drain_test.ts, worker-shutdown-deliveries_test.ts, listener-supervisor_test.ts plus focused worker-drain-budget_test.ts; up to7run artifacts. Core named lifecycle/clock contract reused; keep Worker under500 by moving bounded-budget orchestration into core-owned worker-drain primitive. Proves pause/resume start lifetime, task/trigger health/concurrency, successful drain ack, later abort and real8replacement delivery oracle. Adjacent worker cancellation/recovery/lease/idempotency tests retained. S5 config/standard runtime (<=17files): core config/workers-config.ts, runtime/in-process-job-runner.ts and options JSDoc if needed; core config/runtime tests abort-cleanup-budget_test.ts and job-cancellation_test.ts; plugin src/runtime/workers-config.ts, src/cli/load-workers-config.ts, bin/runtime.ts; runtime budget forwarding subprocess test and private fixture in tests/runtime; up to7run artifacts. Proves schema defaults/validation and actual startup, including no-registry and custom-registry paths; runner cleanup>1s executed. No CLI->runtime layering reversal. -S6 docs/consumer freshness (<=15files): core README, plugin README, docs/site/background-processing/how-to/tune-worker-runtime/index.md; generated docs-prose.generated.ts, export-surface.generated.ts, CLI assets/publish carrier if changed; up to7run artifacts. Proves public option defaults, pause/drain/count/durability/ownership limits; canonical prose/corpus/assets refresh + docs accuracy/preflight/publish freshness. +S6 docs/consumer freshness (<=15files): core README, plugin README, docs/site/background-processing/how-to/tune-worker-runtime.md; generated packages/mcp/src/infrastructure/docs/agent-docs-prose.generated.ts, packages/mcp/src/infrastructure/export-surfaces/export-surface-corpus.generated.ts, packages/cli/src/kernel/assets/embedded.generated.ts and publish-assets.generated.ts if changed; up to7run artifacts. Proves public option defaults, pause/drain/count/durability/ownership limits; canonical prose/corpus/assets refresh + docs accuracy/preflight/publish freshness. S7 final gates/evaluation (<=7run files): all required local static/fitness/publication gates, full scaffold.runtime one pass on supported runtime, resource cleanup, separate MuseSpark implementation evaluation, acceptance checkboxes/body mappings, ready transition and current head required CI/thread gate. No merge/ready-merge/publication. ## Risk register and mitigations diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index 9600a1553c..e49c1bce2e 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -14,3 +14,8 @@ All running findings reproduced as listed in research.md; corrected Postgres har ## Design Core owns named idle/running/paused/draining/stopped lifecycle, monotonic disposable clock, total25s budget, default5s abort cleanup and1s settlement reservation; plugin wires existing queues/supervisors/runner. Per-attempt intake pause barrier preserves execution signals and lease renewal, start lifetime, task coverage and resumed concurrency. Explicit preserveDeliveryCount nack defaults false; shutdown first-cause only, fenced/idempotent one-refund handback. Config/schema/native loader and both standard runtime constructors forward data-only budgets into the real runner. No helper agent, no dependency on the separate process-shutdown PR. plan.md enumerates ordered <30-file slices, runtime/failure/concurrency/durability cases and all publication/fitness/runtime/consumer gates. Separate feature PLAN-EVAL required before production. + + +## Independent Plan-Gate PASS + +[GLM5.3/provider-default independent PLAN-EVAL PASS](https://github.com/rickylabs/netscript/pull/2645#issuecomment-6931112123) at89effd610:8/8Plan-Gate,9/9acceptance coverage, no production edits, separate session/family confirmed. Retain verdict. Correct S6 documentation path to tune-worker-runtime.md and name generated carriers; pin committed claims racing pause to ordinary charged requeue (preservation is draining/shutdown only), with an S4 race oracle. These are evaluator bookkeeping/advisory repairs, no architectural re-plan. S2 may begin. From 261ffccb13a7b20f6d0e30faf2b19765fb6dc735 Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Mon, 16 Nov 2026 23:32:17 +0100 Subject: [PATCH 05/20] feat(workers): define bounded drain and fenced shutdown handback --- .../worklog.md | 9 + .../plugin-workers-core/src/shutdown/mod.ts | 11 ++ .../src/shutdown/worker-drain.ts | 75 ++++++++ .../src/shutdown/worker-lifecycle.ts | 68 +++++++ .../tests/runtime/worker-drain-budget_test.ts | 114 +++++++++++ .../queue/adapters/_redis-lease-script.ts | 3 +- packages/queue/adapters/_redis-lease.ts | 9 +- packages/queue/adapters/postgres.adapter.ts | 147 +++++++------- packages/queue/ports/message-queue.ts | 7 + packages/queue/tests/postgres-adapter_test.ts | 1 + .../queue/tests/shutdown-refund-races_test.ts | 180 ++++++++++++++++++ 11 files changed, 542 insertions(+), 82 deletions(-) create mode 100644 packages/plugin-workers-core/src/shutdown/worker-drain.ts create mode 100644 packages/plugin-workers-core/src/shutdown/worker-lifecycle.ts create mode 100644 packages/plugin-workers-core/tests/runtime/worker-drain-budget_test.ts create mode 100644 packages/queue/tests/shutdown-refund-races_test.ts diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index e49c1bce2e..63ccbad953 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -19,3 +19,12 @@ Core owns named idle/running/paused/draining/stopped lifecycle, monotonic dispos ## Independent Plan-Gate PASS [GLM5.3/provider-default independent PLAN-EVAL PASS](https://github.com/rickylabs/netscript/pull/2645#issuecomment-6931112123) at89effd610:8/8Plan-Gate,9/9acceptance coverage, no production edits, separate session/family confirmed. Retain verdict. Correct S6 documentation path to tune-worker-runtime.md and name generated carriers; pin committed claims racing pause to ordinary charged requeue (preservation is draining/shutdown only), with an S4 race oracle. These are evaluator bookkeeping/advisory repairs, no architectural re-plan. S2 may begin. + + +## S2 core contract and receipt settlement + +Public contracts first: core ./shutdown named lifecycle/monotonic disposable clock/default total25s/default grace5s/resolver and fixed work-abort-settlement primitive; queue NackOptions.preserveDeliveryCount defaultsfalse, requeue-only. Redis/Garnet handback encodes the refunded envelope in the existing atomic fenced receipt transition, bypassing limit exhaustion only on that explicit live handback; expiry recovery remains charged. Postgres generates an opaque per-claim locked_by receipt, atomically refunds/unlocks within the live fence, guards ack/release and once-only settlement. Simplified existing envelope/DLQ plumbing reduces touched Postgres581->568lines. No new schema/dependency/command/Streams. + +Independent implementation-lane slice review: diff follows approved boundaries; no orphaned timers, root/barrel mirror, hidden lifecycle registry or generic transport abstraction. The ordinary default nack stays charged. Core tests assert19s work plus5s cleanup plus1s settlement share one monotonic25s deadline and clear timers. Native tests prove repeated8handbacks, max-limit bypass, terminal preservation ignored, duplicate/stale/expired contexts cannot refund/remove newer receipts. Closed Redis contexts legitimately reject AbortError; the test pins that safe outcome and unchanged newer claim rather than requiring reopening. + +Validation: structured live Redis/Garnet/Postgres suite28pass/0fail/0ignored (shutdown requeue + refund races + existing Redis lease cases); budget/Postgres adjacent8pass/0fail; scoped check/lint/fmt0; core shutdown doclint0; release:preflight fourPASS and docs:accuracyPASS. Original config/runner/Worker reds remain pending S4/S5, other adapter reds pending S3. Three positively owned backend containers retained for following slices, immutable IDs/mounts privately recorded; foreign resources untouched. diff --git a/packages/plugin-workers-core/src/shutdown/mod.ts b/packages/plugin-workers-core/src/shutdown/mod.ts index 2a167b7e1f..c7e82a1341 100644 --- a/packages/plugin-workers-core/src/shutdown/mod.ts +++ b/packages/plugin-workers-core/src/shutdown/mod.ts @@ -11,3 +11,14 @@ export type { ShutdownResource, ShutdownState, } from './shutdown-manager.ts'; + +export { drainWorker, type WorkerDrainActions } from './worker-drain.ts'; +export { + DEFAULT_ABORT_GRACE_PERIOD_MS, + DEFAULT_WORKER_DRAIN_TIMEOUT_MS, + resolveWorkerDrainBudget, + type WorkerDrainBudget, + type WorkerDrainClock, + type WorkerDrainOptions, + type WorkerLifecycleState, +} from './worker-lifecycle.ts'; diff --git a/packages/plugin-workers-core/src/shutdown/worker-drain.ts b/packages/plugin-workers-core/src/shutdown/worker-drain.ts new file mode 100644 index 0000000000..88605dc752 --- /dev/null +++ b/packages/plugin-workers-core/src/shutdown/worker-drain.ts @@ -0,0 +1,75 @@ +/** Bounded graceful work, abort cleanup, and queue settlement. @module */ +import { WebPlatformJobRunnerClock } from '../adapters/web-platform-job-runner-clock.ts'; +import { + resolveWorkerDrainBudget, + type WorkerDrainClock, + type WorkerDrainOptions, +} from './worker-lifecycle.ts'; + +/** Fixed phases wired by a worker's composition root. */ +export interface WorkerDrainActions { + /** Stop intake and join all admitted deliveries, including tasks and trigger jobs. */ + pauseIntake(): Promise; + /** Cancel remaining executions; this does not terminate in-process JavaScript. */ + abort(): void; + /** Join runner cleanup and the paused delivery barriers after abort. */ + abortCompleted(): Promise; + /** Stop listeners and close only the queues owned by this worker. */ + close(): Promise; +} + +/** + * Stop intake, allow work to finish, then abort and settle within one monotonic deadline. + * A work-window timeout requests cancellation; uncompleted cleanup/settlement rejects + * with TimeoutError. Process owners remain responsible for physical termination. + */ +export async function drainWorker( + options: WorkerDrainOptions, + actions: WorkerDrainActions, +): Promise { + const budget = resolveWorkerDrainBudget(options); + const clock = options.drainClock ?? new WebPlatformJobRunnerClock(() => performance.now()); + const started = clock.now(); + let failure: unknown; + let timedOut = false; + try { + await until(actions.pauseIntake(), started + budget.workMs, clock); + } catch (error) { + failure = error; + } + try { + actions.abort(); + timedOut = !await until( + actions.abortCompleted(), + started + budget.totalMs - budget.settlementMs, + clock, + ); + } catch (error) { + failure ??= error; + } + try { + timedOut = !await until(actions.close(), started + budget.totalMs, clock) || timedOut; + } catch (error) { + failure ??= error; + } + if (timedOut) throw new DOMException('Worker drain budget elapsed.', 'TimeoutError'); + if (failure !== undefined) throw failure; +} + +async function until( + work: Promise, + deadline: number, + clock: WorkerDrainClock, +): Promise { + if (clock.now() > deadline) { + void work.catch(() => {}); + return false; + } + const timeout = Promise.withResolvers(); + const cancel = clock.schedule(Math.max(0, deadline - clock.now()), () => timeout.resolve(false)); + try { + return await Promise.race([work.then(() => true), timeout.promise]); + } finally { + cancel(); + } +} diff --git a/packages/plugin-workers-core/src/shutdown/worker-lifecycle.ts b/packages/plugin-workers-core/src/shutdown/worker-lifecycle.ts new file mode 100644 index 0000000000..e0624973be --- /dev/null +++ b/packages/plugin-workers-core/src/shutdown/worker-lifecycle.ts @@ -0,0 +1,68 @@ +/** Worker lifecycle and shutdown budget contracts. @module */ + +/** Lifecycle of one worker instance; stopped instances are terminal. */ +export type WorkerLifecycleState = 'idle' | 'running' | 'paused' | 'draining' | 'stopped'; + +/** Monotonic time and disposable scheduling for worker drain deadlines. */ +export interface WorkerDrainClock { + /** Nondecreasing elapsed time in milliseconds. */ + now(): number; + /** Schedule a callback; the returned function cancels the timer. */ + schedule(delayMs: number, callback: () => void): () => void; +} + +/** Configuration of the whole worker stop, including abort cleanup and settlement. */ +export interface WorkerDrainOptions { + /** Total stop budget; defaults to 25,000 ms, leaving headroom in a 30-second parent. */ + readonly drainTimeoutMs?: number; + /** + * Handler cleanup after abort, within the total budget. Defaults to min(5,000, + * floor(drainTimeoutMs / 5)); the default total reserves 19 s work + 5 s abort + * cleanup + 1 s settlement, inside the queue's 30-second listener drain. + */ + readonly abortGracePeriodMs?: number; + /** Direct-composition clock; configuration files contain durations only. */ + readonly drainClock?: WorkerDrainClock; +} + +/** Resolved durations within one total worker stop deadline. */ +export interface WorkerDrainBudget { + /** Whole stop, including settlement. */ + readonly totalMs: number; + /** Time available before aborting admitted work. */ + readonly workMs: number; + /** Requested handler cleanup; actual waiting is capped by the remaining total. */ + readonly abortGracePeriodMs: number; + /** Reserved time to settle listeners and close owned queues. */ + readonly settlementMs: number; +} + +/** Default total stop budget, below the process and queue drain budgets. */ +export const DEFAULT_WORKER_DRAIN_TIMEOUT_MS: number = 25_000; + +/** One fifth of the default total, allowing cleanup longer than one second. */ +export const DEFAULT_ABORT_GRACE_PERIOD_MS: number = DEFAULT_WORKER_DRAIN_TIMEOUT_MS / 5; + +/** Validate and resolve worker durations before allocating runtime resources. */ +export function resolveWorkerDrainBudget(options: WorkerDrainOptions = {}): WorkerDrainBudget { + const totalMs = options.drainTimeoutMs ?? DEFAULT_WORKER_DRAIN_TIMEOUT_MS; + const abortGracePeriodMs = options.abortGracePeriodMs ?? + Math.min(DEFAULT_ABORT_GRACE_PERIOD_MS, Math.floor(totalMs / 5)); + for ( + const [name, value, minimum, maximum] of [ + ['drainTimeoutMs', totalMs, 1, 2_147_483_647], + ['abortGracePeriodMs', abortGracePeriodMs, 0, totalMs], + ] as const + ) { + if (!Number.isSafeInteger(value) || value < minimum || value > maximum) { + throw new RangeError(`${name} must be an integer between ${minimum} and ${maximum}.`); + } + } + const settlementMs = Math.min(1_000, Math.max(1, Math.floor(totalMs / 25))); + return { + totalMs, + abortGracePeriodMs, + settlementMs, + workMs: Math.max(0, totalMs - abortGracePeriodMs - settlementMs), + }; +} diff --git a/packages/plugin-workers-core/tests/runtime/worker-drain-budget_test.ts b/packages/plugin-workers-core/tests/runtime/worker-drain-budget_test.ts new file mode 100644 index 0000000000..797fbbbfd0 --- /dev/null +++ b/packages/plugin-workers-core/tests/runtime/worker-drain-budget_test.ts @@ -0,0 +1,114 @@ +import { assertEquals, assertRejects, assertThrows } from '@std/assert'; +import { + drainWorker, + resolveWorkerDrainBudget, + type WorkerDrainClock, +} from '../../src/shutdown/mod.ts'; + +class ManualDrainClock implements WorkerDrainClock { + time = 0; + timers = new Set<{ at: number; callback: () => void }>(); + now(): number { + return this.time; + } + schedule(ms: number, callback: () => void): () => void { + const timer = { at: this.time + ms, callback }; + this.timers.add(timer); + return () => { + this.timers.delete(timer); + }; + } + advance(ms: number): void { + this.time += ms; + for (const timer of this.timers) { + if (timer.at <= this.time) { + this.timers.delete(timer); + timer.callback(); + } + } + } +} + +const turn = async () => { + for (let i = 0; i < 12; i++) await Promise.resolve(); +}; + +Deno.test('worker total budget derives work, cleanup and settlement before allocating resources', () => { + assertEquals(resolveWorkerDrainBudget(), { + totalMs: 25_000, + abortGracePeriodMs: 5_000, + settlementMs: 1_000, + workMs: 19_000, + }); + assertEquals(resolveWorkerDrainBudget({ drainTimeoutMs: 100 }), { + totalMs: 100, + abortGracePeriodMs: 20, + settlementMs: 4, + workMs: 76, + }); + for (const drainTimeoutMs of [0, -1, NaN, Infinity, 1.5, 2_147_483_648]) { + assertThrows(() => resolveWorkerDrainBudget({ drainTimeoutMs }), RangeError); + } + for (const abortGracePeriodMs of [-1, NaN, Infinity, 1.5, 101]) { + assertThrows( + () => resolveWorkerDrainBudget({ drainTimeoutMs: 100, abortGracePeriodMs }), + RangeError, + ); + } +}); + +Deno.test('worker work that finishes inside the window is not aborted while running', async () => { + const clock = new ManualDrainClock(); + const work = Promise.withResolvers(); + let aborted = false; + let closed = false; + const done = drainWorker({ drainClock: clock }, { + pauseIntake: () => work.promise, + abort: () => { + aborted = true; + }, + abortCompleted: () => Promise.resolve(), + close: () => { + closed = true; + return Promise.resolve(); + }, + }); + clock.advance(18_999); + await turn(); + assertEquals(aborted, false); + work.resolve(); + await done; + assertEquals(aborted, true); + assertEquals(closed, true); + assertEquals(clock.timers.size, 0); +}); + +Deno.test('worker abort and close share the original total deadline, disposing every timer', async () => { + const clock = new ManualDrainClock(); + const pending = Promise.withResolvers(); + let aborted = false; + let closing = false; + const done = drainWorker({ drainClock: clock }, { + pauseIntake: () => pending.promise, + abort: () => { + aborted = true; + }, + abortCompleted: () => pending.promise, + close: () => { + closing = true; + return pending.promise; + }, + }); + const rejection = assertRejects(() => done, DOMException, 'Worker drain budget elapsed.'); + clock.advance(19_000); + await turn(); + assertEquals(aborted, true); + assertEquals(closing, false); + clock.advance(5_000); + await turn(); + assertEquals(closing, true); + clock.advance(1_000); + await rejection; + assertEquals(clock.timers.size, 0); + pending.resolve(); +}); diff --git a/packages/queue/adapters/_redis-lease-script.ts b/packages/queue/adapters/_redis-lease-script.ts index a8e7fb93c3..760d254f34 100644 --- a/packages/queue/adapters/_redis-lease-script.ts +++ b/packages/queue/adapters/_redis-lease-script.ts @@ -120,7 +120,8 @@ if op == 'nack' or op == 'reclaim' then local error = terminal(request.deadLetter) if error then return error end else - local error = insert(KEYS[2], record.envelope) + local bytes = op == 'nack' and request.requeueEnvelope or record.envelope + local error = insert(KEYS[2], bytes) if error then return error end if op == 'reclaim' then redis.call('HINCRBY', KEYS[7], 'reclaimed', 1) end end diff --git a/packages/queue/adapters/_redis-lease.ts b/packages/queue/adapters/_redis-lease.ts index 7c63de4b49..3d8f076f29 100644 --- a/packages/queue/adapters/_redis-lease.ts +++ b/packages/queue/adapters/_redis-lease.ts @@ -85,8 +85,9 @@ export class RedisLeases { /** Settle only a still-owned, unexpired receipt. */ async settle(lease: RedisLease, options?: NackOptions, signal?: AbortSignal): Promise { + const preserve = options?.preserveDeliveryCount === true && options.requeue !== false; const deadLetter = options && (!(options.requeue ?? true) || - lease.envelope.deliveryCount >= lease.maxDeliveries) + !preserve && lease.envelope.deliveryCount >= lease.maxDeliveries) ? this.deadLetter(lease, options) : undefined; await this.whileAvailable(() => @@ -94,6 +95,12 @@ export class RedisLeases { op: options ? 'nack' : 'ack', fence: lease.fence, deadLetter, + requeueEnvelope: preserve + ? JSON.stringify({ + ...lease.envelope, + deliveryCount: Math.max(0, lease.envelope.deliveryCount - 1), + }) + : undefined, }), signal); if (deadLetter) this.scheduleOutboxFlush(); } diff --git a/packages/queue/adapters/postgres.adapter.ts b/packages/queue/adapters/postgres.adapter.ts index 5b77e358d2..a29c9a6cb3 100644 --- a/packages/queue/adapters/postgres.adapter.ts +++ b/packages/queue/adapters/postgres.adapter.ts @@ -109,6 +109,7 @@ interface PostgresMessageRow extends Record { readonly payload: unknown; readonly headers: unknown; readonly delivery_count: number; + readonly locked_by: string; readonly enqueued_at: Date | string; } @@ -127,7 +128,6 @@ export class PostgresAdapter implements MessageQueue { private readonly explicitClient?: PostgresQueueClient; private readonly explicitDeadLetterStore?: DeadLetterStorePort; private readonly url?: string; - private readonly consumerId = crypto.randomUUID(); private client: PostgresQueueClient | null = null; private deadLetterStore: DeadLetterStorePort | null = null; private schemaReady: Promise | null = null; @@ -256,7 +256,7 @@ export class PostgresAdapter implements MessageQueue { continue; } - await this.processRow(row, handler); + await this.processRow(row, handler, options?.visibilityTimeout ?? this.visibilityTimeout); } } catch (error) { if (signal.aborted) { @@ -392,8 +392,8 @@ export class PostgresAdapter implements MessageQueue { FOR UPDATE SKIP LOCKED LIMIT 1 ) - RETURNING message_id, payload, headers, delivery_count, enqueued_at`, - [this.queueName, this.consumerId, visibilityTimeout], + RETURNING message_id, payload, headers, delivery_count, enqueued_at, locked_by`, + [this.queueName, crypto.randomUUID(), visibilityTimeout], ); return result.rows[0] ?? null; } @@ -402,23 +402,14 @@ export class PostgresAdapter implements MessageQueue { private async processRow( row: PostgresMessageRow, handler: (message: T, context: MessageContext) => Promise, + visibilityTimeout: number, ): Promise { - let payload: T; - let headers = normalizeHeaders(row.headers); - let messageId = row.message_id; - let enqueuedAt = new Date(row.enqueued_at); - let deliveryCount = row.delivery_count; - - const rawMessage = row.payload; - if (isMessageEnvelope(rawMessage)) { - payload = rawMessage.payload; - headers = rawMessage.headers; - messageId = rawMessage.messageId; - enqueuedAt = new Date(rawMessage.enqueuedAt); - deliveryCount = row.delivery_count; - } else { - payload = rawMessage as T; - } + const envelope = isMessageEnvelope(row.payload) ? row.payload : undefined; + const payload = envelope ? envelope.payload : row.payload as T; + const headers = envelope?.headers ?? normalizeHeaders(row.headers); + const messageId = envelope?.messageId ?? row.message_id; + const enqueuedAt = new Date(envelope?.enqueuedAt ?? row.enqueued_at); + const deliveryCount = row.delivery_count; let settled = false; const context = this.createContext( @@ -429,6 +420,8 @@ export class PostgresAdapter implements MessageQueue { headers, deliveryCount, () => settled = true, + row.locked_by, + visibilityTimeout, ); try { @@ -444,7 +437,7 @@ export class PostgresAdapter implements MessageQueue { } } - /** Creates the public message context for a claimed storage row. */ + /** Creates one settlement boundary for this claimed storage row. */ private createContext( storageMessageId: string, messageId: string, @@ -453,85 +446,79 @@ export class PostgresAdapter implements MessageQueue { headers: Record, deliveryCount: number, markSettled: () => void, + receipt: string, + visibilityTimeout: number, ): MessageContext { + let settlement: Promise | undefined; + const settle = (operation: () => Promise): Promise => { + return settlement ??= operation().then(markSettled, (error: unknown) => { + settlement = undefined; + throw error; + }); + }; return createMessageContext( messageId, enqueuedAt, headers, deliveryCount, - async () => { - await this.ack(storageMessageId); - markSettled(); - }, - async (options: NackOptions = {}) => { - if ((options.requeue ?? true) && deliveryCount < this.maxRetries + 1) { - await this.release(storageMessageId); - } else { - await this.deadLetter( - storageMessageId, - messageId, - payload, - enqueuedAt, - headers, - deliveryCount, - deliveryCount >= this.maxRetries + 1 ? 'max_attempts_exceeded' : options.reason, - options, - ); - await this.ack(storageMessageId); - } - markSettled(); - }, + () => settle(() => this.ack(storageMessageId, receipt, visibilityTimeout)), + (options: NackOptions = {}) => + settle(async () => { + const requeue = options.requeue ?? true; + const preserve = requeue && options.preserveDeliveryCount === true; + if (requeue && (preserve || deliveryCount < this.maxRetries + 1)) { + await this.release( + storageMessageId, + receipt, + deliveryCount, + visibilityTimeout, + preserve, + ); + } else { + const store = await this.ensureDeadLetterStore(); + await store.append(toDeadLetterRecord( + { messageId, queueName: this.queueName, payload, headers, deliveryCount, enqueuedAt }, + deliveryCount >= this.maxRetries + 1 + ? 'max_attempts_exceeded' + : options.reason ?? 'nack_without_requeue', + options, + )); + await this.ack(storageMessageId, receipt, visibilityTimeout); + } + }), ); } - /** Deletes a message row after successful processing. */ - private async ack(messageId: string): Promise { + /** Delete only the live, unreplaced claim that produced this context. */ + private async ack(messageId: string, receipt: string, visibilityTimeout: number): Promise { const client = await this.ensureClient(); await client.query( `DELETE FROM ${this.tableIdentifier} - WHERE queue_name = $1 AND message_id = $2`, - [this.queueName, messageId], + WHERE queue_name = $1 AND message_id = $2 AND locked_by = $3 + AND locked_at > NOW() - ($4::integer * INTERVAL '1 millisecond')`, + [this.queueName, messageId, receipt, visibilityTimeout], ); } - /** Releases a message row so another listener can claim it again. */ - private async release(messageId: string): Promise { + /** Refund and unlock in one fenced update; retrying the same receipt cannot refund twice. */ + private async release( + messageId: string, + receipt: string, + deliveryCount: number, + visibilityTimeout: number, + preserve: boolean, + ): Promise { const client = await this.ensureClient(); await client.query( `UPDATE ${this.tableIdentifier} - SET locked_at = NULL, - locked_by = NULL, - available_at = NOW() - WHERE queue_name = $1 AND message_id = $2`, - [this.queueName, messageId], + SET locked_at = NULL, locked_by = NULL, available_at = NOW(), + delivery_count = delivery_count - CASE WHEN $5::boolean THEN 1 ELSE 0 END + WHERE queue_name = $1 AND message_id = $2 AND locked_by = $3 + AND delivery_count = $4 + AND locked_at > NOW() - ($6::integer * INTERVAL '1 millisecond')`, + [this.queueName, messageId, receipt, deliveryCount, preserve, visibilityTimeout], ); } - - /** Records an exhausted or explicitly rejected message in the dead-letter store. */ - private async deadLetter( - _storageMessageId: string, - messageId: string, - payload: T, - enqueuedAt: Date, - headers: Record, - deliveryCount: number, - reason: NackOptions['reason'], - options: NackOptions, - ): Promise { - const store = await this.ensureDeadLetterStore(); - await store.append(toDeadLetterRecord( - { - messageId, - queueName: this.queueName, - payload, - headers, - deliveryCount, - enqueuedAt, - }, - reason ?? 'nack_without_requeue', - options, - )); - } } function quoteQualifiedIdentifier(name: string): string { diff --git a/packages/queue/ports/message-queue.ts b/packages/queue/ports/message-queue.ts index f069b5e7d8..64b35ad9ec 100644 --- a/packages/queue/ports/message-queue.ts +++ b/packages/queue/ports/message-queue.ts @@ -20,6 +20,13 @@ export interface NackOptions { */ requeue?: boolean; + /** + * Return a controlled-shutdown delivery without charging this attempt. Effective + * only when requeue is true; defaults to false. Refunds this live claim once, + * preserving identity and metadata. Ordinary failures and expired claims count. + */ + preserveDeliveryCount?: boolean; + /** * Terminal failure reason to store when `requeue` is false. */ diff --git a/packages/queue/tests/postgres-adapter_test.ts b/packages/queue/tests/postgres-adapter_test.ts index bf10bb5f0a..c4d82ea065 100644 --- a/packages/queue/tests/postgres-adapter_test.ts +++ b/packages/queue/tests/postgres-adapter_test.ts @@ -79,6 +79,7 @@ class FakePostgresClient implements PostgresQueueClient { payload: row.payload, headers: row.headers, delivery_count: row.deliveryCount, + locked_by: row.lockedBy, enqueued_at: row.enqueuedAt, } as unknown as Row], }); diff --git a/packages/queue/tests/shutdown-refund-races_test.ts b/packages/queue/tests/shutdown-refund-races_test.ts new file mode 100644 index 0000000000..145389790a --- /dev/null +++ b/packages/queue/tests/shutdown-refund-races_test.ts @@ -0,0 +1,180 @@ +import { assert, assertEquals } from '@std/assert'; +import { delay } from '@std/async'; +import { Pool } from 'npm:pg@^8.23.1'; +import { PostgresAdapter } from '../adapters/postgres.adapter.ts'; +import type { MessageContext, MessageQueue } from '../ports/mod.ts'; +import { MemoryDeadLetterStore } from '../testing/memory-queue.ts'; +import { bounded, LeaseHarness } from './_fixtures/redis-lease-harness.ts'; + +interface Fixture extends AsyncDisposable { + queue(): MessageQueue; + depths(): Promise<{ ready: number; processing: number; dead: number }>; +} + +function fixture(kind: string, url: string, limit: number): Fixture { + if (kind !== 'Postgres') { + const harness = new LeaseHarness(url); + const inspector = harness.queue(); + return { + queue: () => harness.queue(), + depths: async () => { + const health = await inspector.getHealth(); + return { ready: health.ready, processing: health.processing, dead: health.deadLetterDepth }; + }, + [Symbol.asyncDispose]: () => harness[Symbol.asyncDispose](), + }; + } + const client = new Pool({ connectionString: url }); + const tableName = `refund_${crypto.randomUUID().replaceAll('-', '')}`; + const deadLetterStore = new MemoryDeadLetterStore(); + return { + queue: () => + new PostgresAdapter({ + client, + tableName, + deadLetterStore, + maxRetries: limit - 1, + pollInterval: 1, + }), + depths: async () => { + const { rows } = await client.query( + `SELECT count(*) FILTER (WHERE locked_by IS NULL) AS ready, + count(*) FILTER (WHERE locked_by IS NOT NULL) AS processing FROM "${tableName}"`, + ); + return { + ready: Number(rows[0].ready), + processing: Number(rows[0].processing), + dead: await deadLetterStore.depth(), + }; + }, + [Symbol.asyncDispose]: async () => { + try { + await client.query(`DROP TABLE IF EXISTS "${tableName}"`); + } finally { + await client.end(); + } + }, + }; +} + +function hold(queue: MessageQueue, limit: number, visibilityTimeout = 1_000) { + const received = Promise.withResolvers(); + const released = Promise.withResolvers(); + const controller = new AbortController(); + const done = queue.listen(async (_message, context) => { + received.resolve(context); + await released.promise; + }, { + signal: controller.signal, + maxDeliveries: limit, + visibilityTimeout, + recoveryGrace: 0, + pollIntervalMs: 10, + }); + void done.catch(() => {}); + return { + received: () => bounded(received.promise), + stop: async () => { + controller.abort(); + released.resolve(); + try { + await bounded(done); + } finally { + await queue.stop(); + } + }, + }; +} + +for ( + const [kind, url] of [ + ['Redis', Deno.env.get('NETSCRIPT_TEST_REDIS_URL')], + ['Garnet', Deno.env.get('NETSCRIPT_TEST_GARNET_URL')], + ['Postgres', Deno.env.get('NETSCRIPT_TEST_POSTGRES_URL')], + ] as const +) { + Deno.test({ + name: + `${kind} shutdown refund at the delivery limit is once-only and cannot settle a replacement`, + ignore: !url, + async fn() { + await using backend = fixture(kind, url!, 1); + const queue = backend.queue(); + await queue.enqueue('healthy'); + const first = hold(queue, 1); + let second: ReturnType | undefined; + try { + const old = await first.received(); + await Promise.all([ + old.nack({ preserveDeliveryCount: true }), + old.nack({ preserveDeliveryCount: true }), + ]); + await old.ack(); + assertEquals(await backend.depths(), { ready: 1, processing: 0, dead: 0 }); + await first.stop(); + second = hold(backend.queue(), 1); + const current = await second.received(); + assertEquals(current.messageId, old.messageId); + assertEquals(current.deliveryCount, 1); + for ( + const operation of [() => old.nack({ preserveDeliveryCount: true }), () => old.ack()] + ) { + try { + await operation(); + } catch (error) { + assert(error instanceof Error && error.name === 'AbortError'); + } + } + assertEquals(await backend.depths(), { ready: 0, processing: 1, dead: 0 }); + await current.ack(); + } finally { + await first.stop(); + await second?.stop(); + } + assertEquals(await backend.depths(), { ready: 0, processing: 0, dead: 0 }); + }, + }); + Deno.test({ + name: `${kind} expired shutdown receipt cannot refund or acknowledge a newer delivery`, + ignore: !url, + async fn() { + await using backend = fixture(kind, url!, 5); + const queue = backend.queue(); + await queue.enqueue('expired'); + const first = hold(queue, 5, 30); + let second: ReturnType | undefined; + try { + const old = await first.received(); + await delay(60); + second = hold(backend.queue(), 5); + const current = await second.received(); + assertEquals(current.deliveryCount, 2, 'Expiry recovery remains a charged attempt.'); + await old.nack({ preserveDeliveryCount: true }); + await old.ack(); + await first.stop(); + assertEquals(await backend.depths(), { ready: 0, processing: 1, dead: 0 }); + await current.ack(); + } finally { + await first.stop(); + await second?.stop(); + } + assertEquals(await backend.depths(), { ready: 0, processing: 0, dead: 0 }); + }, + }); + Deno.test({ + name: `${kind} preservation never changes a terminal nack`, + ignore: !url, + async fn() { + await using backend = fixture(kind, url!, 1); + const queue = backend.queue(); + await queue.enqueue('terminal'); + const consumer = hold(queue, 1); + try { + await (await consumer.received()).nack({ requeue: false, preserveDeliveryCount: true }); + } finally { + await consumer.stop(); + } + assertEquals(await backend.depths(), { ready: 0, processing: 0, dead: 1 }); + }, + }); +} From 014730e1ca4e155fe1059054dc46d4d1dd75603d Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Mon, 16 Nov 2026 23:49:14 +0100 Subject: [PATCH 06/20] feat(queue): preserve native and polling shutdown handback --- .llm/harness/debt/arch-debt.md | 6 +- .../context-pack.md | 4 +- .../drift.md | 3 + .../leak-report.md | 4 + .../plan-intake-drain-addendum.md | 31 +++ .../plan.md | 5 + .../worklog.md | 11 + .../queue/adapters/_kv-polling/message.ts | 66 ++++++ .../queue/adapters/_kv-polling/settlement.ts | 116 ++++++++++ packages/queue/adapters/deno-kv.adapter.ts | 69 ++++-- packages/queue/adapters/kv-polling.adapter.ts | 202 +++++------------- packages/queue/testing/memory-queue.ts | 74 ++++--- packages/queue/tests/kv-polling-dlq_test.ts | 8 +- .../queue/tests/listener-drain-owner_test.ts | 80 +++++++ .../tests/polling-settlement-outage_test.ts | 32 +++ .../queue/tests/shutdown-refund-local_test.ts | 134 ++++++++++++ 16 files changed, 646 insertions(+), 199 deletions(-) create mode 100644 .llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-intake-drain-addendum.md create mode 100644 packages/queue/adapters/_kv-polling/message.ts create mode 100644 packages/queue/adapters/_kv-polling/settlement.ts create mode 100644 packages/queue/tests/listener-drain-owner_test.ts create mode 100644 packages/queue/tests/polling-settlement-outage_test.ts create mode 100644 packages/queue/tests/shutdown-refund-local_test.ts diff --git a/.llm/harness/debt/arch-debt.md b/.llm/harness/debt/arch-debt.md index ff3ffdd9e9..3b9134e356 100644 --- a/.llm/harness/debt/arch-debt.md +++ b/.llm/harness/debt/arch-debt.md @@ -3206,7 +3206,11 @@ match the merged exemplars). IMPL-EVAL must not FAIL a slice for retaining eithe Folding these into redis.adapter.ts would deepen its existing orchestration size and mix independently reviewed failure boundaries. Issue #2351 adds one private admission/drain module to keep the adapter below the 500-line gate while preserving independent receipt lifetimes. - Direct children now total 20. + Direct children now total 20. The #2635 pause/shutdown handback fix groups polling message/key + and receipt settlement into one private `_kv-polling/` concern directory (21 direct children). + This reduces the touched polling adapter from811 to727lines and enables CAS/fenced handback + without deepening its existing F-1 error. No new public export, dependency or scanner exemption. + The same cardinality closing gate applies; no further directory growth is accepted by analogy. - **Owner:** NetScript queue maintainers. - **Target:** Next queue adapter organization pass after the 0.0.9 recovery fix. - **Linked plan:** .llm/runs/orch-divybot-6536806908--2395/plan.md; issue #2395. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md index 5ebc08e867..0288c6bf18 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md @@ -1,9 +1,11 @@ # Context pack -Issues2633/2635, PR2645, branch feat/worker-pause-drain, base b0eb079575968e843b781579673c9fe2ae250e07. Read plan.md/research.md/worklog.md/drift.md/supervisor.md and the two live issues in full. Plan-Gate required before product source changes. Four new tests are executable against unchanged main and include genuine red behavior plus failure controls. See research for exact results and excluded harness errors. No current run-owned resources: all five container identities including an earlier startup replacement, and five associated anonymous volumes proven by mount events, cleaned; foreign resources untouched. +Issues2633/2635, PR2645, branch feat/worker-pause-drain, base b0eb079575968e843b781579673c9fe2ae250e07. Read plan.md/research.md/worklog.md/drift.md/supervisor.md and the two live issues in full. Plan-Gate required before product source changes. Four new tests are executable against unchanged main and include genuine red behavior plus failure controls. See research for exact results and excluded harness errors. Research resources cleaned: all five container identities including an earlier startup replacement, and five associated anonymous volumes proven by mount events. Current implementation retains three positively owned Redis/Garnet/Postgres backends for S2/S3, with immutable IDs/mounts privately recorded; foreign resources untouched. Primary paths: plugins/workers/worker/{worker,worker-options,listener-supervisor,queue-consumer,job-dispatcher,delivery-lease,job-runner-pool}.ts; plugins/workers/bin/runtime.ts; plugins/workers/src/cli/load-workers-config.ts; core config/workers-config.ts and runtime/in-process-job-runner.ts; existing core src/shutdown/ subpath. Queue ports/message-queue.ts, Redis lease/script, Postgres, KV polling, native Deno KV, AMQP, memory. Actual public types inspected with deno doc first. Use Deno2.9.7 through mise. rtk/rg absent; plain commands. Scoped structured wrappers, --unstable-kv for checks. agentic matrix valid; current aliases maint:leak-check, maint:teardown, check:review-threads. Native Aspire guard rejects this local remote-Docker topology; full end-of-PR supported-runner smoke required, never bypass/bridge. User no helper agents; independent evaluator session/family separation mandatory. No merge, status:ready-merge, force push/rebase, publication, lock/cache deletion/reload, root fmt or saga transport/Streams edits. Public artifacts/comments/bodies must contain no operator absolute paths, hostnames/ports, session/thread IDs or downstream names. Keep raw logs private, public evidence repo-relative. Evaluator writes plan-eval.md and phase comment, leaves artifact uncommitted. No source changes, commits, pushes, labels/readiness changes or PR watching by evaluator. + +Current checkpoint: initial Plan-Gate approved; S2native receipts28/28 and core/Postgres8/8green, S3a portable handback/local guards and adjacent gates green. Added listener ownership contract is unimplemented and requires the independent PLAN-EVAL follow-up in plan-intake-drain-addendum.md before use. Raw ownership/drain red reports remain private; public results are in worklog. No Worker lifecycle/config or AMQP implementation yet. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md index 452f7ec099..b4aad2b777 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md @@ -7,3 +7,6 @@ Research corrections: task callbacks do not use activeJobs, so drain waits per-l Plan-evaluator corrections: actual docs path tune-worker-runtime.md, not a directory index; exact generated carriers named. A committed claim racing pause is an ordinary charged requeue; draining handback preserves the attempt. The latter follows the already locked shutdown-only opt-in and is pinned by an S4 race test. + + +S3a stays within approved adapter scope. Its handback storage-failure test exposed an unjoined callback rejection; red fault test precedes the error-propagation fix, which keeps the original processing entry durable and fails the listener. Native listener post-intake expiry can resolve a pause generation before handlers finish;3real/scaled owner cases red with3default controls green. That rework-forcing contract discovery is stopped at a separate plan checkpoint, with exact proposal/files/gates in plan-intake-drain-addendum.md. No new-option implementation yet. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md index e1e1a45928..6b19994d4d 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md @@ -1,3 +1,7 @@ # Research resource hygiene Commands: maint:leak-check and maint:teardown preview against this run and worktree. Reporter raw output retained privately. Four active research backend containers (plus one earlier owned Garnet startup replacement) positively identified by recorded immutable IDs and matching test-owner label; all removed by exact ID after preview. Subsequent scan of that exact owner set: zero containers. All five anonymous volumes removed only after native mount events proved association with the five run-created container identities and a reference scan proved no other container used them. The earlier Garnet startup replacement was included. No owned volume remains. Foreign/unknown-owner resources untouched. No process or container pattern kill. + +## Active implementation resources + +Three run-created Redis/Garnet/Postgres backend containers are intentionally retained for S2/S3 validation. Immutable IDs and native mount identities recorded privately at creation, with the dedicated implementation test-owner label. No AppHost or foreign container/process started or stopped. Final cleanup will preview the repo teardown tool and remove only those exact proven identities/mounts; the research zero-survivor result above is historical, not a claim that these active implementation resources are absent. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-intake-drain-addendum.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-intake-drain-addendum.md new file mode 100644 index 0000000000..5f57e18877 --- /dev/null +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-intake-drain-addendum.md @@ -0,0 +1,31 @@ +# Intake drain ownership addendum + +## Reason and reproduction + +S3 work exposed a hidden dependency in the approved pause barrier design: RedisListener.stop starts a timer equal to visibilityTimeout; when it expires, it aborts settlement, stops receipt IO and resolves the listen generation even if a callback is still running. KV polling also ends its listen generation after its native visibility-based wait. A supervisor pause barrier cannot represent admitted work if its backend ends that generation first. This is the same production behavior as main; no owner-control implementation has been made yet. + +New listener-drain-owner_test.ts uses a50ms native drain and a callback held across500ms, with an extra proposed option passed through a variable so it executes against the old public contract. Live Redis/Garnet and KV polling:3owner-option cases FAIL because listen ends before callback completion;3ordinary-default controls PASS,0ignored. Redis/Garnet lease is extended2s before cancellation, separating the short listener drain from claim expiry; the owner case must keep receipt renewal usable as well. The first200ms KV polling probe was too close to its100ms loop cadence and is excluded as nondeterministic;500ms settles the distinct control and ownership outcomes. The S2 native handback/fencing gates remain28/28green. + +## Locked additional contract + +Add optional ListenOptions.drainTimeoutMs for Redis and KV polling only. Omission retains the existing visibilityTimeout-based post-intake drain. Values are finite positive safe integers through2,147,483,647, or positive Infinity. Reject NaN, zero, negative, fractional and excessive finite values before listener allocation. The value controls backend waiting after intake cancellation; it does not extend visibility or change the worker total budget. + +Infinity delegates admitted-callback lifetime to a bounded outer owner. Worker passes this option for jobs, task and trigger listeners because its core drain already owns one25s total deadline with19s work/5s abort cleanup/1s settlement. Pause has no stop deadline: active jobs continue with their existing execution signals and available lease-renewal capabilities. Resume still waits the previous listener generation. Stop still cancels execution after its work window and rejects within its original total if cleanup/close cannot settle. No timeout bypass or unbounded stop is introduced at the Worker boundary; direct queue consumers get their existing finite default. + +Redis policy keeps the existing default and finite-validation source names. For Infinity, do not arm the native post-intake expiry timer; keep settlement/receipt IO alive until admitted callbacks settle, then run existing close. Native expiry/reclaim/attempt counting, Lua commands, all receipt fencing and outbox semantics are unchanged. KV polling uses the explicit owner budget for its finally wait; retain its old default/visibility behavior when omitted. No change to lease visibility/renewal semantics on any provider; no long visibility timeouts used as a workaround. Native Deno KV/Postgres/memory and the planned AMQP native-channel implementation already join callbacks without this independent short timeout, so they need no new behavior for the option. + +A claim committed before pause but delivered to the Worker after pause remains an ordinary charged requeue. Draining handback remains the shutdown-only opt-in. All previous config, default, typed first-cause, count, fencing and ownership decisions stand. No Redis Streams, saga transport or generic pause/transport abstraction. + +## Files/slices and gates + +S3a already implements the previously approved portable handback for memory/DenoKV/KV polling and the CAS/once guards; it is kept separate from this unimplemented contract. S3a gates:9new local retry contracts,9local fence/terminal/late-context cases,22adjacent/outage/lifecycle cases, static0, queue combined doclint1 with exact per-entrypoint baseline unchanged, preflight/accuracyPASS. The failing owner-option test is explicit future-contract evidence, not included as a false green. + +S3b additional file group (<=10files): packages/queue/ports/message-queue.ts, adapters/_redis-lease-state.ts, adapters/_redis-listener.ts, adapters/kv-polling.adapter.ts; tests/listener-drain-owner_test.ts plus existing Redis listener policy/lease tests; up to3run docs. Implement contract first then native adaptations; no source for this additional behavior until independent PLAN-EVAL approval. Prove6/6owner/default controls, invalid values allocate no resources, and preserve existing Redis/Garnet default finite stop/expiry/claim/outage regressions. Implement AMQP from the original S3 plan as a separate subsequent slice (<30files), without coupling its publisher/consumer logic to this option. + +S4 add forwarding/long pause/drain proof to the already named worker/listener test group: each jobs/tasks/triggers listen receives the owner option, pause keeps start and callbacks alive beyond a scaled former native budget, resume waits the same generation without over-concurrency, stop remains within the configured total. Existing graceful-success ack, tasks-before-activeJobs, race count, eight restarts, noncooperative budget and fatal-supervisor tests remain required. S4 stays<30files. + +S6 update queue README + worker/core/tuning docs for provider-specific post-intake ownership and the separately bounded worker total; this is one additional docs file, S6 remains<30. S7 retains every static/fitness/publication/consumer/full supported scaffold-runtime and independent final evaluation gate. No gate weakened or replaced. + +## Risk / open decisions + +No open design decision. Required before code: independent feature PLAN-EVAL follow-up for this added public option (same primary route, fresh session per T3). The main risk is accidentally making default queue stop unbounded:3negative controls and retained native stop/reclaim tests pin omission behavior. The companion risk is conflating visibility with drain waiting: explicit JSDoc, policy separation and extended-live-lease oracle pin it. Worker must never pass this option without the separately enforced total drain; explicit forwarding plus noncooperative budget tests pin that boundary. Existing visibility limitations of providers without extendVisibility remain unchanged and documented; the option promises callback joining, not stronger delivery exclusivity. No new dependency/lock/command or root surface; primitive numeric field adds no slow type/private reference risk. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md index 054afc9521..8845004552 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan.md @@ -57,3 +57,8 @@ Archetype3 all F1-F19 plus integration/plugin overlays. Structured scoped source ## Open decisions / deferred scope No must-resolve design choice remains. Exact private fixture/test filenames may change without altering contracts (safe to defer). Crash/outage recovery before a counted handback can be committed remains provider-limited and explicitly documented; ordinary handler failures must count on all adapters. General queue transaction/outbox redesign, broad Postgres/DenoKV transport changes, process lifecycle fixes owned by PR2641, saga SSE2637, all saga transports/Streams, dependency bumps and unrelated architecture migration are deferred. Any discovered contract/budget/fencing divergence is recorded and evaluated before implementing that change. + + +## Required implementation-discovery checkpoint + +S3a reproduced a native listener timeout that invalidates long-lived pause barriers. See plan-intake-drain-addendum.md for the proposed provider-specific ListenOptions.drainTimeoutMs ownership contract,3red/3default-control cases and unchanged Worker total. That added contract remains unimplemented until independent feature PLAN-EVAL follow-up. Original S3 is split into portable KV/memory handback (S3a), listener owner policy, then original AMQP work; each group<30files and all final gates retained. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index 63ccbad953..7b37d166b6 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -28,3 +28,14 @@ Public contracts first: core ./shutdown named lifecycle/monotonic disposable clo Independent implementation-lane slice review: diff follows approved boundaries; no orphaned timers, root/barrel mirror, hidden lifecycle registry or generic transport abstraction. The ordinary default nack stays charged. Core tests assert19s work plus5s cleanup plus1s settlement share one monotonic25s deadline and clear timers. Native tests prove repeated8handbacks, max-limit bypass, terminal preservation ignored, duplicate/stale/expired contexts cannot refund/remove newer receipts. Closed Redis contexts legitimately reject AbortError; the test pins that safe outcome and unchanged newer claim rather than requiring reopening. Validation: structured live Redis/Garnet/Postgres suite28pass/0fail/0ignored (shutdown requeue + refund races + existing Redis lease cases); budget/Postgres adjacent8pass/0fail; scoped check/lint/fmt0; core shutdown doclint0; release:preflight fourPASS and docs:accuracyPASS. Original config/runner/Worker reds remain pending S4/S5, other adapter reds pending S3. Three positively owned backend containers retained for following slices, immutable IDs/mounts privately recorded; foreign resources untouched. + + +## S3a portable native/KV/memory handback + +Memory and native Deno KV explicit handback preserve envelope identity/count once, ordinary requeue and thrown-handler failures charge. Native Deno KV publishes before native ack and closes retained contexts on automatic success; failed persistence still throws to native recovery. KV polling extracts private message/key and settlement modules; claims get opaque receipts, CAS move refunds only owned live entries, explicit settlement suppresses automatic duplicate settlement, stale/expired callbacks cannot affect replacements. Active deliveries use independent identities so immediate handback cannot corrupt concurrency bookkeeping. Failed settlement previously escaped an unjoined async callback; new fault test red (uncaught settlement outage) before fixing listener failure propagation with durable processing entry retained. Stop cancels the local listener even when given an external signal; per-generation failure closures cannot abort a later listener. KV polling shrinks811->727lines. Existing private terminal fixture now installs its claimed row before settlement, retaining its terminal/DLQ assertions under fencing. + +Gates:9/9local handback/count controls;9/9duplicate/terminal/late-context guards;22/22adjacent lifecycle/DLQ/memory/outage tests; scoped check/lint/fmt0; queue combined doclint1 and exact per-entrypoint baseline unchanged; preflight/accuracyPASS. Three positively owned backends remain active for the next slices, immutable identities/mounts retained privately. + +## Pause-barrier discovery and scoped plan follow-up + +Running listener-drain-owner_test.ts reveals Redis/Garnet and KV polling resolve their listener after a native visibility-based wait even while the callback remains active.500msheld callback vs50msnative limit:3proposed-owner cases red,3ordinary-default controls green,0ignored. The initial200ms polling observation was near the100ms cadence and is excluded. This would invalidate a long pause barrier despite independent execution signals. Added plan-intake-drain-addendum.md locks an explicit optional listener drain owner budget (Infinity only under the Worker's separately bounded25s total), default finite behavior unchanged. No added-option implementation before independent PLAN-EVAL follow-up. No AMQP or Worker lifecycle implementation yet. diff --git a/packages/queue/adapters/_kv-polling/message.ts b/packages/queue/adapters/_kv-polling/message.ts new file mode 100644 index 0000000000..076dc3a0d4 --- /dev/null +++ b/packages/queue/adapters/_kv-polling/message.ts @@ -0,0 +1,66 @@ +/** Private durable polling-queue message and keys. @module */ +import type { KvKey } from '@netscript/kv'; + +/** + * Internal message structure stored in KV + */ +export interface QueueMessage { + /** Unique message ID */ + id: string; + /** Message payload */ + payload: T; + /** Queue name */ + queue: string; + /** When the message was enqueued */ + enqueuedAt: string; + /** When the message becomes available (for delayed messages) */ + availableAt: string; + /** Number of delivery attempts */ + attempts: number; + /** Maximum retry attempts */ + maxAttempts: number; + /** Priority (higher = more important) */ + priority: number; + /** Custom headers */ + headers: Record; + /** Deduplication ID */ + deduplicationId?: string; + /** When the message was claimed for processing */ + claimedAt?: string; + /** Which worker claimed the message */ + claimedBy?: string; + /** Identity of this delivery, independent of its worker or refunded count. */ + claimReceipt?: string; + /** Visibility timeout (when the claim expires) */ + visibilityTimeout?: string; + /** Originating durable identity, independent of payload/deduplication identity. */ + retentionId?: string; +} + +/** + * KV key prefixes for queue data + */ +export const KvPrefixes = { + /** Pending messages ready for processing */ + pending: 'queue:pending', + /** Messages currently being processed */ + processing: 'queue:processing', + /** Dead letter queue for failed messages */ + dlq: 'queue:dlq', + /** Deduplication tracking */ + dedup: 'queue:dedup', +} as const; + +/** Pending key preserves the adapter's priority/time ordering. */ +export function pollingPendingKey( + queue: string, + priority: number, + timestamp: string, + id: string, +): KvKey { + return [KvPrefixes.pending, queue, String(1000 - priority).padStart(4, '0'), timestamp, id]; +} +/** One processing slot per durable message identity. */ +export function pollingProcessingKey(queue: string, id: string): KvKey { + return [KvPrefixes.processing, queue, id]; +} diff --git a/packages/queue/adapters/_kv-polling/settlement.ts b/packages/queue/adapters/_kv-polling/settlement.ts new file mode 100644 index 0000000000..f7bf4d6fd7 --- /dev/null +++ b/packages/queue/adapters/_kv-polling/settlement.ts @@ -0,0 +1,116 @@ +/** Receipt-fenced settlement for polling queues. @module */ +import type { AtomicMutation, KvEntry, KvKey, WatchableKv } from '@netscript/kv'; +import { + type DeadLetterStorePort, + type NackOptions, + QueueError, + QueueErrorCode, +} from '../../ports/mod.ts'; +import { toDeadLetterRecord } from '../_envelope.ts'; +import { pollingPendingKey, pollingProcessingKey, type QueueMessage } from './message.ts'; + +async function owned( + kv: WatchableKv, + queue: string, + message: QueueMessage, + recovering = false, +): Promise> | undefined> { + const entry = await kv.get>(pollingProcessingKey(queue, message.id)); + if ( + !entry || entry.value.claimReceipt !== message.claimReceipt || + entry.value.claimedBy !== message.claimedBy || entry.value.claimedAt !== message.claimedAt + ) return; + if (entry.value.visibilityTimeout) { + const expired = Date.parse(entry.value.visibilityTimeout) <= Date.now(); + if (recovering !== expired) return; + } + return entry; +} + +async function transition( + kv: WatchableKv, + entry: KvEntry, + mutations: AtomicMutation[], +): Promise { + if (kv.atomic) { + const result = await kv.atomic( + [{ key: entry.key, versionstamp: entry.versionstamp }], + mutations, + ); + if (!result.ok && (await kv.get(entry.key))?.versionstamp === entry.versionstamp) { + throw new QueueError( + 'Polling receipt settlement was refused.', + QueueErrorCode.DEQUEUE_FAILED, + ); + } + return; + } + // Caller-provided non-atomic stores retain the documented single-consumer fallback. + // Publish before deletion so a failed handback cannot lose the original claim. + for (const mutation of mutations) { + if (mutation.type === 'set') await kv.set(mutation.key, mutation.value); + else if (mutation.type === 'delete') { + const current = await kv.get(entry.key); + if (current?.versionstamp === entry.versionstamp) await kv.delete(mutation.key); + } + } +} + +/** A stale or replaced context cannot delete another claim. */ +export async function acknowledgePollingMessage( + kv: WatchableKv, + queue: string, + message: QueueMessage, +): Promise { + const entry = await owned(kv, queue, message); + if (entry) await transition(kv, entry, [{ type: 'delete', key: entry.key }]); +} + +/** Refund and move the live receipt atomically; crash recovery always remains charged. */ +export async function rejectPollingMessage( + kv: WatchableKv, + queue: string, + message: QueueMessage, + options: NackOptions, + policy: Readonly<{ retryBaseDelay: number; retryMaxDelay: number }>, + store: () => Promise, + recovering = false, +): Promise { + const entry = await owned(kv, queue, message, recovering); + if (!entry) return; + const requeue = options.requeue ?? true; + const preserve = requeue && options.preserveDeliveryCount === true && !recovering; + if (!requeue || !preserve && message.attempts >= message.maxAttempts) { + await (await store()).append(toDeadLetterRecord( + { + messageId: message.id, + queueName: queue, + payload: message.payload, + headers: message.headers, + deliveryCount: message.attempts, + enqueuedAt: message.enqueuedAt, + }, + message.attempts >= message.maxAttempts + ? 'max_attempts_exceeded' + : options.reason ?? 'nack_without_requeue', + options, + )); + await transition(kv, entry, [{ type: 'delete', key: entry.key }]); + return; + } + const delay = preserve + ? 0 + : Math.min(policy.retryBaseDelay * 2 ** (message.attempts - 1), policy.retryMaxDelay); + const availableAt = new Date(Date.now() + delay).toISOString(); + const value: QueueMessage = { + ...message, + availableAt, + attempts: preserve ? Math.max(0, message.attempts - 1) : message.attempts, + claimedAt: undefined, + claimedBy: undefined, + claimReceipt: undefined, + visibilityTimeout: undefined, + }; + const key: KvKey = pollingPendingKey(queue, message.priority, availableAt, message.id); + await transition(kv, entry, [{ type: 'set', key, value }, { type: 'delete', key: entry.key }]); +} diff --git a/packages/queue/adapters/deno-kv.adapter.ts b/packages/queue/adapters/deno-kv.adapter.ts index d617068f92..9ccc5eecfb 100644 --- a/packages/queue/adapters/deno-kv.adapter.ts +++ b/packages/queue/adapters/deno-kv.adapter.ts @@ -337,8 +337,16 @@ export class DenoKvAdapter implements MessageQueue { new Date(envelope.enqueuedAt), envelope.headers, envelope.deliveryCount + 1, + envelope, ); - await handler(payload, context); + try { + await handler(payload, context); + await context.ack(); + } catch { + // Publish the counted retry before native acknowledgement; failed persistence + // still throws to the native queue so the original delivery is retained. + await context.nack({ requeue: true }); + } } /** @@ -350,31 +358,52 @@ export class DenoKvAdapter implements MessageQueue { enqueuedAt: Date, headers: Record, deliveryCount: number, + envelope?: MessageEnvelope, ): MessageContext { + let settlement: Promise | undefined; + const settle = (operation: () => Promise): Promise => { + return settlement ??= operation().catch((error: unknown) => { + settlement = undefined; + throw error; + }); + }; return createMessageContext( messageId, enqueuedAt, headers, deliveryCount, - async () => {}, - async (options: NackOptions = {}) => { - if (options.requeue ?? true) { - return; - } - const store = await this.ensureDeadLetterStore(); - await store.append(toDeadLetterRecord( - { - messageId, - queueName: this.queueName, - payload, - headers, - deliveryCount, - enqueuedAt, - }, - options.reason ?? 'nack_without_requeue', - options, - )); - }, + () => settle(() => Promise.resolve()), + (options: NackOptions = {}) => + settle(async () => { + if (options.requeue ?? true) { + const addressed = envelope ?? { + ...createEnvelope(payload, { headers }, this.queueName), + messageId, + enqueuedAt: enqueuedAt.toISOString(), + }; + const dispatcher = await this.ensureDispatcher(); + await dispatcher.enqueue({ + ...addressed, + deliveryCount: options.preserveDeliveryCount + ? Math.max(0, deliveryCount - 1) + : deliveryCount, + }); + return; + } + const store = await this.ensureDeadLetterStore(); + await store.append(toDeadLetterRecord( + { + messageId, + queueName: this.queueName, + payload, + headers, + deliveryCount, + enqueuedAt, + }, + options.reason ?? 'nack_without_requeue', + options, + )); + }), ); } } diff --git a/packages/queue/adapters/kv-polling.adapter.ts b/packages/queue/adapters/kv-polling.adapter.ts index 37f8cf65be..ec3c098d8f 100644 --- a/packages/queue/adapters/kv-polling.adapter.ts +++ b/packages/queue/adapters/kv-polling.adapter.ts @@ -28,7 +28,13 @@ import type { NackOptions, } from '../ports/mod.ts'; import { QueueError, QueueErrorCode } from '../ports/mod.ts'; -import { toDeadLetterRecord } from './_envelope.ts'; +import { + KvPrefixes, + pollingPendingKey, + pollingProcessingKey, + type QueueMessage, +} from './_kv-polling/message.ts'; +import { acknowledgePollingMessage, rejectPollingMessage } from './_kv-polling/settlement.ts'; import { KvDeadLetterStore } from './kv-dead-letter-store.ts'; import { cancelPendingPage } from './_pending-retention.ts'; @@ -48,40 +54,6 @@ export type { } from '@netscript/kv'; export type { EnqueueOptions, ListenOptions, MessageContext, MessageQueue } from '../ports/mod.ts'; -/** - * Internal message structure stored in KV - */ -interface QueueMessage { - /** Unique message ID */ - id: string; - /** Message payload */ - payload: T; - /** Queue name */ - queue: string; - /** When the message was enqueued */ - enqueuedAt: string; - /** When the message becomes available (for delayed messages) */ - availableAt: string; - /** Number of delivery attempts */ - attempts: number; - /** Maximum retry attempts */ - maxAttempts: number; - /** Priority (higher = more important) */ - priority: number; - /** Custom headers */ - headers: Record; - /** Deduplication ID */ - deduplicationId?: string; - /** When the message was claimed for processing */ - claimedAt?: string; - /** Which worker claimed the message */ - claimedBy?: string; - /** Visibility timeout (when the claim expires) */ - visibilityTimeout?: string; - /** Originating durable identity, independent of payload/deduplication identity. */ - retentionId?: string; -} - /** * Options for creating a KvPollingAdapter */ @@ -154,20 +126,6 @@ export interface KvPollingAdapterOptions { deadLetterStore?: DeadLetterStorePort; } -/** - * KV key prefixes for queue data - */ -const KvPrefixes = { - /** Pending messages ready for processing */ - pending: 'queue:pending', - /** Messages currently being processed */ - processing: 'queue:processing', - /** Dead letter queue for failed messages */ - dlq: 'queue:dlq', - /** Deduplication tracking */ - dedup: 'queue:dedup', -} as const; - /** * KV Polling Queue Adapter * @@ -310,16 +268,14 @@ export class KvPollingAdapter implements MessageQueue { * Priority is inverted so higher priority sorts first. */ private pendingKey(priority: number, timestamp: string, id: string): KvKey { - // Invert priority so higher priority (100) sorts before lower (0) - const invertedPriority = String(1000 - priority).padStart(4, '0'); - return [KvPrefixes.pending, this.queueName, invertedPriority, timestamp, id]; + return pollingPendingKey(this.queueName, priority, timestamp, id); } /** * Create a KV key for a processing message. */ private processingKey(id: string): KvKey { - return [KvPrefixes.processing, this.queueName, id]; + return pollingProcessingKey(this.queueName, id); } /** @@ -429,7 +385,7 @@ export class KvPollingAdapter implements MessageQueue { key: KvKey, message: QueueMessage, versionstamp: string | null, - ): Promise { + ): Promise | null> { const now = new Date(); const visibilityExpires = new Date(now.getTime() + this.visibilityTimeout); @@ -438,33 +394,35 @@ export class KvPollingAdapter implements MessageQueue { attempts: message.attempts + 1, claimedAt: now.toISOString(), claimedBy: this.workerId, + claimReceipt: crypto.randomUUID(), visibilityTimeout: visibilityExpires.toISOString(), }; // Try atomic operation if available if (kv.atomic) { const result = await kv.atomic( - [{ key, versionstamp }], + [{ key, versionstamp }, { key: this.processingKey(message.id), versionstamp: null }], [ { type: 'delete', key }, { type: 'set', key: this.processingKey(message.id), value: claimedMessage }, ], ); - return result.ok; + return result.ok ? claimedMessage : null; } // Fallback: non-atomic (less safe but works with all KV backends) // Check if message still exists const current = await kv.get>(key); if (!current || !current.value || current.versionstamp !== versionstamp) { - return false; // Message was claimed by another worker + return null; // Message was claimed by another worker } - // Delete from pending and add to processing + if (await kv.get(this.processingKey(message.id))) return null; + // Non-atomic stores retain their single-consumer best-effort semantics. await kv.delete(key); await kv.set(this.processingKey(message.id), claimedMessage); - return true; + return claimedMessage; } /** @@ -488,12 +446,7 @@ export class KvPollingAdapter implements MessageQueue { if (claimed) { this.log(`Claimed message ${message.id}`); - return { - ...message, - attempts: message.attempts + 1, - claimedAt: now.toISOString(), - claimedBy: this.workerId, - }; + return claimed; } // Another worker claimed it, try next @@ -503,75 +456,26 @@ export class KvPollingAdapter implements MessageQueue { return null; } - /** - * Acknowledge successful message processing. - */ - private async ack(messageId: string): Promise { - const kv = await this.ensureKv(); - await kv.delete(this.processingKey(messageId)); - this.log(`Acked message ${messageId}`); + /** Settle only this live processing receipt. */ + private async ack(message: QueueMessage): Promise { + await acknowledgePollingMessage(await this.ensureKv(), this.queueName, message); } - /** - * Negative acknowledge - return message to queue or send to DLQ. - */ + /** Atomically hand back a receipt, preserving attempts only for controlled shutdown. */ private async nack( message: QueueMessage, options: NackOptions, + recovering = false, ): Promise { - const kv = await this.ensureKv(); - const now = new Date(); - const requeue = options.requeue ?? true; - - // Delete from processing - await kv.delete(this.processingKey(message.id)); - - if (!requeue || message.attempts >= message.maxAttempts) { - const store = await this.ensureDeadLetterStore(); - const reason = message.attempts >= message.maxAttempts - ? 'max_attempts_exceeded' - : options.reason ?? 'nack_without_requeue'; - - await store.append(toDeadLetterRecord( - { - messageId: message.id, - queueName: this.queueName, - payload: message.payload, - headers: message.headers, - deliveryCount: message.attempts, - enqueuedAt: message.enqueuedAt, - }, - reason, - options, - )); - - this.log(`Message ${message.id} sent to DLQ after ${message.attempts} attempts`); - } else { - // Requeue with exponential backoff - const backoffDelay = Math.min( - this.retryBaseDelay * Math.pow(2, message.attempts - 1), - this.retryMaxDelay, - ); - - const availableAt = new Date(now.getTime() + backoffDelay); - - const requeuedMessage: QueueMessage = { - ...message, - availableAt: availableAt.toISOString(), - claimedAt: undefined, - claimedBy: undefined, - visibilityTimeout: undefined, - }; - - await kv.set( - this.pendingKey(message.priority, availableAt.toISOString(), message.id), - requeuedMessage, - ); - - this.log( - `Message ${message.id} requeued with ${backoffDelay}ms delay (attempt ${message.attempts})`, - ); - } + await rejectPollingMessage( + await this.ensureKv(), + this.queueName, + message, + options, + { retryBaseDelay: this.retryBaseDelay, retryMaxDelay: this.retryMaxDelay }, + () => this.ensureDeadLetterStore(), + recovering, + ); } /** @@ -597,7 +501,7 @@ export class KvPollingAdapter implements MessageQueue { if (message.visibilityTimeout && new Date(message.visibilityTimeout) < now) { this.log(`Recovering timed-out message ${message.id}`); - await this.nack(message, { requeue: true }); + await this.nack(message, { requeue: true }, true); } } if (inspected < 100) this.recoveryStart = undefined; @@ -607,17 +511,20 @@ export class KvPollingAdapter implements MessageQueue { * Create a message context for the handler. */ private createContext(message: QueueMessage): MessageContext { + let settlement: Promise | undefined; + const settle = (operation: () => Promise): Promise => { + return settlement ??= operation().catch((error: unknown) => { + settlement = undefined; + throw error; + }); + }; return { messageId: message.id, deliveryCount: message.attempts, enqueuedAt: new Date(message.enqueuedAt), headers: message.headers, - ack: async () => { - await this.ack(message.id); - }, - nack: async (options?: { requeue?: boolean }) => { - await this.nack(message, options ?? { requeue: true }); - }, + ack: () => settle(() => this.ack(message)), + nack: (options: NackOptions = {}) => settle(() => this.nack(message, options)), }; } @@ -636,16 +543,20 @@ export class KvPollingAdapter implements MessageQueue { } this.running = true; - this.abortController = new AbortController(); + const controller = new AbortController(); + this.abortController = controller; const kv = await this.ensureKv(); const concurrency = options?.concurrency ?? 1; - const signal = options?.signal ?? this.abortController.signal; + const signal = options?.signal + ? AbortSignal.any([options.signal, controller.signal]) + : controller.signal; this.log(`Starting listener with concurrency ${concurrency}`); // Track active processing - const activeProcessing = new Set(); + const activeProcessing = new Set(); + let processingFailure: { error: unknown } | undefined; // Recovery loop - run periodically to recover timed-out messages const recoveryInterval = setInterval(async () => { @@ -675,7 +586,8 @@ export class KvPollingAdapter implements MessageQueue { } // Process message concurrently - activeProcessing.add(message.id); + const delivery = Symbol(message.id); + activeProcessing.add(delivery); (async () => { const context = this.createContext(message); @@ -683,18 +595,21 @@ export class KvPollingAdapter implements MessageQueue { try { await handler(message.payload, context); // Auto-ack on success - await this.ack(message.id); + await context.ack(); } catch (error) { console.error( `[KvPollingAdapter:${this.queueName}] Handler error for message ${message.id}:`, error, ); // Auto-nack with requeue on error - await this.nack(message, { requeue: true }); + await context.nack({ requeue: true }); } finally { - activeProcessing.delete(message.id); + activeProcessing.delete(delivery); } - })(); + })().catch((error: unknown) => { + processingFailure ??= { error }; + controller.abort(); + }); } } finally { clearInterval(recoveryInterval); @@ -710,6 +625,7 @@ export class KvPollingAdapter implements MessageQueue { this.running = false; this.log('Listener stopped'); } + if (processingFailure) throw processingFailure.error; } /** diff --git a/packages/queue/testing/memory-queue.ts b/packages/queue/testing/memory-queue.ts index 0a023e5a88..6c492ddd15 100644 --- a/packages/queue/testing/memory-queue.ts +++ b/packages/queue/testing/memory-queue.ts @@ -18,6 +18,7 @@ type MemoryQueueItem = { readonly headers: Record; deliveryCount: number; settled: boolean; + receipt?: symbol; }; /** @@ -243,19 +244,14 @@ export class MemoryQueueAdapter implements MessageQueue { handler: (message: T, context: MessageContext) => Promise, ): Promise { item.deliveryCount++; + item.receipt = Symbol(); const context = this.createContext(item); try { await handler(item.message, context); - const wasRequeued = this.isPending(item); - if (!item.settled && !wasRequeued) { - item.settled = true; - } + await context.ack(); } catch (error) { - const wasRequeued = this.isPending(item); - if (!item.settled && !wasRequeued) { - this.requeue(item); - } + await context.nack({ requeue: true }); throw error; } } @@ -264,35 +260,49 @@ export class MemoryQueueAdapter implements MessageQueue { * Build the message context passed to memory queue handlers. */ private createContext(item: MemoryQueueItem): MessageContext { + const receipt = item.receipt; + let settlement: Promise | undefined; + const settle = (operation: () => Promise): Promise => { + if (settlement) return settlement; + if (item.receipt !== receipt || item.settled) return Promise.resolve(); + return settlement = operation().catch((error: unknown) => { + settlement = undefined; + throw error; + }); + }; return { messageId: item.id, deliveryCount: item.deliveryCount, enqueuedAt: item.enqueuedAt, headers: item.headers, - ack: () => { - item.settled = true; - return Promise.resolve(); - }, - nack: async (options = {}) => { - item.settled = true; - if (options.requeue ?? true) { - this.requeue(item); - } - if (!(options.requeue ?? true)) { - await this.deadLetterStore.append(toDeadLetterRecord( - { - messageId: item.id, - queueName: 'memory', - payload: item.message, - headers: item.headers, - deliveryCount: item.deliveryCount, - enqueuedAt: item.enqueuedAt, - }, - options.reason ?? 'nack_without_requeue', - options, - )); - } - }, + ack: () => + settle(() => { + item.settled = true; + return Promise.resolve(); + }), + nack: (options = {}) => + settle(async () => { + if (options.requeue ?? true) { + if (options.preserveDeliveryCount) { + item.deliveryCount = Math.max(0, item.deliveryCount - 1); + } + this.requeue(item); + } else { + await this.deadLetterStore.append(toDeadLetterRecord( + { + messageId: item.id, + queueName: 'memory', + payload: item.message, + headers: item.headers, + deliveryCount: item.deliveryCount, + enqueuedAt: item.enqueuedAt, + }, + options.reason ?? 'nack_without_requeue', + options, + )); + item.settled = true; + } + }), }; } diff --git a/packages/queue/tests/kv-polling-dlq_test.ts b/packages/queue/tests/kv-polling-dlq_test.ts index a787e012b8..e60222b126 100644 --- a/packages/queue/tests/kv-polling-dlq_test.ts +++ b/packages/queue/tests/kv-polling-dlq_test.ts @@ -77,8 +77,12 @@ Deno.test('KvPollingAdapter routes terminal nacks through the dead-letter store' deadLetterStore, }) as unknown as KvPollingHarness<{ id: string }>; - await queue.nack(createMessage('msg-explicit', 1, 3), { requeue: false }); - await queue.nack(createMessage('msg-max', 3, 3), { requeue: true }); + const explicit = createMessage('msg-explicit', 1, 3); + const exhausted = createMessage('msg-max', 3, 3); + await kv.set(['queue:processing', 'jobs', explicit.id], explicit); + await kv.set(['queue:processing', 'jobs', exhausted.id], exhausted); + await queue.nack(explicit, { requeue: false }); + await queue.nack(exhausted, { requeue: true }); assertEquals(deadLetterStore.records.map((record) => record.reason), [ 'nack_without_requeue', diff --git a/packages/queue/tests/listener-drain-owner_test.ts b/packages/queue/tests/listener-drain-owner_test.ts new file mode 100644 index 0000000000..f2640ad9a5 --- /dev/null +++ b/packages/queue/tests/listener-drain-owner_test.ts @@ -0,0 +1,80 @@ +import { assertEquals } from '@std/assert'; +import { delay } from '@std/async'; +import { MemoryKvAdapter } from '@netscript/kv'; +import { KvPollingAdapter } from '../adapters/kv-polling.adapter.ts'; +import { RedisAdapter } from '../adapters/redis.adapter.ts'; +import type { MessageContext, MessageQueue } from '../ports/mod.ts'; +import { bounded } from './_fixtures/redis-lease-harness.ts'; + +for ( + const [kind, url] of [ + ['KV polling', undefined], + ['Redis', Deno.env.get('NETSCRIPT_TEST_REDIS_URL')], + ['Garnet', Deno.env.get('NETSCRIPT_TEST_GARNET_URL')], + ] as const +) { + for (const owner of [false, true]) { + Deno.test({ + name: `${kind} ${ + owner ? 'lets the worker own admitted work' : 'keeps its default bounded drain' + } after intake cancellation`, + ignore: kind !== 'KV polling' && !url, + async fn() { + await using kv = new MemoryKvAdapter(); + const queue: MessageQueue = kind === 'KV polling' + ? new KvPollingAdapter({ + kv, + queueName: crypto.randomUUID(), + visibilityTimeout: 50, + pollInterval: 1, + }) + : new RedisAdapter(url!, `drain-owner-${crypto.randomUUID()}`); + await queue.enqueue('active'); + const entered = Promise.withResolvers(); + const release = Promise.withResolvers(); + const callbackFinished = Promise.withResolvers(); + const intake = new AbortController(); + let listenerFinished = false; + // Executable on the previous contract: unknown fields go through a variable. + const options = { + signal: intake.signal, + visibilityTimeout: 50, + ...(owner ? { drainTimeoutMs: Infinity } : {}), + }; + const done = queue.listen(async (_message, context) => { + entered.resolve(context); + try { + await release.promise; + await context.ack(); + } finally { + callbackFinished.resolve(); + } + }, options).finally(() => listenerFinished = true); + void done.catch(() => {}); + try { + const context = await bounded(entered.promise); + if (context.extendVisibility) assertEquals(await context.extendVisibility(2_000), true); + intake.abort(); + await delay(500); + assertEquals( + listenerFinished, + !owner, + 'The backend ended the pause barrier before the handler finished.', + ); + if (owner && context.extendVisibility) { + assertEquals(await context.extendVisibility(2_000), true); + } + } finally { + intake.abort(); + release.resolve(); + await bounded(callbackFinished.promise); + try { + await bounded(done); + } finally { + await queue.stop(); + } + } + }, + }); + } +} diff --git a/packages/queue/tests/polling-settlement-outage_test.ts b/packages/queue/tests/polling-settlement-outage_test.ts new file mode 100644 index 0000000000..42afe7cd6e --- /dev/null +++ b/packages/queue/tests/polling-settlement-outage_test.ts @@ -0,0 +1,32 @@ +import { assertEquals, assertRejects } from '@std/assert'; +import { stub } from '@std/testing/mock'; +import { MemoryKvAdapter } from '@netscript/kv'; +import { KvPollingAdapter } from '../adapters/kv-polling.adapter.ts'; +import { bounded } from './_fixtures/redis-lease-harness.ts'; + +Deno.test('polling settlement failure rejects the listener and retains the durable delivery', async () => { + await using kv = new MemoryKvAdapter(); + const queue = new KvPollingAdapter({ kv, queueName: 'settlement-outage', pollInterval: 1 }); + await queue.enqueue('retain'); + const entered = Promise.withResolvers(); + const settle = Promise.withResolvers(); + const controller = new AbortController(); + const done = queue.listen(async (_message, delivery) => { + entered.resolve(); + await settle.promise; + await delivery.nack({ preserveDeliveryCount: true }); + }, { signal: controller.signal }); + void done.catch(() => {}); + try { + await bounded(entered.promise); + using _broken = stub(kv, 'atomic', () => Promise.reject(new Error('settlement outage'))); + settle.resolve(); + await assertRejects(() => bounded(done), Error, 'settlement outage'); + assertEquals(await queue.getStats(), { pending: 0, processing: 1, dlq: 0 }); + } finally { + controller.abort(); + settle.resolve(); + await bounded(done).catch(() => {}); + await queue.stop(); + } +}); diff --git a/packages/queue/tests/shutdown-refund-local_test.ts b/packages/queue/tests/shutdown-refund-local_test.ts new file mode 100644 index 0000000000..d32fb6a9bf --- /dev/null +++ b/packages/queue/tests/shutdown-refund-local_test.ts @@ -0,0 +1,134 @@ +import { assertEquals } from '@std/assert'; +import { MemoryKvAdapter } from '@netscript/kv'; +import { DenoKvAdapter } from '../adapters/deno-kv.adapter.ts'; +import { KvPollingAdapter } from '../adapters/kv-polling.adapter.ts'; +import { MemoryDeadLetterStore, MemoryQueueAdapter } from '../testing/memory-queue.ts'; +import type { MessageContext, MessageQueue } from '../ports/mod.ts'; +import { bounded } from './_fixtures/redis-lease-harness.ts'; + +async function queues( + kind: string, + test: (create: () => MessageQueue, dead: MemoryDeadLetterStore) => Promise, +): Promise { + await using kv = new MemoryKvAdapter(); + await using native = kind === 'Deno KV' ? await Deno.openKv(':memory:') : null; + const dead = new MemoryDeadLetterStore(); + const memory = new MemoryQueueAdapter({ deadLetterStore: dead }); + const queueName = crypto.randomUUID(); + await test( + () => + native + ? new DenoKvAdapter({ kv: native, queueName, deadLetterStore: dead }) + : kind === 'memory' + ? memory + : new KvPollingAdapter({ + kv, + queueName, + deadLetterStore: dead, + maxRetries: 0, + pollInterval: 1, + }), + dead, + ); +} + +function hold(queue: MessageQueue) { + const entered = Promise.withResolvers<{ message: string; delivery: MessageContext }>(); + const release = Promise.withResolvers(); + const stop = new AbortController(); + const done = queue.listen(async (message, delivery) => { + entered.resolve({ message, delivery }); + await release.promise; + }, { signal: stop.signal }); + void done.catch(() => {}); + return { + claim: () => bounded(entered.promise), + close: async () => { + stop.abort(); + release.resolve(); + try { + await bounded(done); + } finally { + await queue.stop(); + } + }, + }; +} + +for (const kind of ['memory', 'KV polling', 'Deno KV']) { + Deno.test(`${kind} duplicate handback and old ack cannot affect the replacement`, () => + queues(kind, async (create, dead) => { + const original = create(); + await original.enqueue('original'); + const holds: ReturnType[] = []; + try { + const first = hold(original); + holds.push(first); + const { delivery: old } = await first.claim(); + await Promise.all([ + old.nack({ preserveDeliveryCount: true }), + old.nack({ preserveDeliveryCount: true }), + ]); + await old.ack(); + await first.close(); + const second = hold(create()); + holds.push(second); + const next = await second.claim(); + assertEquals(next.delivery.messageId, old.messageId); + assertEquals(next.delivery.deliveryCount, 1); + await old.nack({ preserveDeliveryCount: true }); + await old.ack(); + await next.delivery.ack(); + await second.close(); + const probe = create(); + await probe.enqueue('sentinel'); + const last = hold(probe); + holds.push(last); + const received = await last.claim(); + assertEquals( + received.message, + 'sentinel', + 'A stale context duplicated the completed message.', + ); + assertEquals(received.delivery.deliveryCount, 1); + await received.delivery.ack(); + assertEquals(await dead.depth(), 0); + } finally { + for (const item of holds) await item.close(); + } + })); + Deno.test(`${kind} automatic success closes retained contexts to late handback`, () => + queues(kind, async (create, dead) => { + const firstQueue = create(); + await firstQueue.enqueue('completed'); + const first = hold(firstQueue); + const holds = [first]; + try { + const { delivery: old } = await first.claim(); + await first.close(); + await old.nack({ preserveDeliveryCount: true }); + const probe = create(); + await probe.enqueue('sentinel'); + const last = hold(probe); + holds.push(last); + const next = await last.claim(); + assertEquals(next.message, 'sentinel'); + await next.delivery.ack(); + assertEquals(await dead.depth(), 0); + } finally { + for (const item of holds) await item.close(); + } + })); + Deno.test(`${kind} preservation cannot turn a terminal nack into a retry`, () => + queues(kind, async (create, dead) => { + const queue = create(); + await queue.enqueue('terminal'); + const active = hold(queue); + try { + await (await active.claim()).delivery.nack({ requeue: false, preserveDeliveryCount: true }); + assertEquals(await dead.depth(), 1); + } finally { + await active.close(); + } + })); +} From 031332fb97d03407706d43b403396024999a5c88 Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Tue, 17 Nov 2026 00:01:47 +0100 Subject: [PATCH 07/20] docs(harness): retain listener drain ownership approval --- .../drift.md | 3 + .../plan-eval.md | 56 +++++++++++++++++++ .../plan-intake-drain-addendum.md | 2 +- .../worklog.md | 5 ++ 4 files changed, 65 insertions(+), 1 deletion(-) diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md index b4aad2b777..07c12e4837 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md @@ -10,3 +10,6 @@ Plan-evaluator corrections: actual docs path tune-worker-runtime.md, not a direc S3a stays within approved adapter scope. Its handback storage-failure test exposed an unjoined callback rejection; red fault test precedes the error-propagation fix, which keeps the original processing entry durable and fails the listener. Native listener post-intake expiry can resolve a pause generation before handlers finish;3real/scaled owner cases red with3default controls green. That rework-forcing contract discovery is stopped at a separate plan checkpoint, with exact proposal/files/gates in plan-intake-drain-addendum.md. No new-option implementation yet. + + +Follow-up PLAN-EVAL cycle2 settlement: addendum risk text now pins the shared name with the already approved Worker-total stop option (Listener JSDoc/S6 docs must cross-reference the per-listener post-intake cutoff against the unchanged25s Worker total). Wording only, settled in place under the cycle2 authorization; no product/tests or design change. Original Plan-Gate PASS history preserved; follow-up verdict PASS appended to plan-eval.md at014730e1. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-eval.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-eval.md index bf7207ee5b..44ea73e289 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-eval.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-eval.md @@ -66,3 +66,59 @@ All nine acceptance boxes (#2633 boxes 1–4, #2635 boxes 1–5) are mapped to c - No resources were launched for this evaluation; this run's research backends and volumes were cleaned by proven identity per leak-report.md. Raw reproduction reports and diagnostics remain private to the operator; only repo-relative public evidence is used here. PR https://github.com/rickylabs/netscript/pull/2645, head 89effd610f04deba89f08149f5d36ed822c6eb27, checks and reviews: required CI policy-skipped (draft, command-postgres green), no reviews yet. + +--- + +# Follow-up PLAN-EVAL — intake drain ownership addendum (cycle 2) + +- Follow-up evaluator session: independent GLM 5.3 provider-default (OpenCode) session, opened 2026-11-16, verdict 2026-11-17, fresh per T3 (not the generator or any prior evaluator session) +- Run: feat-worker-pause-drain--issues-2633-2635 +- Scope: ONLY the added ownership contract in `plan-intake-drain-addendum.md` plus continued original-plan acceptance/gate coverage; the still-in-progress implementation (S2/S3a context evidence) is not being approved or rejected here, and no implementation PASS or ready state is claimed +- Evaluated head: `014730e1ca4e155fe1059054dc46d4d1dd75603d` on feat/worker-pause-drain; the original Plan-Gate PASS above at `89effd610` is preserved unchanged +- Route attestation: live matrix lookup (feature tier, plan evaluator) confirms primary `glm_5_3/provider_default`, fallback `fable_5_1/low`; requested pinned primary observed exactly in this fresh session, options{}; fallback unused, no substitution. Generator: Codex gpt-6.1-sol/xhigh (OpenAI vendor family), separate session and separate vendor family. Feature tier, cycle 2 of 2. + +## Addendum scope verification (load-bearing claims reverified in tree at the exact head) + +- `packages/queue/adapters/_redis-listener.ts` — `stop()` arms `drainTimer` at `policy.drainTimeout`; expiry aborts the settlement/receipt controller, closes leases and resolves `expired`; `run()`'s finally races callback settlement (`allSettled`) against `expired`, so the listener generation ends at the visibility-derived timeout even with a held callback. `redisListenerPolicy` (`_redis-lease-state.ts`) derives `drainTimeout` from `options?.visibilityTimeout ?? 30_000`. Addendum claim confirmed in source. +- `packages/queue/adapters/kv-polling.adapter.ts` — `listen()`'s finally wait is bounded by the constructor `visibilityTimeout` with 100 ms poll steps, then resolves with active processing possibly remaining. Addendum claim confirmed; the excluded 200 ms probe is legitimately near-cadence and the 500 ms case settles it. +- Independent re-run of the committed `packages/queue/tests/listener-drain-owner_test.ts` KV polling half at the exact head (in-memory KV only; the lane's retained backend containers untouched by this review): default-bounded-drain control green, owner case red at exactly the pause-barrier assertion. Matches the lane's recorded six-case result: 3 owner-option reds (KV polling, Redis, Garnet), 3 ordinary-default controls green, 0 ignored, one unique assertion. The proposed option passes through a variable against the old public contract and is ignored there, as claimed. +- Worker boundary: the already-approved S2 core drain primitive enforces one monotonic total budget (`started + budget.totalMs` with work/settlement slices) and rejects `TimeoutError` on hung close/cleanup; the three forwarding call sites for jobs/tasks/triggers exist (`worker.ts` job listen, `queue-consumer.ts` task and trigger listens), so S4 forwarding and the unchanged-total claims are meetable exactly as locked. +- Omission-behavior safety: `ListenOptions` currently has no `drainTimeoutMs` field; no native adaptation, Worker lifecycle/config, runner-default or AMQP changes exist at this head — the mandatory stop before implementing the rework-forcing public option is honored. +- Provider exemption rationale verified: Postgres joins callbacks inline in its listen loop, Deno KV `stop()` awaits the listener completion, memory joins per its committed S3a contracts; the planned AMQP native-channel design has no independent cutoff. No new behavior is needed there, and AMQP stays decoupled as its own subsequent slice. +- Implementation-lane S2/S3a gate numbers (Redis/Garnet/Postgres 28/28, budget/PG 8/8, S3a handback/fence/adjacent suites, doc-lint baseline unchanged) are recorded by the lane and used as context only; not re-run and not approved by this verdict. + +## Checklist results (8 boxes scoped to the addendum) + +| Plan-Gate item (addendum scope) | Result | Evidence / location | +| ----------------------------------------- | ------ | ------------------- | +| Research present and current | PASS | Addendum documents the discovery with a live six-case reproduction at the exact head (3 red owner cases / 3 green default controls / 0 ignored), honest 200 ms near-cadence probe exclusion, and the extended-2 s lease oracle separating listener drain from claim expiry; the committed red test is in-tree executable evidence, reverified independently this session | +| Decisions locked | PASS | Addendum "Locked additional contract": field home (`ListenOptions.drainTimeoutMs` in queue ports), provider scope (Redis/Garnet + KV polling), value domain (finite positive safe integers through 2,147,483,647 or +Infinity; NaN/zero/negative/fractional/excessive rejected before listener allocation), omission semantics, Infinity delegated to a bounded outer owner, Redis "do not arm the post-intake timer" policy, KV polling finally-wait budget, visibility/drain separation, pause-raced committed claim stays an ordinary charged requeue, draining handback stays shutdown-only | +| Open-decision sweep | PASS | Addendum states no open design decision; evaluator-run sweep found none rework-forcing: explicit-Infinity direct consumers opt in knowingly (default finite unchanged), non-scoped adapters ignore the field per the existing provider-specific-field convention, the worker total/stop math is untouched, and config stays forwarding-only | +| Commit slices | PASS | S3b additional group ≤10 files (4 named source files + the committed owner test + existing Redis listener policy/lease tests, verified to exist + up to 3 run docs), contract-first; AMQP as its own subsequent <30 slice; S4 proofs added to the already-named worker/listener test group (<30); S6 +1 docs file (<30); S7 unchanged; all slices remain <30 and name proofs plus proving gates | +| Risk register | PASS | Default-unbounded-stop risk pinned by 3 negative controls plus retained native stop/reclaim regressions; visibility-conflation risk pinned by explicit JSDoc, policy separation and the extended-live-lease oracle; worker forwarding pinned by explicit forwarding plus noncooperative-budget tests; same-name hazard now settled below; no new dependency/lock/command/root surface | +| Gate set selected | PASS | Original Archetype 3 + overlays gate set unchanged; the addendum only adds scoped proofs (6/6 owner/default controls, invalid-values-allocate-no-resources, retained default finite stop/expiry/claim/outage regressions, S4 forwarding/long-pause/stop-bound proofs, S6 docs) and states no gate weakened or replaced — verified against S1–S7 | +| Deferred scope explicit | PASS | Addendum enumerates its non-changes (no Streams, no saga transport, no generic pause/transport abstraction, no lease/visibility semantics change, no new exclusivity guarantee, AMQP decoupled); original deferred list from plan.md stands | +| jsr-audit surface scan | PASS | One optional primitive numeric field through the existing queue ports interface follows the existing provider-specific-field pattern (JSDoc, others ignore); no new exports/subpaths/dependencies/slow types; S3b/S7 doc-lint zero-new-diagnostics against the exact per-entrypoint baseline plus publish dry-run/audit and consumer gates retained | + +## Acceptance coverage + +9/9 remains. #2633 boxes 1–4 and #2635 boxes 1–5 keep the approved mapping from plan.md "Acceptance evidence to produce"; the addendum adds evidence obligations inside #2635 box1 (pause keeps admitted job/task/trigger work alive beyond the former native cutoffs) and box2 (drain success and stop stay within the worker total) without altering any acceptance box. + +## Repairs settled in place (cycle 2 authorization) + +1. Addendum risk paragraph now requires the Listener field's JSDoc and the S6 docs to cross-reference the per-listener post-intake cutoff against the already approved, same-named Worker-total stop option (unchanged 25 s total). Wording only, settled in place rather than bounced back; recorded in `drift.md`. No product/tests or design change. + +## Findings + +None blocking. The six-case control design, the honest near-cadence probe exclusion, and the separation of listener drain from claim expiry (2 s extended lease with a re-checked usable heartbeat) were verified and stand as sound future-contract evidence. The committed red owner cases must not be recorded as false greens — the addendum already commits to exactly that, and S3b must prove 6/6 green once the contract lands. + +## Notes + +- Feature tier cycle 2/2, narrow implementation-discovery checkpoint after the cycle-1 PASS. On this PASS the generator may begin S3b under the approved slice/gate set. The three positively owned implementation backend containers retained by the lane were not touched, launched or cleaned by this review; no resources were used beyond an in-memory KV re-run. +- Raw red reports and diagnostics remain private to the operator; only repo-relative public evidence is used here. PR remains draft with `status:plan-eval`; new owner-option tests and original Worker/config reds are intentionally pending, so no ready transition, merge or publication is implied, and final independent IMPL-EVAL plus S7 gates remain required. + +## Verdict + +`PASS` (follow-up; the added ownership contract and continued acceptance/gate coverage are approved; not an implementation approval) + +PR https://github.com/rickylabs/netscript/pull/2645, head 014730e1ca4e155fe1059054dc46d4d1dd75603d, checks and reviews: required CI policy-skipped (draft skip labels; command-postgres green), no reviews yet. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-intake-drain-addendum.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-intake-drain-addendum.md index 5f57e18877..5a2ddf67a8 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-intake-drain-addendum.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/plan-intake-drain-addendum.md @@ -28,4 +28,4 @@ S6 update queue README + worker/core/tuning docs for provider-specific post-inta ## Risk / open decisions -No open design decision. Required before code: independent feature PLAN-EVAL follow-up for this added public option (same primary route, fresh session per T3). The main risk is accidentally making default queue stop unbounded:3negative controls and retained native stop/reclaim tests pin omission behavior. The companion risk is conflating visibility with drain waiting: explicit JSDoc, policy separation and extended-live-lease oracle pin it. Worker must never pass this option without the separately enforced total drain; explicit forwarding plus noncooperative budget tests pin that boundary. Existing visibility limitations of providers without extendVisibility remain unchanged and documented; the option promises callback joining, not stronger delivery exclusivity. No new dependency/lock/command or root surface; primitive numeric field adds no slow type/private reference risk. +No open design decision. Required before code: independent feature PLAN-EVAL follow-up for this added public option (same primary route, fresh session per T3). The main risk is accidentally making default queue stop unbounded:3negative controls and retained native stop/reclaim tests pin omission behavior. The companion risk is conflating visibility with drain waiting: explicit JSDoc, policy separation and extended-live-lease oracle pin it. A same-name hazard: this Listener field shares its name with the already approved Worker-total stop option, so its JSDoc and the S6 docs must cross-reference the per-listener post-intake cutoff against that unchanged25s Worker total. Worker must never pass this option without the separately enforced total drain; explicit forwarding plus noncooperative budget tests pin that boundary. Existing visibility limitations of providers without extendVisibility remain unchanged and documented; the option promises callback joining, not stronger delivery exclusivity. No new dependency/lock/command or root surface; primitive numeric field adds no slow type/private reference risk. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index 7b37d166b6..a38e87489e 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -39,3 +39,8 @@ Gates:9/9local handback/count controls;9/9duplicate/terminal/late-context guards ## Pause-barrier discovery and scoped plan follow-up Running listener-drain-owner_test.ts reveals Redis/Garnet and KV polling resolve their listener after a native visibility-based wait even while the callback remains active.500msheld callback vs50msnative limit:3proposed-owner cases red,3ordinary-default controls green,0ignored. The initial200ms polling observation was near the100ms cadence and is excluded. This would invalidate a long pause barrier despite independent execution signals. Added plan-intake-drain-addendum.md locks an explicit optional listener drain owner budget (Infinity only under the Worker's separately bounded25s total), default finite behavior unchanged. No added-option implementation before independent PLAN-EVAL follow-up. No AMQP or Worker lifecycle implementation yet. + + +## Listener ownership addendum approved + +[Fresh GLM5.3/provider-default PLAN-EVAL PASS](https://github.com/rickylabs/netscript/pull/2645#issuecomment-6931962078) approves the added optional listener drain ownership contract at014730e1c. Independent KV polling default control green / owner case red, source-verified native Redis timeout, core worker25s total boundary and all three forwarding sites;8/8Plan-Gate,9/9continued acceptance. Evaluator settled one wording repair: docs/JSDoc distinguish per-listener post-intake cutoff from same-named Worker total stop option. No implementation waiver or gate reduction. Retain verdict; implement S3b contract/native adapters next. Corrected the verdict's final-line head transcription to the exact evaluated head already stated in its metadata. From 0fbed1b49658bc6e46facb99b2df2694284db227 Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Tue, 17 Nov 2026 00:11:39 +0100 Subject: [PATCH 08/20] feat(queue): allow bounded owners to retain admitted deliveries --- .../worklog.md | 7 +++++ packages/queue/adapters/_redis-lease-state.ts | 5 +++- packages/queue/adapters/_redis-listener.ts | 12 ++++---- packages/queue/adapters/kv-polling.adapter.ts | 12 +++++++- packages/queue/ports/message-queue.ts | 8 +++++ .../queue/tests/listener-drain-policy_test.ts | 30 +++++++++++++++++++ 6 files changed, 67 insertions(+), 7 deletions(-) create mode 100644 packages/queue/tests/listener-drain-policy_test.ts diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index a38e87489e..d4b4a67e54 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -44,3 +44,10 @@ Running listener-drain-owner_test.ts reveals Redis/Garnet and KV polling resolve ## Listener ownership addendum approved [Fresh GLM5.3/provider-default PLAN-EVAL PASS](https://github.com/rickylabs/netscript/pull/2645#issuecomment-6931962078) approves the added optional listener drain ownership contract at014730e1c. Independent KV polling default control green / owner case red, source-verified native Redis timeout, core worker25s total boundary and all three forwarding sites;8/8Plan-Gate,9/9continued acceptance. Evaluator settled one wording repair: docs/JSDoc distinguish per-listener post-intake cutoff from same-named Worker total stop option. No implementation waiver or gate reduction. Retain verdict; implement S3b contract/native adapters next. Corrected the verdict's final-line head transcription to the exact evaluated head already stated in its metadata. + + +## S3b listener drain ownership + +Approved optional ListenOptions.drainTimeoutMs separates Redis/KV-polling post-intake waiting from worker total stop. Omission preserves native visibility-based finite behavior. Positive Infinity leaves admitted callbacks/receipts alive for an outer bounded owner; no Infinity timer is scheduled. Invalid finite/nonfinite values rejected before allocation. Listener JSDoc explicitly cross-references WorkerOptions.drainTimeoutMs, whose total remains finite25s by default. No visibility/claim/reclaim/Lua change. + +Previously red owner/default controls now6/6green on real Redis/Garnet and KV polling; combined with existing native lease suite16pass/0fail/1pre-existingCI-only ignore. Policy validation/no-allocation1/1green; scoped check/lint/fmt0, release:preflight/docs:accuracyPASS. AMQP work is independent and follows in its own commit; Worker forwarding is S4. diff --git a/packages/queue/adapters/_redis-lease-state.ts b/packages/queue/adapters/_redis-lease-state.ts index c7bfec5212..b64234c199 100644 --- a/packages/queue/adapters/_redis-lease-state.ts +++ b/packages/queue/adapters/_redis-lease-state.ts @@ -83,7 +83,10 @@ const DELAYED_POLL_INTERVAL_ENV = 'NETSCRIPT_REDIS_DELAYED_POLL_INTERVAL_MS'; export function redisListenerPolicy(options?: ListenOptions): RedisListenerPolicy { return { concurrency: leaseDuration(options?.concurrency ?? 1, 'concurrency'), - drainTimeout: leaseDuration(options?.visibilityTimeout ?? 30_000, 'visibilityTimeout'), + drainTimeout: options?.drainTimeoutMs === Infinity ? Infinity : leaseDuration( + options?.drainTimeoutMs ?? options?.visibilityTimeout ?? 30_000, + options?.drainTimeoutMs === undefined ? 'visibilityTimeout' : 'drainTimeoutMs', + ), pollIntervalMs: pollInterval( options?.pollIntervalMs, POLL_INTERVAL_ENV, diff --git a/packages/queue/adapters/_redis-listener.ts b/packages/queue/adapters/_redis-listener.ts index 0d6e4e330b..0b6a1636ad 100644 --- a/packages/queue/adapters/_redis-listener.ts +++ b/packages/queue/adapters/_redis-listener.ts @@ -27,11 +27,13 @@ export class RedisListener { this.intake.abort(); this.wake(); if (!this.ready) this.closeLeases(); - this.drainTimer = setTimeout(() => { - this.settlement.abort(); - this.closeLeases(); - this.expired.resolve(); - }, this.policy.drainTimeout); + if (this.policy.drainTimeout !== Infinity) { + this.drainTimer = setTimeout(() => { + this.settlement.abort(); + this.closeLeases(); + this.expired.resolve(); + }, this.policy.drainTimeout); + } }; /** Admit no more than the configured number of handler-plus-settlement operations. */ diff --git a/packages/queue/adapters/kv-polling.adapter.ts b/packages/queue/adapters/kv-polling.adapter.ts index ec3c098d8f..c9c645f8cc 100644 --- a/packages/queue/adapters/kv-polling.adapter.ts +++ b/packages/queue/adapters/kv-polling.adapter.ts @@ -542,6 +542,16 @@ export class KvPollingAdapter implements MessageQueue { ); } + const drainTimeout = options?.drainTimeoutMs ?? this.visibilityTimeout; + if ( + drainTimeout !== Infinity && + (!Number.isSafeInteger(drainTimeout) || drainTimeout < 1 || drainTimeout > 2_147_483_647) + ) { + throw new QueueError( + 'drainTimeoutMs must be a positive integer through 2147483647 or Infinity.', + QueueErrorCode.CONFIGURATION_ERROR, + ); + } this.running = true; const controller = new AbortController(); this.abortController = controller; @@ -615,7 +625,7 @@ export class KvPollingAdapter implements MessageQueue { clearInterval(recoveryInterval); // Wait for active processing to complete (with timeout) - const timeout = this.visibilityTimeout; + const timeout = drainTimeout; const startTime = Date.now(); while (activeProcessing.size > 0 && Date.now() - startTime < timeout) { diff --git a/packages/queue/ports/message-queue.ts b/packages/queue/ports/message-queue.ts index 64b35ad9ec..9c5c2cb632 100644 --- a/packages/queue/ports/message-queue.ts +++ b/packages/queue/ports/message-queue.ts @@ -195,6 +195,14 @@ export interface EnqueueOptions { * Options for listening to messages. */ export interface ListenOptions { + /** + * Redis/KV-polling wait for admitted callbacks after intake cancellation. + * Defaults to the backend visibility timeout; positive Infinity delegates the + * wait to an outer owner. This is separate from WorkerOptions.drainTimeoutMs, + * which bounds the whole worker stop; workers retain that total deadline. + */ + drainTimeoutMs?: number; + /** * Number of messages to process concurrently. * Default: 1 (sequential processing) diff --git a/packages/queue/tests/listener-drain-policy_test.ts b/packages/queue/tests/listener-drain-policy_test.ts new file mode 100644 index 0000000000..1765317d2e --- /dev/null +++ b/packages/queue/tests/listener-drain-policy_test.ts @@ -0,0 +1,30 @@ +import { assertEquals, assertRejects, assertThrows } from '@std/assert'; +import { spy } from '@std/testing/mock'; +import { MemoryKvAdapter } from '@netscript/kv'; +import { redisListenerPolicy } from '../adapters/_redis-lease-state.ts'; +import { KvPollingAdapter } from '../adapters/kv-polling.adapter.ts'; +import { QueueError } from '../ports/errors.ts'; + +Deno.test('listener drain policy validates before resources and keeps the old default', async () => { + assertEquals(redisListenerPolicy({ visibilityTimeout: 123 }).drainTimeout, 123); + assertEquals(redisListenerPolicy({ drainTimeoutMs: Infinity }).drainTimeout, Infinity); + assertEquals(redisListenerPolicy({ drainTimeoutMs: 456 }).drainTimeout, 456); + await using kv = new MemoryKvAdapter(); + const queue = new KvPollingAdapter({ kv }); + using intervals = spy(globalThis, 'setInterval'); + for (const drainTimeoutMs of [NaN, -Infinity, -1, 0, 1.5, 2_147_483_648]) { + assertThrows(() => redisListenerPolicy({ drainTimeoutMs }), QueueError); + await assertRejects( + () => queue.listen(() => Promise.resolve(), { drainTimeoutMs }), + QueueError, + ); + } + assertEquals(intervals.calls.length, 0); + const cancelled = new AbortController(); + cancelled.abort(); + await queue.listen(() => Promise.resolve(), { + signal: cancelled.signal, + drainTimeoutMs: Infinity, + }); + await queue.stop(); +}); From 0483eee49e7db26d5fce289f8bdb929e2d0683f8 Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Tue, 17 Nov 2026 00:19:57 +0100 Subject: [PATCH 09/20] fix(queue): confirm AMQP handback before acknowledging deliveries --- .../leak-report.md | 2 + .../worklog.md | 7 + packages/queue/adapters/amqp.adapter.ts | 322 ++++++++++++------ 3 files changed, 222 insertions(+), 109 deletions(-) diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md index 6b19994d4d..acb39ed740 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md @@ -5,3 +5,5 @@ Commands: maint:leak-check and maint:teardown preview against this run and workt ## Active implementation resources Three run-created Redis/Garnet/Postgres backend containers are intentionally retained for S2/S3 validation. Immutable IDs and native mount identities recorded privately at creation, with the dedicated implementation test-owner label. No AppHost or foreign container/process started or stopped. Final cleanup will preview the repo teardown tool and remove only those exact proven identities/mounts; the research zero-survivor result above is historical, not a claim that these active implementation resources are absent. + +AMQP validation adds one positively owned RabbitMQ backend; active implementation set is four containers with recorded immutable IDs/mounts. No foreign resources touched. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index d4b4a67e54..d6acb1c3bc 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -51,3 +51,10 @@ Running listener-drain-owner_test.ts reveals Redis/Garnet and KV polling resolve Approved optional ListenOptions.drainTimeoutMs separates Redis/KV-polling post-intake waiting from worker total stop. Omission preserves native visibility-based finite behavior. Positive Infinity leaves admitted callbacks/receipts alive for an outer bounded owner; no Infinity timer is scheduled. Invalid finite/nonfinite values rejected before allocation. Listener JSDoc explicitly cross-references WorkerOptions.drainTimeoutMs, whose total remains finite25s by default. No visibility/claim/reclaim/Lua change. Previously red owner/default controls now6/6green on real Redis/Garnet and KV polling; combined with existing native lease suite16pass/0fail/1pre-existingCI-only ignore. Policy validation/no-allocation1/1green; scoped check/lint/fmt0, release:preflight/docs:accuracyPASS. AMQP work is independent and follows in its own commit; Worker forwarding is S4. + + +## Original S3 AMQP handback + +The pinned provider's non-native async path double-acknowledged; replace consumption with existing upstream amqplib basic.cancel/prefetch and one per-receipt settlement boundary. Configure the existing Fedify enqueue path with the actual queue name and native retry. Preserve/count the same envelope on an owned confirm publisher and await broker confirmation before old ack. Native error/cancel/close callbacks are owned, retained contexts cannot publish through a closed receipt, and stop closes the owned connection even after listen already completed. No new dependency or fake provider seam. + +Real backend handback suite23/23 executes all seven backend variants (Redis/Garnet/Postgres/AMQP/DenoKV/KVpoll/memory),8shutdown restarts, explicit and actual thrown failures plus native DLQ tests,0ignored. After channel-edge cleanup strengthening, focused AMQP/policy5/5green. Scoped check/lint/fmt0, new private method JSDoc supplied; queue combined doclint1/exact per-entrypoint baseline unchanged, no newdiagnostics. Preflight/accuracyPASS. Four owned implementation backends now retained, each immutable ID and mount recorded privately; no foreign cleanup. Worker integration follows S4. diff --git a/packages/queue/adapters/amqp.adapter.ts b/packages/queue/adapters/amqp.adapter.ts index 930ea75af6..929c14d636 100644 --- a/packages/queue/adapters/amqp.adapter.ts +++ b/packages/queue/adapters/amqp.adapter.ts @@ -10,6 +10,7 @@ import { AmqpMessageQueue } from '@fedify/amqp'; // Fedify 2.4 uses this type surface; its channel operations remain supported by amqplib 2. // @ts-types="npm:@types/amqplib@^0.10.7" import { connect } from 'amqplib'; +import { Buffer } from 'node:buffer'; import type { DeadLetterStorePort, EnqueueOptions, @@ -23,12 +24,18 @@ import { createEnvelope, createMessageContext, isMessageEnvelope, + type MessageEnvelope, toDeadLetterRecord, } from './_envelope.ts'; import { KvDeadLetterStore } from './kv-dead-letter-store.ts'; export type { EnqueueOptions, ListenOptions, MessageContext, MessageQueue } from '../ports/mod.ts'; +type AmqpConnection = Awaited>; +type AmqpChannel = Awaited>; +type ConfirmChannel = Awaited>; +type NativeDelivery = NonNullable[1]>[0]>; + /** * AMQP (RabbitMQ) queue adapter implementation. * @@ -38,6 +45,10 @@ export class AmqpAdapter implements MessageQueue { private queue!: AmqpMessageQueue; private readonly connection: ReturnType; private listening = false; + private listenerDone?: Promise; + private stopping?: Promise; + private publisher?: Promise; + private closed = false; private abortController?: AbortController; private deadLetterStore: DeadLetterStorePort | null = null; private readonly explicitDeadLetterStore?: DeadLetterStorePort; @@ -59,26 +70,17 @@ export class AmqpAdapter implements MessageQueue { explicitDeadLetterStore?: DeadLetterStorePort, ) { this.explicitDeadLetterStore = explicitDeadLetterStore; - try { - this.connection = connect(url); - this.connection.then((connection) => { - this.queue = new AmqpMessageQueue(connection); - }).catch((error) => { - throw new QueueConnectionError( - `Failed to connect to RabbitMQ: ${ - error instanceof Error ? error.message : String(error) - }`, - error instanceof Error ? error : undefined, - ); - }); - } catch (error) { + this.connection = connect(url).then((connection) => { + connection.on('error', () => {}); // Active operations observe their own failure boundary. + this.queue = new AmqpMessageQueue(connection, { queue: queueName, nativeRetrial: true }); + return connection; + }, (error: unknown) => { throw new QueueConnectionError( - `Failed to initialize AMQP queue: ${ - error instanceof Error ? error.message : String(error) - }`, + 'Failed to connect to RabbitMQ.', error instanceof Error ? error : undefined, ); - } + }); + void this.connection.catch(() => {}); } /** @@ -143,95 +145,187 @@ export class AmqpAdapter implements MessageQueue { } } - /** - * Listen for AMQP messages until stopped or aborted. - * - * @param handler - Async callback invoked for each decoded message. - * @param options - Listener cancellation and concurrency settings. - */ - async listen( + /** Consume with native cancellation, keeping admitted callbacks alive through intake stop. */ + listen( handler: (message: T, context: MessageContext) => Promise, options?: ListenOptions, ): Promise { - if (this.listening) { - throw new QueueError('Queue is already listening', QueueErrorCode.CONFIGURATION_ERROR); + if (this.listening || this.closed) { + return Promise.reject( + new QueueError('Queue is not available for listening.', QueueErrorCode.CONFIGURATION_ERROR), + ); + } + const concurrency = options?.concurrency ?? 1; + if (!Number.isSafeInteger(concurrency) || concurrency < 1) { + return Promise.reject( + new QueueError( + 'concurrency must be a positive integer.', + QueueErrorCode.CONFIGURATION_ERROR, + ), + ); } - this.listening = true; - this.abortController = new AbortController(); + const controller = new AbortController(); + this.abortController = controller; + const signal = options?.signal + ? AbortSignal.any([options.signal, controller.signal]) + : controller.signal; + const done = this.consume(handler, signal, concurrency).finally(() => { + this.listening = false; + }); + this.listenerDone = done; + return done; + } - const signal = options?.signal; - if (signal) { - signal.addEventListener('abort', () => { - this.abortController?.abort(); - }); - } + /** Stop the consumer, then close only the publisher and connection owned by this adapter. */ + stop(): Promise { + return this.stopping ??= (async () => { + this.closed = true; + this.abortController?.abort(); + try { + await this.listenerDone; + } finally { + await (await this.connection).close(); + } + })(); + } + /** Own native consumer cancellation, callback joining, and channel failure. */ + private async consume( + handler: (message: T, context: MessageContext) => Promise, + signal: AbortSignal, + concurrency: number, + ): Promise { + if (signal.aborted) return; + const connection = await this.connection; + const channel = await connection.createChannel(); + const stopped = Promise.withResolvers(); + const failed = Promise.withResolvers(); + void failed.promise.catch(() => {}); + const active = new Set>(); + let failure: { error: unknown } | undefined; + let closing = false; + let usable = true; + let tag: string | undefined; + const onAbort = () => stopped.resolve(); + const onError = (error: unknown) => { + failure ??= { error }; + usable = false; + failed.reject(error); + }; + const onClose = () => { + usable = false; + if (!closing) onError(new QueueConnectionError('AMQP consumer closed.')); + }; + channel.on('error', onError); + channel.on('close', onClose); + connection.on('close', onClose); + signal.addEventListener('abort', onAbort, { once: true }); + if (signal.aborted) onAbort(); try { - await this.connection; - await this.queue.listen(async (rawMessage) => { - let payload: T; - let headers: Record = {}; - let messageId: string; - let enqueuedAt: Date; - let deliveryCount: number; - - if (isMessageEnvelope(rawMessage)) { - payload = rawMessage.payload; - headers = rawMessage.headers; - messageId = rawMessage.messageId; - enqueuedAt = new Date(rawMessage.enqueuedAt); - deliveryCount = rawMessage.deliveryCount + 1; - } else { - payload = rawMessage as T; - messageId = crypto.randomUUID(); - enqueuedAt = new Date(); - deliveryCount = 1; + await channel.assertQueue(this.queueName, { durable: true }); + await channel.prefetch(concurrency); + tag = (await channel.consume(this.queueName, (message) => { + if (!message) { + if (!closing) onError(new QueueConnectionError('AMQP consumer cancelled.')); + return; } - - await handler( - payload, - this.createContext(messageId, payload, enqueuedAt, headers, deliveryCount), - ); - }, { signal: this.abortController.signal }); - } catch (error) { - if (error instanceof Error && error.name === 'AbortError') { - return; - } - throw new QueueError( - `Queue listener failed: ${error instanceof Error ? error.message : String(error)}`, - QueueErrorCode.DEQUEUE_FAILED, - { - cause: error instanceof Error ? error : undefined, - context: { queueName: this.queueName }, - }, - ); + if (signal.aborted) { + if (usable) { + try { + channel.nack(message, false, true); + } catch (error) { + onError(error); + } + } + return; + } + const task = this.deliver(message, channel, () => usable, handler) + .catch(onError).finally(() => active.delete(task)); + active.add(task); + }, { noAck: false })).consumerTag; + await Promise.race([stopped.promise, failed.promise]); } finally { - this.listening = false; + closing = true; + try { + if (tag) await channel.cancel(tag).catch(() => {}); + if (!failure) await Promise.race([Promise.allSettled([...active]), failed.promise]); + } finally { + usable = false; + await channel.close().catch(() => {}); + signal.removeEventListener('abort', onAbort); + channel.removeListener('close', onClose); + connection.removeListener('close', onClose); + } } } - /** - * Stop listening and close the AMQP connection when possible. - */ - async stop(): Promise { - if (!this.listening) { - return; + /** Dispatch one native receipt with exactly one terminal settlement. */ + private async deliver( + message: NativeDelivery, + channel: AmqpChannel, + usable: () => boolean, + handler: (message: T, context: MessageContext) => Promise, + ): Promise { + const raw: unknown = JSON.parse(message.content.toString('utf8')); + const envelope = isMessageEnvelope(raw) ? raw : createEnvelope(raw as T); + const context = this.createContext( + envelope.messageId, + envelope.payload, + new Date(envelope.enqueuedAt), + envelope.headers, + envelope.deliveryCount + 1, + { + ack: () => { + if (!usable()) throw new QueueConnectionError('AMQP receipt is no longer owned.'); + channel.ack(message); + return Promise.resolve(); + }, + requeue: async (preserve) => { + if (!usable()) throw new QueueConnectionError('AMQP receipt is no longer owned.'); + await this.publishRetry({ + ...envelope, + deliveryCount: preserve ? envelope.deliveryCount : envelope.deliveryCount + 1, + }); + if (!usable()) throw new QueueConnectionError('AMQP receipt is no longer owned.'); + channel.ack(message); + }, + }, + ); + try { + await handler(envelope.payload, context); + await context.ack(); + } catch { + try { + await context.nack({ requeue: true }); + } catch (error) { + if (usable()) channel.nack(message, false, true); + else throw error; + } } + } - this.abortController?.abort(); - this.listening = false; - await new Promise((resolve) => setTimeout(resolve, 100)); - + /** Confirm the retry before acknowledging its predecessor; uncertain outcomes are at-least-once. */ + private async publishRetry(envelope: MessageEnvelope): Promise { + if (this.closed) throw new QueueConnectionError('AMQP adapter is stopped.'); + const pending = this.publisher ??= this.connection.then(async (connection) => { + const channel = await connection.createConfirmChannel(); + channel.on('error', () => {}); + await channel.assertQueue(this.queueName, { durable: true }); + return channel; + }); try { - const connection = await this.connection; - if ( - connection && typeof (connection as { close?: () => Promise }).close === 'function' - ) { - await (connection as { close: () => Promise }).close(); - } - } catch { - // Ignore shutdown errors. + const channel = await pending; + await new Promise((resolve, reject) => { + channel.sendToQueue(this.queueName, Buffer.from(JSON.stringify(envelope)), { + persistent: true, + contentType: 'application/json', + }, (error: unknown) => error ? reject(error) : resolve()); + }); + } catch (error) { + if (this.publisher === pending) this.publisher = undefined; + await pending.then((channel) => channel.close()).catch(() => {}); + throw error; } } @@ -244,31 +338,41 @@ export class AmqpAdapter implements MessageQueue { enqueuedAt: Date, headers: Record, deliveryCount: number, + io?: Readonly<{ ack(): Promise; requeue(preserve: boolean): Promise }>, ): MessageContext { + let settlement: Promise | undefined; + const settle = (operation: () => Promise): Promise => + settlement ??= operation().catch((error: unknown) => { + settlement = undefined; + throw error; + }); return createMessageContext( messageId, enqueuedAt, headers, deliveryCount, - async () => {}, - async (options: NackOptions = {}) => { - if (options.requeue ?? true) { - return; - } - const store = this.ensureDeadLetterStore(); - await store.append(toDeadLetterRecord( - { - messageId, - queueName: this.queueName, - payload, - headers, - deliveryCount, - enqueuedAt, - }, - options.reason ?? 'nack_without_requeue', - options, - )); - }, + () => settle(() => io?.ack() ?? Promise.resolve()), + (options: NackOptions = {}) => + settle(async () => { + if (options.requeue ?? true) { + await io?.requeue(options.preserveDeliveryCount === true); + return; + } + const store = this.ensureDeadLetterStore(); + await store.append(toDeadLetterRecord( + { + messageId, + queueName: this.queueName, + payload, + headers, + deliveryCount, + enqueuedAt, + }, + options.reason ?? 'nack_without_requeue', + options, + )); + await io?.ack(); + }), ); } } From 1802d739a620112a80e01bc6bd0b1b93f0494be2 Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Tue, 17 Nov 2026 00:36:00 +0100 Subject: [PATCH 10/20] feat(workers): pause intake and drain admitted work before abort --- .../context-pack.md | 4 +- .../drift.md | 3 + .../worklog.md | 13 + .../src/runtime/job-retry.ts | 4 +- .../tests/runtime/retry-first-cause_test.ts | 23 ++ plugins/workers/bin/runtime.ts | 1 + .../workers/worker/job-dead-letters_test.ts | 4 +- plugins/workers/worker/job-dispatcher.ts | 23 +- plugins/workers/worker/job-dispatcher_test.ts | 243 +-------------- plugins/workers/worker/job-execution.ts | 40 ++- plugins/workers/worker/job-retry_test.ts | 2 + plugins/workers/worker/listener-supervisor.ts | 145 +++++---- plugins/workers/worker/mod.ts | 6 + plugins/workers/worker/queue-consumer.ts | 15 +- .../worker/shutdown-first-cause_test.ts | 90 ++++++ .../worker/worker-abort-lifecycle_test.ts | 272 ++++++++++++++++ .../worker/worker-intake-lifecycle_test.ts | 217 +++++++++++++ plugins/workers/worker/worker-options.ts | 39 ++- .../worker/worker-shutdown-deliveries_test.ts | 6 +- plugins/workers/worker/worker.ts | 293 +++++++++--------- 20 files changed, 975 insertions(+), 468 deletions(-) create mode 100644 packages/plugin-workers-core/tests/runtime/retry-first-cause_test.ts create mode 100644 plugins/workers/worker/shutdown-first-cause_test.ts create mode 100644 plugins/workers/worker/worker-abort-lifecycle_test.ts create mode 100644 plugins/workers/worker/worker-intake-lifecycle_test.ts diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md index 0288c6bf18..33ccb76247 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md @@ -1,6 +1,6 @@ # Context pack -Issues2633/2635, PR2645, branch feat/worker-pause-drain, base b0eb079575968e843b781579673c9fe2ae250e07. Read plan.md/research.md/worklog.md/drift.md/supervisor.md and the two live issues in full. Plan-Gate required before product source changes. Four new tests are executable against unchanged main and include genuine red behavior plus failure controls. See research for exact results and excluded harness errors. Research resources cleaned: all five container identities including an earlier startup replacement, and five associated anonymous volumes proven by mount events. Current implementation retains three positively owned Redis/Garnet/Postgres backends for S2/S3, with immutable IDs/mounts privately recorded; foreign resources untouched. +Issues2633/2635, PR2645, branch feat/worker-pause-drain, base b0eb079575968e843b781579673c9fe2ae250e07. Read plan.md/research.md/worklog.md/drift.md/supervisor.md and the two live issues in full. Plan-Gate required before product source changes. Four new tests are executable against unchanged main and include genuine red behavior plus failure controls. See research for exact results and excluded harness errors. Research resources cleaned: all five container identities including an earlier startup replacement, and five associated anonymous volumes proven by mount events. Current implementation retains four positively owned Redis/Garnet/Postgres/AMQP backends for implementation, with immutable IDs/mounts privately recorded; foreign resources untouched. Primary paths: plugins/workers/worker/{worker,worker-options,listener-supervisor,queue-consumer,job-dispatcher,delivery-lease,job-runner-pool}.ts; plugins/workers/bin/runtime.ts; plugins/workers/src/cli/load-workers-config.ts; core config/workers-config.ts and runtime/in-process-job-runner.ts; existing core src/shutdown/ subpath. Queue ports/message-queue.ts, Redis lease/script, Postgres, KV polling, native Deno KV, AMQP, memory. Actual public types inspected with deno doc first. @@ -8,4 +8,4 @@ Use Deno2.9.7 through mise. rtk/rg absent; plain commands. Scoped structured wra Public artifacts/comments/bodies must contain no operator absolute paths, hostnames/ports, session/thread IDs or downstream names. Keep raw logs private, public evidence repo-relative. Evaluator writes plan-eval.md and phase comment, leaves artifact uncommitted. No source changes, commits, pushes, labels/readiness changes or PR watching by evaluator. -Current checkpoint: initial Plan-Gate approved; S2native receipts28/28 and core/Postgres8/8green, S3a portable handback/local guards and adjacent gates green. Added listener ownership contract is unimplemented and requires the independent PLAN-EVAL follow-up in plan-intake-drain-addendum.md before use. Raw ownership/drain red reports remain private; public results are in worklog. No Worker lifecycle/config or AMQP implementation yet. +Current checkpoint: original and listener-owner addendum Plan-Gates approved; S2/S3 portable and native handback committed and live-tested on all seven backend variants. S4 lifecycle, pause generations, bounded stop, portable shutdown-only refund and first-cause controls implemented; full Worker113/0/0 and static gates green. S5 standard runtime configuration and standalone default, S6 docs, S7 final validation/supported full smoke and independent IMPL-EVAL remain. No acceptance completion claim yet. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md index 07c12e4837..50f206391b 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md @@ -13,3 +13,6 @@ S3a stays within approved adapter scope. Its handback storage-failure test expos Follow-up PLAN-EVAL cycle2 settlement: addendum risk text now pins the shared name with the already approved Worker-total stop option (Listener JSDoc/S6 docs must cross-reference the per-listener post-intake cutoff against the unchanged25s Worker total). Wording only, settled in place under the cycle2 authorization; no product/tests or design change. Original Plan-Gate PASS history preserved; follow-up verdict PASS appended to plan-eval.md at014730e1. + + +S4 test contract alignment: older immediate-abort fixtures now assert the work window before ShutdownError. The held progress sink permits only the documented bounded TimeoutError result in the shutdown case; cause, once-only state/cleanup and claim-release assertions are retained. Relevant lifecycle fixtures moved into their own small file, reducing the pre-existing oversized dispatch test. Redis delivery fixture uses500ms total to include the native adapter's unchanged100ms disconnect grace, retaining all accounting assertions. Core retry catch formerly replaced an observed failure with a later abort; red first-cause cases precede its narrow preservation fix. No budget or acceptance contract changed. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index d6acb1c3bc..78faba3196 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -58,3 +58,16 @@ Previously red owner/default controls now6/6green on real Redis/Garnet and KV po The pinned provider's non-native async path double-acknowledged; replace consumption with existing upstream amqplib basic.cancel/prefetch and one per-receipt settlement boundary. Configure the existing Fedify enqueue path with the actual queue name and native retry. Preserve/count the same envelope on an owned confirm publisher and await broker confirmation before old ack. Native error/cancel/close callbacks are owned, retained contexts cannot publish through a closed receipt, and stop closes the owned connection even after listen already completed. No new dependency or fake provider seam. Real backend handback suite23/23 executes all seven backend variants (Redis/Garnet/Postgres/AMQP/DenoKV/KVpoll/memory),8shutdown restarts, explicit and actual thrown failures plus native DLQ tests,0ignored. After channel-edge cleanup strengthening, focused AMQP/policy5/5green. Scoped check/lint/fmt0, new private method JSDoc supplied; queue combined doclint1/exact per-entrypoint baseline unchanged, no newdiagnostics. Preflight/accuracyPASS. Four owned implementation backends now retained, each immutable ID and mount recorded privately; no foreign cleanup. Worker integration follows S4. + + +## S4: pause generations, bounded stop and shutdown-only handback + +Worker now owns explicit lifecycle state, independent intake generation cancellation and one total monotonic drain. Jobs/tasks/triggers pass the approved infinite backend-listener wait while the Worker owns its finite total. Pause keeps admitted signals alive and start pending; resume joins the old generation before admitting replacements. Stop is terminal/idempotent, guards late startup allocation, waits work before abort, then joins runner/listeners and closes identity-deduplicated queues. Health records healthy intentional pause. Top-level grace takes precedence over the supported legacy pool setting. + +Portable nack preservation is used only when the observed first failure is the worker shutdown reason. A business failure observed before retry backoff cannot be reclassified by later shutdown; the core retry helper now retains that earlier error. Earlier deadline/operator cancellation stay charged. Two core red tests reproduced first-cause rewriting before that change; four real dispatch controls prove charged attempts. + +Structured full Worker suite plus core cause tests:113pass/0fail/0ignored, including actual Redis and Garnet eight-restart tests. Intake lifecycle8/8 covers all three families, pause/resume overlap, claim races, terminal stop and bounded held startup. Focused abort lifecycle7/7 retains terminal status, exact reentrant completion, released claims and once-only cleanup. Scope moved those relevant cases out of the legacy1195-line dispatch test into a272-line focused file. Existing KV outage fixture observes portable explicit nack instead of thrown-handler retry; attempt2/write2/finalzero assertions retained. Hung progress may exhaust cleanup and return bounded TimeoutError, which its fixture validates. Redis fixture total raised100->500ms because the unchanged native disconnect grace is100ms; eight restarts/count/DLQ controls unchanged. + +Scoped161-file formatter, check and lint wrappers:0findings. Plugin doc-lint combined2 and every entrypoint match measured baseline, including runtime2/worker0/services0. New lifecycle type is an identity export for runtime public closure. release:preflight fourPASS; docs:accuracyPASS. Four exact owned backend containers remain retained for subsequent slices. Configuration/default standalone runner changes and S6 documentation remain pending; no acceptance completion claim yet. + +S4 quality:gate (quality scan + architecture fitness) exit0; no new doctrine exemption. Worker476lines, focused abort fixture272lines; legacy dispatch test reduced by241lines. diff --git a/packages/plugin-workers-core/src/runtime/job-retry.ts b/packages/plugin-workers-core/src/runtime/job-retry.ts index 6e9b32a4c1..43bc350d1b 100644 --- a/packages/plugin-workers-core/src/runtime/job-retry.ts +++ b/packages/plugin-workers-core/src/runtime/job-retry.ts @@ -24,8 +24,8 @@ export async function executeJobWithRetry { + const controller = new AbortController(); + const result = await executeJobWithRetry( + () => { + controller.abort(new DOMException('Later shutdown.', 'ShutdownError')); + return Promise.reject(cause); + }, + resolveJobRetryPolicy({ id: 'first-cause', maxRetries: 0 }), + { signal: controller.signal }, + ) + .catch((error: unknown) => error); + assertStrictEquals(result, cause); + }); +} diff --git a/plugins/workers/bin/runtime.ts b/plugins/workers/bin/runtime.ts index d456f3d892..22a304c627 100644 --- a/plugins/workers/bin/runtime.ts +++ b/plugins/workers/bin/runtime.ts @@ -10,6 +10,7 @@ import { createQueue } from '@netscript/queue'; import { deadline } from '@std/async'; import type { JobMessage } from '@netscript/plugin-workers-core/runtime'; +export type { WorkerLifecycleState } from '@netscript/plugin-workers-core/shutdown'; import { readSeedSampleRequest, watchSeedSample } from '../src/runtime/seed-sample.ts'; import { getKv } from '@netscript/kv'; import type { ChildHealthMonitor, ChildHealthSnapshot } from '@netscript/plugin/health'; diff --git a/plugins/workers/worker/job-dead-letters_test.ts b/plugins/workers/worker/job-dead-letters_test.ts index d108c2ca46..195f5cb070 100644 --- a/plugins/workers/worker/job-dead-letters_test.ts +++ b/plugins/workers/worker/job-dead-letters_test.ts @@ -127,9 +127,9 @@ Deno.test('KV Worker listener requeues after a job DLQ write outage instead of a void started.catch(() => undefined); try { await harness.queue.enqueue({ jobId: 'exhausted', topic: 'jobs', triggeredBy: 'queue' }); - await until(async () => await durable.depth() === 1 && harness.returned() === 1); + await until(async () => await durable.depth() === 1 && harness.returned() === 2); await until(async () => (await harness.queue.getStats()).processing === 0); - assertEquals(harness.rejected(), 1, 'non-lease queues retain rejection-based retry semantics'); + assertEquals(harness.rejected(), 0, 'explicit retry settlement returns without a second nack'); assertEquals(attempts, 2, 'released idempotency claim permits transport redelivery'); assertEquals(writes, 2); assertEquals(await harness.queue.getStats(), { pending: 0, processing: 0, dlq: 0 }); diff --git a/plugins/workers/worker/job-dispatcher.ts b/plugins/workers/worker/job-dispatcher.ts index 1fc1b09b49..fe6982c78e 100644 --- a/plugins/workers/worker/job-dispatcher.ts +++ b/plugins/workers/worker/job-dispatcher.ts @@ -56,6 +56,7 @@ export async function processWorkerJob( let claim: WorkerIdempotencyClaim | undefined; const lease = new WorkerDeliveryLease(queueContext, context.shutdownSignal); let attempts = 0; + let attemptFailed = false; let writingDeadLetter = false; let terminalRecorded = false; let executionSucceeded = false; @@ -183,6 +184,9 @@ export async function processWorkerJob( (attempt) => { attempts = attempt; }, + () => { + attemptFailed = true; + }, ); addJobStepEvent('state_update', { status: result.success ? 'completed' : 'failed' }); @@ -224,8 +228,14 @@ export async function processWorkerJob( writingDeadLetter || terminalRecorded || executionSucceeded; try { if (requeue) { - if (!queueContext?.extendVisibility) throw error; - await queueContext.nack({ requeue: true }); + if (!queueContext) throw error; + const shutdown = !attemptFailed && error === lease.signal.reason && + error instanceof Error && + error.name === 'ShutdownError'; + await queueContext.nack({ + requeue: true, + ...(shutdown ? { preserveDeliveryCount: true } : {}), + }); return; } if (!(error instanceof Error && error.name === 'AbortError')) { @@ -372,8 +382,13 @@ export async function processWorkerTask( executionSucceeded || lease.signal.aborted || (claimAttempted && !claim) || (claim && !claim.claimed && !claim.alreadyApplied) ) { - if (!queueContext?.extendVisibility) throw error; - await queueContext.nack({ requeue: true }); + if (!queueContext) throw error; + const shutdown = error === lease.signal.reason && error instanceof Error && + error.name === 'ShutdownError'; + await queueContext.nack({ + requeue: true, + ...(shutdown ? { preserveDeliveryCount: true } : {}), + }); } } finally { await lease.stop(); diff --git a/plugins/workers/worker/job-dispatcher_test.ts b/plugins/workers/worker/job-dispatcher_test.ts index 5c833d5e9e..a8db701504 100644 --- a/plugins/workers/worker/job-dispatcher_test.ts +++ b/plugins/workers/worker/job-dispatcher_test.ts @@ -1,14 +1,7 @@ import { configure, configureLogging, type LogRecord, resetLogging } from '@netscript/logger'; import { stub } from '@std/testing/mock'; import { KvDeadLetterStore } from '@netscript/queue/adapters/kv-dead-letter-store'; -import { - assert, - assertEquals, - assertInstanceOf, - assertNotEquals, - assertRejects, - assertStrictEquals, -} from 'jsr:@std/assert@^1'; +import { assert, assertEquals, assertNotEquals, assertRejects } from 'jsr:@std/assert@^1'; import { MemoryKvAdapter } from '@netscript/kv'; import type { JobContext, @@ -530,240 +523,6 @@ async function bounded(promise: Promise, timeoutMs = 2_000): Promise { } } -for (const cause of ['timeout', 'cancel', 'shutdown'] as const) { - Deno.test(`Worker queue-to-Deno handler aborts on ${cause} and records the terminal cause`, async () => { - await using kv = new MemoryKvAdapter(); - const queue = new MemoryQueueAdapter({ pollInterval: 1 }); - const taskQueue = new MemoryQueueAdapter({ pollInterval: 1 }); - const idempotency = new KvWorkerIdempotencyStore({ kv }); - const ready = Promise.withResolvers(); - const finish = Promise.withResolvers(); - const terminal = Promise.withResolvers(); - const completed: WorkerCompleteExecutionOptions[] = []; - const executionState: WorkerExecutionState = { - create: () => Promise.resolve({ id: 'execution-1' }), - start: () => Promise.resolve({ id: 'execution-1' }), - progress: () => Promise.resolve({ id: 'execution-1' }), - complete: (_id, options) => { - completed.push(options); - terminal.resolve(options); - return Promise.resolve({ id: 'execution-1' }); - }, - }; - const job: JobDefinition = { - id: `lifecycle-${cause}`, - enabled: true, - topic: 'jobs', - executionType: 'deno', - entrypoint: import.meta.url, - timeout: cause === 'timeout' ? 100 : 10_000, - handler: async (context) => { - ready.resolve(context); - await Promise.race([ - new Promise((resolve) => { - if (context.signal.aborted) resolve(); - else context.signal.addEventListener('abort', () => resolve(), { once: true }); - }), - finish.promise, - ]); - return { success: true }; - }, - }; - const worker = new Worker({ - workerId: 'cancellation-lifecycle', - queue, - taskQueue, - registry: { get: () => Promise.resolve(job) }, - executionState, - taskRegistry: { get: () => Promise.resolve(undefined) }, - taskExecutor: { execute: () => Promise.reject(new Error('Task path must not run.')) }, - idempotency, - workerPoolOptions: { abortGracePeriodMs: 5 }, - }); - const started = worker.start(); - void started.catch(() => undefined); - const claimInput = { concept: 'job' as const, targetId: job.id, idempotencyKey: 'delivery-1' }; - try { - await queue.enqueue({ - jobId: job.id, - topic: 'jobs', - triggeredBy: 'manual', - idempotencyKey: 'delivery-1', - }); - const context = await bounded(ready.promise); - assertInstanceOf(context.signal, AbortSignal); - assert(typeof context.deadlineAt === 'number'); - assertEquals(worker.activeJobCount, 1); - assertEquals(worker.cancel('absent'), false); - let stopped: Promise | undefined; - if (cause === 'cancel') { - assertEquals(worker.cancel('execution-1'), true); - assertEquals(worker.cancel('execution-1'), false); - assert(context.signal.aborted); - } else if (cause === 'shutdown') { - stopped = worker.stop(); - assertEquals(worker.stop(), stopped); - assert(context.signal.aborted); - } - const result = await bounded(terminal.promise); - assert(context.signal.aborted); - assertInstanceOf(context.signal.reason, DOMException); - assertEquals( - context.signal.reason.name, - cause === 'timeout' - ? 'TimeoutError' - : cause === 'shutdown' - ? 'ShutdownError' - : 'AbortError', - ); - assertEquals(result.status, cause === 'timeout' ? 'timeout' : 'cancelled'); - await bounded(stopped ?? worker.stop()); - await bounded(started); - assertEquals(worker.activeJobCount, 0); - assertEquals(worker.isRunning, false); - assertEquals(completed.length, 1); - const retry = await idempotency.claim(claimInput); - assertEquals(retry.claimed, true); - assertEquals(retry.alreadyApplied, false); - await idempotency.release(retry.key); - } finally { - finish.resolve(); - await bounded(worker.stop()); - await bounded(started); - } - }); -} - -async function verifyBlockedProgressWorker( - cause: 'timeout' | 'cancel' | 'shutdown', - reentrant = false, -): Promise { - await using kv = new MemoryKvAdapter(); - const queue = new MemoryQueueAdapter({ pollInterval: 1 }); - const taskQueue = new MemoryQueueAdapter({ pollInterval: 1 }); - let queueStops = 0; - let taskStops = 0; - const stopQueue = queue.stop.bind(queue); - const stopTasks = taskQueue.stop.bind(taskQueue); - queue.stop = async () => { - queueStops++; - await stopQueue(); - }; - taskQueue.stop = async () => { - taskStops++; - await stopTasks(); - }; - const idempotency = new KvWorkerIdempotencyStore({ kv }); - const ready = Promise.withResolvers(); - const sinkEntered = Promise.withResolvers(); - const finishProgress = Promise.withResolvers(); - const terminal = Promise.withResolvers(); - const completed: WorkerCompleteExecutionOptions[] = []; - const executionState: WorkerExecutionState = { - create: () => Promise.resolve({ id: 'execution-1' }), - start: () => Promise.resolve({ id: 'execution-1' }), - progress: async () => { - sinkEntered.resolve(); - await finishProgress.promise; - return { id: 'execution-1' }; - }, - complete: (_id, options) => { - completed.push(options); - terminal.resolve(options); - return Promise.resolve({ id: 'execution-1' }); - }, - }; - const job: JobDefinition = { - id: `progress-drain-${cause}`, - enabled: true, - topic: 'jobs', - executionType: 'deno', - entrypoint: import.meta.url, - timeout: cause === 'timeout' ? 500 : 10000, - handler: (context) => { - ready.resolve(context); - void context.reportProgress?.(10); - return { success: true }; - }, - }; - const worker = new Worker({ - workerId: 'progress-drain-lifecycle', - queue, - taskQueue, - registry: { get: () => Promise.resolve(job) }, - executionState, - taskRegistry: { get: () => Promise.resolve(undefined) }, - taskExecutor: { execute: () => Promise.reject(new Error('Task path must not run.')) }, - idempotency, - workerPoolOptions: { abortGracePeriodMs: 5 }, - }); - const started = worker.start(); - void started.catch(() => undefined); - const claimInput = { - concept: 'job' as const, - targetId: job.id, - idempotencyKey: 'progress-delivery', - }; - let nestedStop: Promise | undefined; - try { - await queue.enqueue({ - jobId: job.id, - topic: 'jobs', - triggeredBy: 'manual', - idempotencyKey: 'progress-delivery', - }); - const context = await bounded(ready.promise); - await bounded(sinkEntered.promise); - await new Promise((resolve) => setTimeout(resolve, 0)); - if (reentrant) { - context.signal.addEventListener('abort', () => { - nestedStop = worker.stop(); - }, { once: true }); - } - let stopped: Promise | undefined; - if (cause === 'cancel') { - assertEquals(worker.cancel('execution-1'), true); - assertEquals(context.signal.aborted, true); - } else if (cause === 'shutdown') { - stopped = worker.stop(); - assertEquals(worker.stop(), stopped); - if (reentrant) assertStrictEquals(nestedStop, stopped); - } - const result = await bounded(terminal.promise); - assertEquals(context.signal.aborted, true); - assertInstanceOf(context.signal.reason, DOMException); - assertEquals( - context.signal.reason.name, - cause === 'cancel' ? 'AbortError' : cause === 'timeout' ? 'TimeoutError' : 'ShutdownError', - ); - assertEquals(result.status, cause === 'timeout' ? 'timeout' : 'cancelled'); - // Neither stop nor claim release waits indefinitely on the still-blocked sink. - await bounded(stopped ?? worker.stop()); - await bounded(started); - assertEquals(worker.activeJobCount, 0); - assertEquals(worker.isRunning, false); - assertEquals(completed.length, 1); - assertEquals(queueStops, 1); - assertEquals(taskStops, 1); - const retry = await idempotency.claim(claimInput); - assertEquals(retry.claimed, true); - assertEquals(retry.alreadyApplied, false); - await idempotency.release(retry.key); - } finally { - finishProgress.resolve(); - await bounded(worker.stop()); - if (nestedStop) await bounded(nestedStop); - await bounded(started); - } -} - -for (const cause of ['timeout', 'cancel', 'shutdown'] as const) { - Deno.test(`Worker cancellation during unawaited progress drain retains ${cause} ownership`, () => - verifyBlockedProgressWorker(cause)); -} -Deno.test('Worker reentrant stop from abort listener shares completion and cleanup', () => - verifyBlockedProgressWorker('shutdown', true)); - Deno.test('processWorkerTask forwards correlation and message trace context into executor options', async () => { await using kv = new MemoryKvAdapter(); const executionState = new MemoryExecutionState(); diff --git a/plugins/workers/worker/job-execution.ts b/plugins/workers/worker/job-execution.ts index 7a91ebe7bc..9a89137309 100644 --- a/plugins/workers/worker/job-execution.ts +++ b/plugins/workers/worker/job-execution.ts @@ -2,6 +2,7 @@ import { workerJobLogger } from './job-logger.ts'; import { DEFAULT_TOPIC, executeJobWithRetry, + isJobAbort, type JobDefinition, type JobMessage, resolveJobRetryPolicy, @@ -22,21 +23,36 @@ export async function executeWorkerJob( traceHeaders: Record, correlationId?: string, onAttempt?: (attempt: number) => void, + onFailedAttempt?: () => void, ): Promise { return await executeJobWithRetry( async () => { - const executionType = jobDef.executionType ?? 'deno'; - return executionType === 'deno' - ? await executeDenoJob( - context, - jobDef, - executionId, - payload, - traceHeaders, - correlationId, - signal, - ) - : await executePolyglotTask(context, jobDef, payload, signal, traceHeaders, correlationId); + try { + const executionType = jobDef.executionType ?? 'deno'; + const result = executionType === 'deno' + ? await executeDenoJob( + context, + jobDef, + executionId, + payload, + traceHeaders, + correlationId, + signal, + ) + : await executePolyglotTask( + context, + jobDef, + payload, + signal, + traceHeaders, + correlationId, + ); + if (!result.success && !signal.aborted) onFailedAttempt?.(); + return result; + } catch (error) { + if (!isJobAbort(error)) onFailedAttempt?.(); + throw error; + } }, resolveJobRetryPolicy(jobDef, jobDef.handler), { signal, clock: context.retryClock, onAttempt }, diff --git a/plugins/workers/worker/job-retry_test.ts b/plugins/workers/worker/job-retry_test.ts index 771e85d938..d6003e0be7 100644 --- a/plugins/workers/worker/job-retry_test.ts +++ b/plugins/workers/worker/job-retry_test.ts @@ -169,6 +169,8 @@ Deno.test('Worker KV listener replays a job after shutdown during backoff and re const queue = makeQueue('kv-polling', kv) as KvPollingAdapter; const options = { workerId: 'shutdown-replay', + drainTimeoutMs: 500, + abortGracePeriodMs: 250, deadLetters: harness.deadLetters, queue, registry: harness.context.registry, diff --git a/plugins/workers/worker/listener-supervisor.ts b/plugins/workers/worker/listener-supervisor.ts index fa96b7662f..faa076b1e5 100644 --- a/plugins/workers/worker/listener-supervisor.ts +++ b/plugins/workers/worker/listener-supervisor.ts @@ -1,6 +1,6 @@ import { ChildHealthMonitor, type ChildHealthSnapshot } from '@netscript/plugin/health'; import { QueueError, QueueErrorCode } from '@netscript/queue'; -import { delay } from '@std/async'; +import { abortable, delay } from '@std/async'; /** Credential-safe category for a listener's most recent failure. */ export type WorkerListenerFailureCause = @@ -24,7 +24,13 @@ export function resolveListenerHealthyResetMs(value?: number): number { return resolved; } -export type WorkerListenerStatus = 'idle' | 'running' | 'restarting' | 'failed' | 'stopped'; +export type WorkerListenerStatus = + | 'idle' + | 'running' + | 'paused' + | 'restarting' + | 'failed' + | 'stopped'; export interface WorkerListenerSnapshot { readonly name: string; @@ -67,6 +73,12 @@ export class WorkerListenerSupervisor { readonly #health: ChildHealthMonitor; #completionResolve!: () => void; #started = false; + #paused = false; + #resumed = Promise.withResolvers(); + #generation?: { + controller: AbortController; + done: ReturnType>; + }; #localAbort = new AbortController(); readonly #run: (signal: AbortSignal) => Promise; @@ -105,15 +117,38 @@ export class WorkerListenerSupervisor { void this.#runLoop(); } + /** Stop intake without ending the supervisor lifetime or cancelling admitted work. */ + pause(): Promise { + if (!this.#started || this.#localAbort.signal.aborted || this.#status === 'failed') { + return Promise.resolve(); + } + this.#resetHealthyBudget(); + if (!this.#paused) this.#resumed = Promise.withResolvers(); + this.#paused = true; + this.#status = 'paused'; + this.#runningSince = undefined; + this.#generation?.controller.abort(); + return this.#generation?.done.promise ?? Promise.resolve(); + } + + /** Resume after the previous listener's admitted callbacks have settled. */ + resume(): void { + if (this.#localAbort.signal.aborted || this.#status === 'failed') return; + this.#paused = false; + this.#resumed.resolve(); + } + async stop(): Promise { - if (this.#status === 'stopped') { - return; + if (!this.#started) { + this.#started = true; + this.#completionResolve(); } this.#status = 'stopped'; if (!['failed', 'crash-looping'].includes(this.#health.snapshot().state)) { this.#health.stopped(); } this.#localAbort.abort(); + this.#resumed.resolve(); await this.completion; } @@ -142,50 +177,69 @@ export class WorkerListenerSupervisor { } async #runLoop(): Promise { - const signal = combineSignals(this.#localAbort.signal, this.#abortSignal); + const signal = this.#abortSignal + ? AbortSignal.any([this.#localAbort.signal, this.#abortSignal]) + : this.#localAbort.signal; try { while (!signal.aborted) { + if (this.#paused) { + try { + await abortable(this.#resumed.promise, signal); + } catch { + if (signal.aborted) break; + } + if (signal.aborted) break; + if (this.#paused) continue; + } + const generation = { + controller: new AbortController(), + done: Promise.withResolvers(), + }; + void generation.done.promise.catch(() => {}); + this.#generation = generation; + const attempt = AbortSignal.any([signal, generation.controller.signal]); this.#status = 'running'; this.#runningSince = this.#now(); this.#health.dependenciesReady(); this.#health.running(); + let failure: { error: unknown } | undefined; try { - await this.#run(signal); - if (signal.aborted) { - break; - } - throw new Error(`Listener '${this.name}' exited unexpectedly.`); + await this.#run(attempt); + if (!attempt.aborted) throw new Error(`Listener '${this.name}' exited unexpectedly.`); } catch (error) { - if (signal.aborted || isAbortError(error)) { - break; + if (!(attempt.aborted && (isAbortError(error) || error === attempt.reason))) { + failure = { error }; } - - this.#resetHealthyBudget(); - this.#runningSince = undefined; + } finally { + if (failure) generation.done.reject(failure.error); + else generation.done.resolve(); + if (this.#generation === generation) this.#generation = undefined; + } + if (!failure) continue; + this.#resetHealthyBudget(); + this.#runningSince = undefined; + this.#recordError(failure.error); + if (signal.aborted || this.#restartBudgetUsed >= this.#maxRestarts) { + this.#status = 'failed'; + this.#health.failed(); + this.#onFailure?.(failure.error, this.snapshot()); + return; + } + this.#restartCount = Math.min(Number.MAX_SAFE_INTEGER, this.#restartCount + 1); + this.#restartBudgetUsed = Math.min(Number.MAX_SAFE_INTEGER, this.#restartBudgetUsed + 1); + this.#health.restarting(); + this.#status = this.#paused ? 'paused' : 'restarting'; + this.#onFailure?.(failure.error, this.snapshot()); + if (this.#paused) continue; + try { + await this.#sleep(this.#nextBackoffMs(), signal); + } catch (error) { + if (signal.aborted || isAbortError(error)) break; this.#recordError(error); - - if (this.#restartBudgetUsed >= this.#maxRestarts) { - this.#status = 'failed'; - this.#health.failed(); - this.#onFailure?.(error, this.snapshot()); - return; - } - - this.#restartCount = Math.min(Number.MAX_SAFE_INTEGER, this.#restartCount + 1); - this.#restartBudgetUsed = Math.min(Number.MAX_SAFE_INTEGER, this.#restartBudgetUsed + 1); - this.#health.restarting(); - this.#status = 'restarting'; + this.#status = 'failed'; + this.#health.failed(); this.#onFailure?.(error, this.snapshot()); - try { - await this.#sleep(this.#nextBackoffMs(), signal); - } catch (sleepError) { - if (signal.aborted || isAbortError(sleepError)) break; - this.#recordError(sleepError); - this.#status = 'failed'; - this.#health.failed(); - this.#onFailure?.(sleepError, this.snapshot()); - return; - } + return; } } this.#status = 'stopped'; @@ -225,23 +279,6 @@ function isAbortError(error: unknown): boolean { return error instanceof Error && error.name === 'AbortError'; } -function combineSignals(first: AbortSignal, second?: AbortSignal): AbortSignal { - if (!second) { - return first; - } - - const controller = new AbortController(); - const abort = () => controller.abort(); - if (first.aborted || second.aborted) { - controller.abort(); - return controller.signal; - } - - first.addEventListener('abort', abort, { once: true }); - second.addEventListener('abort', abort, { once: true }); - return controller.signal; -} - /** Map backend errors to fixed categories; exception text never enters public health. */ function classifyFailure(error: unknown): WorkerListenerFailureCause { let current = error; diff --git a/plugins/workers/worker/mod.ts b/plugins/workers/worker/mod.ts index 38f7da9dcc..094aa8ca9f 100644 --- a/plugins/workers/worker/mod.ts +++ b/plugins/workers/worker/mod.ts @@ -92,3 +92,9 @@ export { JOB_DEAD_LETTER_NAMESPACE } from './job-dead-letters.ts'; export type { DeadLetterRecord, DeadLetterStorePort } from '@netscript/queue'; export type { JobRunnerClock } from '@netscript/plugin-workers-core/runtime'; + +export type { + WorkerDrainClock, + WorkerDrainOptions, + WorkerLifecycleState, +} from '@netscript/plugin-workers-core/shutdown'; diff --git a/plugins/workers/worker/queue-consumer.ts b/plugins/workers/worker/queue-consumer.ts index d8b738d4f7..ab27e49173 100644 --- a/plugins/workers/worker/queue-consumer.ts +++ b/plugins/workers/worker/queue-consumer.ts @@ -1,11 +1,11 @@ -import { createQueue, type MessageContext } from '@netscript/queue'; +import { createQueue, type MessageContext, type MessageQueue } from '@netscript/queue'; import { DEFAULT_TOPIC, type JobDefinition, type JobMessage, type TaskMessage, } from '@netscript/plugin-workers-core/runtime'; -import type { TracedMessageContext, TracedQueue } from '@netscript/telemetry/instrumentation'; +import type { TracedMessageContext } from '@netscript/telemetry/instrumentation'; import type { QueueTriggerConfig, WorkerDispatchContext, @@ -21,6 +21,7 @@ export async function startQueueTriggerListeners( const supervisors: WorkerListenerSupervisor[] = []; for (const config of context.queueTriggers) { const jobDef = await context.registry.get(config.jobId); + if (context.abortController?.signal.aborted) break; if (!jobDef) { console.warn( `[Worker ${context.workerId}] Queue trigger job '${config.jobId}' not found, skipping`, @@ -32,7 +33,7 @@ export async function startQueueTriggerListeners( `[Worker ${context.workerId}] Setting up queue trigger: ${config.queueName} -> ${config.jobId}`, ); - const triggerQueue = createQueue(config.queueName) as TracedQueue; + const triggerQueue = createQueue(config.queueName); context.triggerQueues.push(triggerQueue); const supervisor = createListenerSupervisor( @@ -56,6 +57,7 @@ export function startTaskQueueListener( const supervisor = createListenerSupervisor(queueContext, 'task:tasks', (signal) => { return taskQueue.listen( async (message, context) => { + if (queueContext.canConsume && !await queueContext.canConsume(context)) return; try { await processWorkerTask(dispatchContext, message, context); } catch (error) { @@ -63,11 +65,10 @@ export function startTaskQueueListener( `[Worker ${queueContext.workerId}] Unexpected error processing task '${message.taskId}':`, error instanceof Error ? error.message : String(error), ); - if (!context.extendVisibility) throw error; await context.nack({ requeue: true }); } }, - { signal }, + { signal, drainTimeoutMs: Infinity }, ); }); supervisor.start(); @@ -95,13 +96,14 @@ function createListenerSupervisor( async function listenToTriggerQueue( context: WorkerQueueContext, - queue: TracedQueue, + queue: MessageQueue, config: QueueTriggerConfig, jobDef: JobDefinition, signal: AbortSignal, ): Promise { await queue.listen( async (message, queueContext) => { + if (context.canConsume && !await context.canConsume(queueContext)) return; let validatedMessage = message; if (config.schema) { try { @@ -133,6 +135,7 @@ async function listenToTriggerQueue( { concurrency: config.concurrency ?? 1, signal, + drainTimeoutMs: Infinity, }, ); } diff --git a/plugins/workers/worker/shutdown-first-cause_test.ts b/plugins/workers/worker/shutdown-first-cause_test.ts new file mode 100644 index 0000000000..a4f30946d1 --- /dev/null +++ b/plugins/workers/worker/shutdown-first-cause_test.ts @@ -0,0 +1,90 @@ +import { assert, assertEquals } from '@std/assert'; +import { MemoryKvAdapter } from '@netscript/kv'; +import { KvWorkerIdempotencyStore } from '@netscript/plugin-workers-core/stores'; +import { MemoryQueueAdapter } from '@netscript/queue/testing'; +import type { JobMessage } from '@netscript/plugin-workers-core/runtime'; +import { bounded } from '../../../packages/queue/tests/_fixtures/redis-lease-harness.ts'; +import { recoveryContext } from './test_utils/redis-worker-context.ts'; +import { processWorkerJob } from './job-dispatcher.ts'; + +for (const mode of ['business', 'deadline', 'backoff', 'cancel'] as const) { + Deno.test(`shutdown does not refund an earlier ${mode} cause`, async () => { + await using kv = new MemoryKvAdapter(); + const shutdown = new AbortController(); + const waiting = Promise.withResolvers(); + const context = recoveryContext(new KvWorkerIdempotencyStore({ kv }), { + execute: (_definition, options) => { + if (mode === 'backoff') { + return Promise.resolve({ success: false, duration: 1, error: 'business failure' }); + } + if (mode === 'cancel') { + const active = [...context.activeJobs.values()][0]; + assert(active); + active.abortController.abort(new DOMException('Operator first.', 'AbortError')); + } + shutdown.abort(new DOMException('Later shutdown.', 'ShutdownError')); + if (mode === 'business') throw new Error('Handler first.'); + if (mode === 'deadline') throw new DOMException('Deadline first.', 'TimeoutError'); + assert(options.signal instanceof AbortSignal); + return Promise.resolve({ success: false, duration: 1 }); + }, + }, shutdown.signal); + const dispatch = { + ...context, + registry: { + get: async (id: string) => { + const definition = await context.registry.get(id); + return definition && + { ...definition, maxRetries: mode === 'backoff' ? 2 : 0, retryDelay: 1_000 }; + }, + }, + retryClock: { + now: () => 0, + schedule: () => { + waiting.resolve(); + return () => {}; + }, + }, + }; + const queue = new MemoryQueueAdapter(); + await queue.enqueue({ jobId: 'cause', topic: 'jobs', triggeredBy: 'queue' }); + const firstStop = new AbortController(); + const first = queue.listen(async (message, delivery) => { + await processWorkerJob(dispatch, message, delivery); + firstStop.abort(); + }, { signal: firstStop.signal }); + void first.catch(() => {}); + try { + if (mode === 'backoff') { + await bounded(waiting.promise); + shutdown.abort(new DOMException('Later shutdown.', 'ShutdownError')); + } + await bounded(first); + const nextStop = new AbortController(); + const delivered = Promise.withResolvers(); + const next = queue.listen(async (_message, delivery) => { + try { + assertEquals(delivery.deliveryCount, 2); + await delivery.ack(); + delivered.resolve(); + } catch (error) { + delivered.reject(error); + throw error; + } finally { + nextStop.abort(); + } + }, { signal: nextStop.signal }); + try { + await bounded(delivered.promise); + } finally { + nextStop.abort(); + await bounded(next); + } + } finally { + firstStop.abort(); + shutdown.abort(); + await bounded(first); + await queue.stop(); + } + }); +} diff --git a/plugins/workers/worker/worker-abort-lifecycle_test.ts b/plugins/workers/worker/worker-abort-lifecycle_test.ts new file mode 100644 index 0000000000..b68606ba74 --- /dev/null +++ b/plugins/workers/worker/worker-abort-lifecycle_test.ts @@ -0,0 +1,272 @@ +import { assert, assertEquals, assertInstanceOf, assertStrictEquals } from '@std/assert'; +import { MemoryKvAdapter } from '@netscript/kv'; +import { MemoryQueueAdapter } from '@netscript/queue/testing'; +import { KvWorkerIdempotencyStore } from '@netscript/plugin-workers-core/stores'; +import type { + JobContext, + JobDefinition, + JobMessage, + TaskMessage, +} from '@netscript/plugin-workers-core/runtime'; +import type { WorkerCompleteExecutionOptions, WorkerExecutionState } from './worker-options.ts'; +import { Worker } from './worker.ts'; + +async function bounded(promise: Promise, timeoutMs = 2_000): Promise { + let timer: ReturnType | undefined; + try { + return await Promise.race([ + promise, + new Promise((_resolve, reject) => { + timer = setTimeout(() => reject(new Error('Lifecycle did not settle.')), timeoutMs); + }), + ]); + } finally { + clearTimeout(timer); + } +} + +for (const cause of ['timeout', 'cancel', 'shutdown'] as const) { + Deno.test(`Worker queue-to-Deno handler aborts on ${cause} and records the terminal cause`, async () => { + await using kv = new MemoryKvAdapter(); + const queue = new MemoryQueueAdapter({ pollInterval: 1 }); + const taskQueue = new MemoryQueueAdapter({ pollInterval: 1 }); + const idempotency = new KvWorkerIdempotencyStore({ kv }); + const ready = Promise.withResolvers(); + const finish = Promise.withResolvers(); + const terminal = Promise.withResolvers(); + const completed: WorkerCompleteExecutionOptions[] = []; + const executionState: WorkerExecutionState = { + create: () => Promise.resolve({ id: 'execution-1' }), + start: () => Promise.resolve({ id: 'execution-1' }), + progress: () => Promise.resolve({ id: 'execution-1' }), + complete: (_id, options) => { + completed.push(options); + terminal.resolve(options); + return Promise.resolve({ id: 'execution-1' }); + }, + }; + const job: JobDefinition = { + id: `lifecycle-${cause}`, + enabled: true, + topic: 'jobs', + executionType: 'deno', + entrypoint: import.meta.url, + timeout: cause === 'timeout' ? 100 : 10_000, + handler: async (context) => { + ready.resolve(context); + await Promise.race([ + new Promise((resolve) => { + if (context.signal.aborted) resolve(); + else context.signal.addEventListener('abort', () => resolve(), { once: true }); + }), + finish.promise, + ]); + return { success: true }; + }, + }; + const worker = new Worker({ + workerId: 'cancellation-lifecycle', + queue, + taskQueue, + registry: { get: () => Promise.resolve(job) }, + executionState, + taskRegistry: { get: () => Promise.resolve(undefined) }, + taskExecutor: { execute: () => Promise.reject(new Error('Task path must not run.')) }, + idempotency, + drainTimeoutMs: 200, + workerPoolOptions: { abortGracePeriodMs: 5 }, + }); + const started = worker.start(); + void started.catch(() => undefined); + const claimInput = { concept: 'job' as const, targetId: job.id, idempotencyKey: 'delivery-1' }; + try { + await queue.enqueue({ + jobId: job.id, + topic: 'jobs', + triggeredBy: 'manual', + idempotencyKey: 'delivery-1', + }); + const context = await bounded(ready.promise); + assertInstanceOf(context.signal, AbortSignal); + assert(typeof context.deadlineAt === 'number'); + assertEquals(worker.activeJobCount, 1); + assertEquals(worker.cancel('absent'), false); + let stopped: Promise | undefined; + if (cause === 'cancel') { + assertEquals(worker.cancel('execution-1'), true); + assertEquals(worker.cancel('execution-1'), false); + assert(context.signal.aborted); + } else if (cause === 'shutdown') { + stopped = worker.stop(); + assertEquals(worker.stop(), stopped); + assertEquals(context.signal.aborted, false); + } + const result = await bounded(terminal.promise); + assert(context.signal.aborted); + assertInstanceOf(context.signal.reason, DOMException); + assertEquals( + context.signal.reason.name, + cause === 'timeout' + ? 'TimeoutError' + : cause === 'shutdown' + ? 'ShutdownError' + : 'AbortError', + ); + assertEquals(result.status, cause === 'timeout' ? 'timeout' : 'cancelled'); + await bounded(stopped ?? worker.stop()); + await bounded(started); + assertEquals(worker.activeJobCount, 0); + assertEquals(worker.isRunning, false); + assertEquals(completed.length, 1); + const retry = await idempotency.claim(claimInput); + assertEquals(retry.claimed, true); + assertEquals(retry.alreadyApplied, false); + await idempotency.release(retry.key); + } finally { + finish.resolve(); + await bounded(worker.stop()); + await bounded(started); + } + }); +} + +async function verifyBlockedProgressWorker( + cause: 'timeout' | 'cancel' | 'shutdown', + reentrant = false, +): Promise { + await using kv = new MemoryKvAdapter(); + const queue = new MemoryQueueAdapter({ pollInterval: 1 }); + const taskQueue = new MemoryQueueAdapter({ pollInterval: 1 }); + let queueStops = 0; + let taskStops = 0; + const stopQueue = queue.stop.bind(queue); + const stopTasks = taskQueue.stop.bind(taskQueue); + queue.stop = async () => { + queueStops++; + await stopQueue(); + }; + taskQueue.stop = async () => { + taskStops++; + await stopTasks(); + }; + const idempotency = new KvWorkerIdempotencyStore({ kv }); + const ready = Promise.withResolvers(); + const sinkEntered = Promise.withResolvers(); + const finishProgress = Promise.withResolvers(); + const terminal = Promise.withResolvers(); + const completed: WorkerCompleteExecutionOptions[] = []; + const executionState: WorkerExecutionState = { + create: () => Promise.resolve({ id: 'execution-1' }), + start: () => Promise.resolve({ id: 'execution-1' }), + progress: async () => { + sinkEntered.resolve(); + await finishProgress.promise; + return { id: 'execution-1' }; + }, + complete: (_id, options) => { + completed.push(options); + terminal.resolve(options); + return Promise.resolve({ id: 'execution-1' }); + }, + }; + const job: JobDefinition = { + id: `progress-drain-${cause}`, + enabled: true, + topic: 'jobs', + executionType: 'deno', + entrypoint: import.meta.url, + timeout: cause === 'timeout' ? 500 : 10000, + handler: (context) => { + ready.resolve(context); + void context.reportProgress?.(10); + return { success: true }; + }, + }; + const worker = new Worker({ + workerId: 'progress-drain-lifecycle', + queue, + taskQueue, + registry: { get: () => Promise.resolve(job) }, + executionState, + taskRegistry: { get: () => Promise.resolve(undefined) }, + taskExecutor: { execute: () => Promise.reject(new Error('Task path must not run.')) }, + idempotency, + drainTimeoutMs: 200, + workerPoolOptions: { abortGracePeriodMs: 5 }, + }); + const started = worker.start(); + void started.catch(() => undefined); + const claimInput = { + concept: 'job' as const, + targetId: job.id, + idempotencyKey: 'progress-delivery', + }; + const settleStop = async (completion: Promise) => { + try { + await bounded(completion); + } catch (error) { + // A sink held beyond cleanup may exhaust the stop budget; it must remain bounded. + assertEquals(cause, 'shutdown'); + assertInstanceOf(error, DOMException); + assertEquals(error.name, 'TimeoutError'); + } + }; + let nestedStop: Promise | undefined; + try { + await queue.enqueue({ + jobId: job.id, + topic: 'jobs', + triggeredBy: 'manual', + idempotencyKey: 'progress-delivery', + }); + const context = await bounded(ready.promise); + await bounded(sinkEntered.promise); + await new Promise((resolve) => setTimeout(resolve, 0)); + if (reentrant) { + context.signal.addEventListener('abort', () => { + nestedStop = worker.stop(); + }, { once: true }); + } + let stopped: Promise | undefined; + if (cause === 'cancel') { + assertEquals(worker.cancel('execution-1'), true); + assertEquals(context.signal.aborted, true); + } else if (cause === 'shutdown') { + stopped = worker.stop(); + assertEquals(worker.stop(), stopped); + } + const result = await bounded(terminal.promise); + if (reentrant) assertStrictEquals(nestedStop, stopped); + assertEquals(context.signal.aborted, true); + assertInstanceOf(context.signal.reason, DOMException); + assertEquals( + context.signal.reason.name, + cause === 'cancel' ? 'AbortError' : cause === 'timeout' ? 'TimeoutError' : 'ShutdownError', + ); + assertEquals(result.status, cause === 'timeout' ? 'timeout' : 'cancelled'); + // Neither stop nor claim release waits indefinitely on the still-blocked sink. + await settleStop(stopped ?? worker.stop()); + await settleStop(started); + assertEquals(worker.activeJobCount, 0); + assertEquals(worker.isRunning, false); + assertEquals(completed.length, 1); + assertEquals(queueStops, 1); + assertEquals(taskStops, 1); + const retry = await idempotency.claim(claimInput); + assertEquals(retry.claimed, true); + assertEquals(retry.alreadyApplied, false); + await idempotency.release(retry.key); + } finally { + finishProgress.resolve(); + await settleStop(worker.stop()); + if (nestedStop) await settleStop(nestedStop); + await settleStop(started); + } +} + +for (const cause of ['timeout', 'cancel', 'shutdown'] as const) { + Deno.test(`Worker cancellation during unawaited progress drain retains ${cause} ownership`, () => + verifyBlockedProgressWorker(cause)); +} +Deno.test('Worker reentrant stop from abort listener shares completion and cleanup', () => + verifyBlockedProgressWorker('shutdown', true)); diff --git a/plugins/workers/worker/worker-intake-lifecycle_test.ts b/plugins/workers/worker/worker-intake-lifecycle_test.ts new file mode 100644 index 0000000000..f9ee004f5f --- /dev/null +++ b/plugins/workers/worker/worker-intake-lifecycle_test.ts @@ -0,0 +1,217 @@ +import { assert, assertEquals, assertRejects, assertStrictEquals } from '@std/assert'; +import { delay } from '@std/async'; +import { stub } from '@std/testing/mock'; +import { MemoryKvAdapter } from '@netscript/kv'; +import { KvWorkerIdempotencyStore } from '@netscript/plugin-workers-core/stores'; +import { DenoKvAdapter } from '@netscript/queue/adapters/deno-kv'; +import { MemoryQueueAdapter } from '@netscript/queue/testing'; +import type { ListenOptions, MessageContext, MessageQueue, NackOptions } from '@netscript/queue'; +import type { JobMessage, TaskMessage } from '@netscript/plugin-workers-core/runtime'; +import { bounded, until } from '../../../packages/queue/tests/_fixtures/redis-lease-harness.ts'; +import { recoveryContext } from './test_utils/redis-worker-context.ts'; +import { Worker } from './worker.ts'; + +for (const family of ['jobs', 'tasks', 'triggers'] as const) { + for (const mode of ['pause', 'drain'] as const) { + Deno.test(`${family} ${mode} retains admitted work and owns the backend drain`, async () => { + await using kv = new MemoryKvAdapter(); + const entered = Promise.withResolvers(); + const release = Promise.withResolvers(); + let calls = 0; + let completed = 0; + const context = recoveryContext(new KvWorkerIdempotencyStore({ kv }), { + execute: async (_definition, options) => { + assert(options.signal instanceof AbortSignal); + calls++; + entered.resolve(options.signal); + await release.promise; + return { success: true, duration: 1 }; + }, + }); + const jobs = new MemoryQueueAdapter(); + const tasks = new MemoryQueueAdapter(); + const triggers = new MemoryQueueAdapter(); + const seen = new Map(); + const observed = (queue: MemoryQueueAdapter, name: string): MessageQueue => ({ + nativeRetrial: queue.nativeRetrial, + enqueue: (message, options) => queue.enqueue(message, options), + stop: () => queue.stop(), + listen: (handler, options) => { + seen.set(name, options); + return queue.listen(handler, options); + }, + }); + using _triggers = stub(DenoKvAdapter.prototype, 'listen', (handler, options) => { + seen.set('triggers', options); + return triggers.listen(handler, options); + }); + const worker = new Worker({ + ...context, + queue: observed(jobs, 'jobs'), + taskQueue: observed(tasks, 'tasks'), + queueTriggers: [{ queueName: 'lifecycle-events', jobId: 'event' }], + drainTimeoutMs: 2_000, + abortGracePeriodMs: 500, + executionState: { + ...context.executionState, + complete: (id, result) => { + if (result.status === 'completed') completed++; + return Promise.resolve({ id }); + }, + }, + }); + const enqueue = () => + family === 'jobs' + ? jobs.enqueue({ jobId: 'job', topic: 'jobs', triggeredBy: 'queue' }) + : family === 'tasks' + ? tasks.enqueue({ taskId: 'task', topic: 'tasks', triggeredBy: 'queue' }) + : triggers.enqueue({ event: true }); + let finished = false; + const started = worker.start().finally(() => finished = true); + void started.catch(() => {}); + let stopping: Promise | undefined; + try { + await until(() => Promise.resolve(seen.size === 3)); + for (const options of seen.values()) assertEquals(options?.drainTimeoutMs, Infinity); + await enqueue(); + const signal = await bounded(entered.promise); + if (mode === 'pause') worker.pause(); + else stopping = worker.stop(); + await enqueue(); + assertEquals(signal.aborted, false); + if (mode === 'pause') { + assertEquals(worker.healthStatus.lifecycle, 'paused'); + assertEquals(worker.healthStatus.status, 'healthy'); + assertEquals(worker.isRunning, true); + worker.resume(); + await delay(30); + assertEquals(calls, 1, 'Resume over-admitted before the old generation settled.'); + worker.pause(); + worker.pause(); + } + release.resolve(); + await until(() => Promise.resolve(completed === 1)); + await delay(30); + assertEquals(calls, 1); + if (mode === 'pause') { + assertEquals(finished, false); + assertEquals( + worker.healthStatus.listeners.every((listener) => listener.restartCount === 0), + true, + ); + worker.resume(); + await until(() => Promise.resolve(completed === 2)); + } else await bounded(stopping!); + } finally { + release.resolve(); + await bounded(stopping ?? worker.stop()); + await bounded(started); + await triggers.stop(); + } + }); + } +} + +Deno.test('a committed paused claim is charged while a draining claim preserves its attempt', async () => { + await using kv = new MemoryKvAdapter(); + let executions = 0; + const context = recoveryContext(new KvWorkerIdempotencyStore({ kv }), { + execute: () => { + executions++; + return Promise.resolve({ success: true, duration: 1 }); + }, + }); + const ready = Promise.withResolvers(); + let deliver: Parameters['listen']>[0] | undefined; + const worker = new Worker({ + ...context, + taskQueue: new MemoryQueueAdapter(), + queue: { + nativeRetrial: true, + enqueue: () => Promise.resolve(), + stop: () => Promise.resolve(), + listen: (handler, options) => { + deliver = handler; + ready.resolve(); + return new Promise((resolve) => { + options?.signal?.addEventListener('abort', () => resolve(), { once: true }); + }); + }, + }, + }); + const recorded: NackOptions[] = []; + const delivery: MessageContext = { + messageId: 'committed', + deliveryCount: 1, + enqueuedAt: new Date(), + headers: {}, + ack: () => Promise.resolve(), + nack: (options = {}) => { + recorded.push(options); + return Promise.resolve(); + }, + }; + const message: JobMessage = { jobId: 'raced', topic: 'jobs', triggeredBy: 'queue' }; + const started = worker.start(); + try { + await bounded(ready.promise); + assert(deliver); + worker.pause(); + await deliver(message, delivery); + const stopped = worker.stop(); + await deliver(message, delivery); + await bounded(stopped); + assertEquals(recorded, [{ requeue: true }, { requeue: true, preserveDeliveryCount: true }]); + assertEquals(executions, 0); + } finally { + await worker.stop(); + await started; + } +}); + +Deno.test('stop during initialization is bounded and prevents late listener allocation', async () => { + await using kv = new MemoryKvAdapter(); + const context = recoveryContext(new KvWorkerIdempotencyStore({ kv }), { + execute: () => Promise.resolve({ success: true, duration: 1 }), + }); + const entered = Promise.withResolvers(); + const release = Promise.withResolvers(); + let allocated = 0; + using _listener = stub(DenoKvAdapter.prototype, 'listen', () => { + allocated++; + return Promise.resolve(); + }); + const worker = new Worker({ + ...context, + queue: new MemoryQueueAdapter(), + taskQueue: new MemoryQueueAdapter(), + drainTimeoutMs: 100, + abortGracePeriodMs: 20, + registry: { + get: async (id) => { + entered.resolve(); + await release.promise; + return context.registry.get(id); + }, + }, + queueTriggers: [{ queueName: 'late', jobId: 'late' }], + }); + const started = worker.start(); + try { + await entered.promise; + await assertRejects( + () => bounded(worker.stop(), 1_000), + DOMException, + 'Worker drain budget elapsed.', + ); + } finally { + release.resolve(); + await bounded(started); + } + assertEquals(allocated, 0); + assertEquals(worker.healthStatus.lifecycle, 'stopped'); + assertStrictEquals(worker.stop(), worker.stop()); + worker.resume(); + worker.pause(); + await assertRejects(() => worker.start(), Error, 'stopped Worker'); +}); diff --git a/plugins/workers/worker/worker-options.ts b/plugins/workers/worker/worker-options.ts index 4745748936..c7087a52ef 100644 --- a/plugins/workers/worker/worker-options.ts +++ b/plugins/workers/worker/worker-options.ts @@ -1,3 +1,8 @@ +import type { ChildHealthSnapshot } from '@netscript/plugin/health'; +import type { + WorkerDrainOptions, + WorkerLifecycleState, +} from '@netscript/plugin-workers-core/shutdown'; import type { JobDeadLetterStorePort } from './job-dead-letters.ts'; import type { MessageContext, MessageQueue } from '@netscript/queue'; import type { @@ -13,7 +18,7 @@ import type { import type { TracedMessageContext } from '@netscript/telemetry/instrumentation'; import type { Span } from '@netscript/telemetry/tracer'; import type { WorkerPool, WorkerPoolOptions } from './job-runner-pool.ts'; -import type { WorkerListenerSnapshot } from './listener-supervisor.ts'; +import type { WorkerListenerFailureCause, WorkerListenerSnapshot } from './listener-supervisor.ts'; /** Structural validation schema accepted by worker queue triggers. */ export interface WorkerPayloadSchema { @@ -129,7 +134,7 @@ export interface QueueTriggerConfig { } /** Worker configuration options. */ -export interface WorkerOptions { +export interface WorkerOptions extends WorkerDrainOptions { /** Worker identifier. */ workerId: string; /** Queue name to consume from. */ @@ -228,6 +233,7 @@ export interface WorkerDispatchContext { /** Context required by queue-consumer helpers. */ export interface WorkerQueueContext { readonly workerId: string; + readonly canConsume?: (delivery: MessageContext) => Promise; readonly registry: WorkerJobRegistry; readonly queueTriggers: readonly QueueTriggerConfig[]; readonly triggerQueues: MessageQueue[]; @@ -252,3 +258,32 @@ export function resolveWorkerQueueTriggers( ): readonly QueueTriggerConfig[] { return Object.freeze([...(queueTriggers ?? [])]); } + +/** Health snapshot for a worker runtime. */ +export interface WorkerHealthStatus { + /** Lifecycle of this worker instance. */ + readonly lifecycle: WorkerLifecycleState; + /** Durable job dead-letter sample, populated by readHealthStatus(). */ + readonly deadLetters?: Readonly<{ available: boolean; depth: number | null }>; + /** Aggregate worker health state. */ + readonly status: 'healthy' | 'degraded'; + /** Shared child health contract for the background health route. */ + readonly childHealth: ChildHealthSnapshot; + /** Queue listener health snapshots. */ + readonly listeners: readonly { + /** Listener name. */ + readonly name: string; + /** Listener lifecycle status. */ + readonly status: 'idle' | 'running' | 'paused' | 'restarting' | 'failed' | 'stopped'; + /** Whether the listener is healthy. */ + readonly healthy: boolean; + /** Cumulative restart attempts for this listener instance. */ + readonly restartCount: number; + /** Restart attempts charged since the last healthy running interval. */ + readonly restartBudgetUsed: number; + /** Redacted cause of the most recent failure, retained as incident history on recovery. */ + readonly failureCause?: WorkerListenerFailureCause; + /** Last listener failure message. */ + readonly lastError?: string; + }[]; +} diff --git a/plugins/workers/worker/worker-shutdown-deliveries_test.ts b/plugins/workers/worker/worker-shutdown-deliveries_test.ts index c63e593286..e5ad0a9f79 100644 --- a/plugins/workers/worker/worker-shutdown-deliveries_test.ts +++ b/plugins/workers/worker/worker-shutdown-deliveries_test.ts @@ -44,8 +44,10 @@ for (const [backend, url] of leaseBackends) { }); const workerOptions = { ...dispatch, - drainTimeoutMs: 100, - abortGracePeriodMs: 20, + // Redis already reserves 100 ms for disconnect callbacks after closing sockets. + // Keep the work window short while retaining enough total budget for that close. + drainTimeoutMs: 500, + abortGracePeriodMs: 350, taskQueue: new MemoryQueueAdapter(), queue: { nativeRetrial: jobs.nativeRetrial, diff --git a/plugins/workers/worker/worker.ts b/plugins/workers/worker/worker.ts index bce45f3e13..cb32747aa4 100644 --- a/plugins/workers/worker/worker.ts +++ b/plugins/workers/worker/worker.ts @@ -9,8 +9,13 @@ import { */ import { ChildHealthMonitor, type ChildHealthSnapshot } from '@netscript/plugin/health'; -import { delay } from '@std/async'; -import { createQueue, type MessageQueue } from '@netscript/queue'; +import { + drainWorker, + resolveWorkerDrainBudget, + type WorkerDrainOptions, + type WorkerLifecycleState, +} from '@netscript/plugin-workers-core/shutdown'; +import { createQueue, type MessageContext, type MessageQueue } from '@netscript/queue'; import type { JobRunnerClock, WorkerIdempotencyPort } from '@netscript/plugin-workers-core/runtime'; import type { JobMessage, TaskMessage } from '@netscript/plugin-workers-core/runtime'; import { createWorkerPool, type WorkerPool } from './job-runner-pool.ts'; @@ -21,7 +26,6 @@ import type { Span } from '@netscript/telemetry/tracer'; import { processWorkerJob } from './job-dispatcher.ts'; import { resolveListenerHealthyResetMs, - type WorkerListenerFailureCause, type WorkerListenerSnapshot, WorkerListenerSupervisor, } from './listener-supervisor.ts'; @@ -33,6 +37,7 @@ import { resolveWorkerQueueTriggers, type WorkerDispatchContext, type WorkerExecutionState, + type WorkerHealthStatus, type WorkerJobRegistry, type WorkerOptions, type WorkerQueueContext, @@ -46,6 +51,7 @@ export type { WorkerCreateExecutionOptions, WorkerExecutionRecord, WorkerExecutionState, + WorkerHealthStatus, WorkerJobRegistry, WorkerOptions, WorkerPayloadSchema, @@ -54,33 +60,6 @@ export type { WorkerTaskResult, } from './worker-options.ts'; -/** Health snapshot for a worker runtime. */ -export interface WorkerHealthStatus { - /** Durable job dead-letter sample, populated by readHealthStatus(). */ - readonly deadLetters?: Readonly<{ available: boolean; depth: number | null }>; - /** Aggregate worker health state. */ - readonly status: 'healthy' | 'degraded'; - /** Shared child health contract for the background health route. */ - readonly childHealth: ChildHealthSnapshot; - /** Queue listener health snapshots. */ - readonly listeners: readonly { - /** Listener name. */ - readonly name: string; - /** Listener lifecycle status. */ - readonly status: 'idle' | 'running' | 'restarting' | 'failed' | 'stopped'; - /** Whether the listener is healthy. */ - readonly healthy: boolean; - /** Cumulative restart attempts for this listener instance. */ - readonly restartCount: number; - /** Restart attempts charged since the last healthy running interval. */ - readonly restartBudgetUsed: number; - /** Redacted cause of the most recent failure, retained as incident history on recovery. */ - readonly failureCause?: WorkerListenerFailureCause; - /** Last listener failure message. */ - readonly lastError?: string; - }[]; -} - /** Worker process that consumes queued jobs and tasks for one runtime instance. */ export class Worker { /** Durable terminal job store; also supports listing and controlled reprocessing. */ @@ -108,7 +87,9 @@ export class Worker { private readonly suppliedTaskQueue?: MessageQueue; private queue: MessageQueue | null = null; private taskQueue: MessageQueue | null = null; - private running = false; + private lifecycle: WorkerLifecycleState = 'idle'; + private startup: Promise = Promise.resolve(); + private readonly drainOptions: WorkerDrainOptions; private readonly childHealth = new ChildHealthMonitor(); private stoppedListeners: readonly WorkerListenerSnapshot[] = []; private stopCompletion: Promise | null = null; @@ -117,12 +98,24 @@ export class Worker { /** Create a worker with queue, registry, execution, and task runtime dependencies. */ constructor(options: WorkerOptions) { + const budget = resolveWorkerDrainBudget({ + ...options, + abortGracePeriodMs: options.abortGracePeriodMs ?? + options.workerPoolOptions?.abortGracePeriodMs, + }); + this.drainOptions = { + drainTimeoutMs: budget.totalMs, + abortGracePeriodMs: budget.abortGracePeriodMs, + drainClock: options.drainClock, + }; this.listenerHealthyResetMs = resolveListenerHealthyResetMs(options.listenerHealthyResetMs); this.workerId = options.workerId; this.queueName = options.queueName ?? 'jobs'; this.deadLetters = options.deadLetters ?? createDefaultJobDeadLetterStore(this.queueName); this.suppliedQueue = options.queue; this.suppliedTaskQueue = options.taskQueue; + this.queue = options.queue ?? null; + this.taskQueue = options.taskQueue ?? null; this.concurrency = options.concurrency ?? 1; this.registry = options.registry; this.executionState = options.executionState; @@ -139,6 +132,7 @@ export class Worker { poolSize: this.concurrency, ...options.workerPoolOptions, workerUrl: options.workerUrl, + abortGracePeriodMs: budget.abortGracePeriodMs, }); if (isTelemetryEnabled()) { @@ -154,9 +148,29 @@ export class Worker { return this.workerId; } - /** Whether the worker is currently consuming queues. */ + /** Whether the runtime is alive, including paused or draining work. */ get isRunning(): boolean { - return this.running; + return this.lifecycle === 'running' || this.lifecycle === 'paused' || + this.lifecycle === 'draining'; + } + + /** Whether intake was intentionally paused. */ + get isPaused(): boolean { + return this.lifecycle === 'paused'; + } + + /** Stop intake immediately; admitted jobs and tasks continue on their own signals. */ + pause(): void { + if (this.lifecycle !== 'running') return; + this.lifecycle = 'paused'; + for (const listener of this.listenerSupervisors) void listener.pause().catch(() => {}); + } + + /** Resume once the previous intake generation has joined its admitted work. */ + resume(): void { + if (this.lifecycle !== 'paused') return; + this.lifecycle = 'running'; + for (const listener of this.listenerSupervisors) listener.resume(); } /** Number of active job executions. */ @@ -190,6 +204,7 @@ export class Worker { }); return { status: childHealth.state === 'ready' ? 'healthy' : 'degraded', + lifecycle: this.lifecycle, childHealth, listeners, }; @@ -211,59 +226,67 @@ export class Worker { } } - /** Start queue consumption for this worker instance. */ + /** Start this instance until stopped or its job listener exhausts recovery. */ async start(): Promise { - if (this.running) { - console.warn(`[Worker ${this.workerId}] Already running`); - return; + if (this.isRunning) return; + if (this.lifecycle === 'stopped') { + throw new Error('A stopped Worker cannot be restarted; create a replacement.'); } - - console.log( - `[Worker ${this.workerId}] Starting in-process job runner (queue concurrency: ${this.concurrency})...`, - ); - - this.running = true; + this.lifecycle = 'running'; this.childHealth.registryLoaded(); this.abortController = new AbortController(); - await this.workerPool.initialize(); - - this.workerSpan = startWorkerSpan({ - workerId: this.workerId, - queueName: this.queueName, - concurrency: this.concurrency, - }); - - this.queue = this.suppliedQueue ?? createQueue(this.queueName); - - this.listenerSupervisors.push(...await startQueueTriggerListeners(this.queueContext())); - this.listenerSupervisors.push( - startTaskQueueListener(this.queueContext(), this.dispatchContext()), - ); - - const jobListener = new WorkerListenerSupervisor({ - name: `jobs:${this.queueName}`, - abortSignal: this.abortController.signal, - maxRestarts: this.listenerMaxRestarts, - healthyResetMs: this.listenerHealthyResetMs, - initialBackoffMs: this.listenerInitialBackoffMs, - maxBackoffMs: this.listenerMaxBackoffMs, - run: (signal) => this.listenForJobs(signal), - onFailure: (error, snapshot) => this.reportListenerFailure(snapshot.name, error, snapshot), - }); - this.listenerSupervisors.push(jobListener); - jobListener.start(); - this.childHealth.dependenciesReady(); - this.childHealth.running(); - + const initialized = Promise.withResolvers(); + this.startup = initialized.promise; + let jobListener: WorkerListenerSupervisor | undefined; + let failure: { error: unknown } | undefined; try { - await jobListener.completion; - } finally { - if (jobListener.snapshot().status === 'failed') { - await this.stop(); - } else { - this.running = false; + await this.workerPool.initialize(); + if (!this.canStartListeners()) return; + this.workerSpan = startWorkerSpan({ + workerId: this.workerId, + queueName: this.queueName, + concurrency: this.concurrency, + }); + this.queue ??= this.suppliedQueue ?? createQueue(this.queueName); + const triggers = await startQueueTriggerListeners(this.queueContext()); + if (!this.canStartListeners()) { + await Promise.all(triggers.map((listener) => listener.stop())); + return; + } + this.listenerSupervisors.push(...triggers); + this.listenerSupervisors.push( + startTaskQueueListener(this.queueContext(), this.dispatchContext()), + ); + jobListener = new WorkerListenerSupervisor({ + name: `jobs:${this.queueName}`, + abortSignal: this.abortController.signal, + maxRestarts: this.listenerMaxRestarts, + healthyResetMs: this.listenerHealthyResetMs, + initialBackoffMs: this.listenerInitialBackoffMs, + maxBackoffMs: this.listenerMaxBackoffMs, + run: (signal) => this.listenForJobs(signal), + onFailure: (error, snapshot) => this.reportListenerFailure(snapshot.name, error, snapshot), + }); + this.listenerSupervisors.push(jobListener); + jobListener.start(); + if (this.isPaused) { + for (const listener of this.listenerSupervisors) void listener.pause().catch(() => {}); } + this.childHealth.dependenciesReady(); + this.childHealth.running(); + } catch (error) { + failure = { error }; + this.childHealth.failed(); + } finally { + initialized.resolve(); + } + if (failure) { + await this.stop(); + throw failure.error; } + await jobListener?.completion; + if (jobListener?.snapshot().status === 'failed') await this.stop(); + else if (this.stopCompletion) await this.stopCompletion; } /** Request cancellation of an active local execution; first cause wins. */ @@ -274,55 +297,59 @@ export class Worker { return true; } - /** Abort jobs at drain start and await listeners plus the bounded runner cleanup. */ + /** Wait for admitted work, then cancel leftovers and settle within one total budget. */ stop(): Promise { if (this.stopCompletion) return this.stopCompletion; - if (!this.running && !this.hasRuntimeResources()) { - return Promise.resolve(); + if (!this.isRunning && !this.hasRuntimeResources()) { + this.lifecycle = 'stopped'; + return this.stopCompletion = Promise.resolve(); } + this.lifecycle = 'draining'; const completion = Promise.withResolvers(); this.stopCompletion = completion.promise; void this.drain().then(completion.resolve, completion.reject); return completion.promise; } - /** Drain once; concurrent stop callers share the same completion. */ + /** One bounded shutdown; resource references remain owned even after a timeout. */ private async drain(): Promise { - console.log(`[Worker ${this.workerId}] Stopping...`); + let intake = Promise.resolve(); try { - const shutdownReason = new DOMException('Worker shutdown began.', 'ShutdownError'); - this.abortController?.abort(shutdownReason); - for (const execution of this.activeJobs.values()) { - execution.abortController.abort(shutdownReason); - } - await Promise.all([ - this.workerPool.shutdown(), - ...this.listenerSupervisors.map((supervisor) => supervisor.stop()), - ]); - this.stoppedListeners = this.listenerSupervisors.map((supervisor) => supervisor.snapshot()); + await drainWorker(this.drainOptions, { + pauseIntake: () => { + intake = this.startup.then(async () => { + await Promise.all(this.listenerSupervisors.map((listener) => listener.pause())); + }); + return intake; + }, + abort: () => + this.abortController?.abort(new DOMException('Worker shutdown began.', 'ShutdownError')), + abortCompleted: async () => { + await Promise.all([intake, this.workerPool.shutdown()]); + }, + close: async () => { + const listeners = [...this.listenerSupervisors]; + const queues = new Set([...this.triggerQueues, this.queue, this.taskQueue]); + try { + await Promise.all(listeners.map((listener) => listener.stop())); + } finally { + await Promise.all([...queues].map((queue) => queue?.stop())); + } + this.stoppedListeners = listeners.map((listener) => listener.snapshot()); + }, + }); + } finally { this.listenerSupervisors.length = 0; + this.triggerQueues.length = 0; this.childHealth.stopped(); - await this.stopTriggerQueues(); - await this.waitForActiveJobs(); - - if (this.queue) { - await this.queue.stop(); - this.queue = null; - } - if (this.taskQueue) { - await this.taskQueue.stop(); - this.taskQueue = null; - } - + this.lifecycle = 'stopped'; this.workerSpan?.setAttribute(WorkerAttributes.WORKER_ACTIVE_JOBS, 0); this.workerSpan?.addEvent('worker.stopped'); this.workerSpan?.end(); this.workerSpan = null; this.abortController = null; - this.running = false; - console.log(`[Worker ${this.workerId}] Stopped`); - } finally { - this.stopCompletion = null; + this.queue = null; + this.taskQueue = null; } } @@ -335,6 +362,7 @@ export class Worker { try { await this.queue.listen( async (message, context) => { + if (!await this.admit(context)) return; try { await processWorkerJob( this.dispatchContext(), @@ -350,7 +378,7 @@ export class Worker { throw error; } }, - { signal, concurrency: this.concurrency }, + { signal, concurrency: this.concurrency, drainTimeoutMs: Infinity }, ); } catch (error) { if (error instanceof Error && error.name === 'AbortError') { @@ -361,37 +389,21 @@ export class Worker { } } - /** Stop all trigger queues owned by this worker instance. */ - private async stopTriggerQueues(): Promise { - for (const triggerQueue of this.triggerQueues) { - try { - await triggerQueue.stop(); - } catch (error) { - console.error(`[Worker ${this.workerId}] Error stopping trigger queue:`, error); - } - } - this.triggerQueues.length = 0; + /** Never admit a new execution after a pause/drain boundary. */ + private async admit(delivery: MessageContext): Promise { + if (this.lifecycle === 'running') return true; + await delivery.nack({ + requeue: true, + ...(this.lifecycle === 'draining' || this.lifecycle === 'stopped' + ? { preserveDeliveryCount: true } + : {}), + }); + return false; } - /** Wait for active jobs to finish, cancelling them after the shutdown timeout. */ - private async waitForActiveJobs(): Promise { - const timeout = 30000; - const startTime = Date.now(); - - while (this.activeJobs.size > 0) { - if (Date.now() - startTime > timeout) { - console.warn(`[Worker ${this.workerId}] Timeout waiting for jobs, forcing stop`); - for (const [executionId, ctx] of this.activeJobs) { - ctx.abortController.abort(); - await this.executionState.complete(executionId, { - status: 'cancelled', - error: 'Worker shutdown timeout', - }); - } - break; - } - await delay(100); - } + /** Startup continuations may not allocate listeners after stop was requested. */ + private canStartListeners(): boolean { + return this.lifecycle === 'running' || this.lifecycle === 'paused'; } /** Build the dependency context used by job dispatchers. */ @@ -422,6 +434,7 @@ export class Worker { registry: this.registry, queueTriggers: this.queueTriggers, triggerQueues: this.triggerQueues, + canConsume: (delivery) => this.admit(delivery), abortController: this.abortController, taskQueue: this.suppliedTaskQueue, listenerMaxRestarts: this.listenerMaxRestarts, From 26255556c8e38d63c1d559ec0083dfa5233f942f Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Tue, 17 Nov 2026 00:40:33 +0100 Subject: [PATCH 11/20] test(workers): pin schema budgets and real runtime grace forwarding --- .../worklog.md | 7 ++ .../tests/config/worker-budget_test.ts | 28 ++++++ .../tests/runtime/_fixtures/budget-process.ts | 65 ++++++++++++++ .../tests/runtime/runtime-budget_test.ts | 86 +++++++++++++++++++ 4 files changed, 186 insertions(+) create mode 100644 packages/plugin-workers-core/tests/config/worker-budget_test.ts create mode 100644 plugins/workers/tests/runtime/_fixtures/budget-process.ts create mode 100644 plugins/workers/tests/runtime/runtime-budget_test.ts diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index 78faba3196..20f4c7280b 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -71,3 +71,10 @@ Structured full Worker suite plus core cause tests:113pass/0fail/0ignored, inclu Scoped161-file formatter, check and lint wrappers:0findings. Plugin doc-lint combined2 and every entrypoint match measured baseline, including runtime2/worker0/services0. New lifecycle type is an identity export for runtime public closure. release:preflight fourPASS; docs:accuracyPASS. Four exact owned backend containers remain retained for subsequent slices. Configuration/default standalone runner changes and S6 documentation remain pending; no acceptance completion claim yet. S4 quality:gate (quality scan + architecture fitness) exit0; no new doctrine exemption. Worker476lines, focused abort fixture272lines; legacy dispatch test reduced by241lines. + + +## S5 red contract and real runtime oracle + +Before changing configuration/runtime forwarding, committed schema default/validation cases and isolated real-runtime subprocess cases. Executable final fixture uses a real project config module, dynamic user handler, native Deno KV queues, KV registry/execution state and both startWorkerProcess/startCombinedProcess. No static pool registry or mocked runner. An owned HTTP acknowledgement sink satisfies the unrelated execution mutation publisher. Handler requires1.25s post-deadline cleanup; the terminal state must remain timeout even when cleanup returns success. Authored50ms versus1800ms and explicit overrides both directions test actual reachability/precedence. Child processes and their temporary project/KV paths are owned and removed; no container started. + +Final red report:4pass/6fail/0ignored. Both schema tests fail; all four short-grace cases observe cleanup awaited despite requested50ms, proving neither authored nor explicit grace reaches the runtime. Four long-grace controls pass under S4's new Worker-derived default. Initial fixture failures (missing unrelated streams reference and a project-relative dynamic entrypoint mistake) are excluded from product evidence, repaired before this oracle. The original committed core config/default>1s tests still cover the two main regressions. diff --git a/packages/plugin-workers-core/tests/config/worker-budget_test.ts b/packages/plugin-workers-core/tests/config/worker-budget_test.ts new file mode 100644 index 0000000000..ab1a86fba8 --- /dev/null +++ b/packages/plugin-workers-core/tests/config/worker-budget_test.ts @@ -0,0 +1,28 @@ +import { assertEquals, assertThrows } from '@std/assert'; +import { WorkersConfigSchema } from '../../src/config/mod.ts'; + +Deno.test('worker schema derives the same total and grace defaults as the runtime', () => { + for ( + const [input, total, grace] of [[{}, 25_000, 5_000], [ + { drainTimeoutMs: 10_000 }, + 10_000, + 2_000, + ], [{ drainTimeoutMs: 500, abortGracePeriodMs: 0 }, 500, 0]] as const + ) { + const config = WorkersConfigSchema.parse(input); + assertEquals(config && Reflect.get(config, 'drainTimeoutMs'), total); + assertEquals(config && Reflect.get(config, 'abortGracePeriodMs'), grace); + } + assertEquals(WorkersConfigSchema.parse(undefined), undefined); +}); + +Deno.test('worker schema rejects invalid or contradictory budgets', () => { + for (const value of [0, -1, 1.5, Infinity, NaN, 2_147_483_648]) { + assertThrows(() => WorkersConfigSchema.parse({ drainTimeoutMs: value })); + } + for (const value of [-1, 1.5, Infinity, NaN, 101]) { + assertThrows(() => + WorkersConfigSchema.parse({ drainTimeoutMs: 100, abortGracePeriodMs: value }) + ); + } +}); diff --git a/plugins/workers/tests/runtime/_fixtures/budget-process.ts b/plugins/workers/tests/runtime/_fixtures/budget-process.ts new file mode 100644 index 0000000000..d86454c914 --- /dev/null +++ b/plugins/workers/tests/runtime/_fixtures/budget-process.ts @@ -0,0 +1,65 @@ +import { startCombinedProcess, startWorkerProcess } from '../../../bin/runtime.ts'; +import { createWorkersServiceRuntime } from '../../../services/src/service-runtime.ts'; +import { closeKv, getKv } from '@netscript/kv'; +import { createQueue } from '@netscript/queue'; +import type { RegisterJobInput } from '@netscript/plugin-workers-core/runtime'; + +// A real runtime, native queue, state store and dynamically imported user handler. +// The parent owns this isolated executable and its temporary project/KV file. +const [mode, override] = Deno.args; +// Keep the unrelated mutation publisher on an owned HTTP acknowledgement sink. +const streamSink = Deno.serve( + { hostname: '127.0.0.1', port: 0, onListen: () => {} }, + (request) => new Response(null, { status: request.method === 'PUT' ? 201 : 200 }), +); +Deno.env.set('DURABLE_STREAMS_URL', `http://127.0.0.1:${streamSink.addr.port}`); +const budget = override ? { abortGracePeriodMs: Number(override) } : {}; +await getKv({ provider: 'deno-kv', path: Deno.env.get('KV_URL'), skipServiceDiscovery: true }); +const definitions = new Map([['cleanup', { + id: 'cleanup', + name: 'Cleanup', + enabled: true, + topic: 'jobs', + executionType: 'deno', + entrypoint: 'job.ts', + timeout: 80, + maxRetries: 0, +}]]); +const controller = new AbortController(); +const options = { definitions, jobsDir: Deno.cwd(), queueName: 'budget-jobs', concurrency: 1, ...budget }; +const running = mode === 'combined' + ? startCombinedProcess({ ...options, signal: controller.signal }) + : startWorkerProcess(options); +void running.catch((error) => { + console.error(error); + Deno.exit(1); +}); +const queue = createQueue('budget-jobs'); +await queue.enqueue({ jobId: 'cleanup', topic: 'jobs', triggeredBy: 'queue' }); +const state = createWorkersServiceRuntime().executionState; +for (let index = 0; index < 1_000; index++) { + const records = await state.listByJob('cleanup', { limit: 1 }); + const terminal = records.find((record) => + ['timeout', 'failed', 'cancelled', 'completed'].includes(record.status) + ); + if (terminal) { + let cleaned = false; + try { + cleaned = await Deno.readTextFile('cleaned') === 'yes'; + } catch (error) { + if (!(error instanceof Deno.errors.NotFound)) throw error; + } + if (terminal.status === 'failed') console.error('Execution failed:', terminal.error); + console.log(JSON.stringify({ probe: 'worker-budget', cleaned, status: terminal.status })); + if (mode === 'combined') { + controller.abort(); + await running; + } + await queue.stop(); + await closeKv(); + await streamSink.shutdown(); + Deno.exit(0); + } + await new Promise((resolve) => setTimeout(resolve, 10)); +} +throw new Error('Runtime did not settle the timed-out job.'); diff --git a/plugins/workers/tests/runtime/runtime-budget_test.ts b/plugins/workers/tests/runtime/runtime-budget_test.ts new file mode 100644 index 0000000000..ba15574ca1 --- /dev/null +++ b/plugins/workers/tests/runtime/runtime-budget_test.ts @@ -0,0 +1,86 @@ +import { assertEquals } from '@std/assert'; +import { fromFileUrl, join } from '@std/path'; + +for (const mode of ['worker', 'combined']) { + for ( + const [configured, override, expected] of [[50, undefined, false], [1_800, undefined, true], [ + 50, + 1_800, + true, + ], [1_800, 50, false]] as const + ) { + Deno.test(`${mode} runtime forwards authored grace ${configured} and explicit override ${override ?? 'omitted'}`, async () => { + const root = await Deno.makeTempDir({ prefix: 'worker-budget-' }); + let child: Deno.ChildProcess | undefined; + try { + await Deno.writeTextFile( + join(root, 'netscript.config.ts'), + `export default ${ + JSON.stringify({ + name: 'budget-test', + databases: { config: [] }, + workers: { drainTimeoutMs: 4_000, abortGracePeriodMs: configured }, + }) + };`, + ); + await Deno.writeTextFile( + join(root, 'job.ts'), + `export default async ({signal}) => { +await new Promise(resolve => signal.addEventListener('abort', resolve, {once:true})); +await new Promise(resolve => setTimeout(resolve, 1250)); +await Deno.writeTextFile('cleaned', 'yes'); +return {success:true}; +};`, + ); + const config = fromFileUrl(new URL('../../../../deno.json', import.meta.url)); + const fixture = fromFileUrl(new URL('./_fixtures/budget-process.ts', import.meta.url)); + const env: Record = { + ...Deno.env.toObject(), + KV_URL: join(root, 'runtime.sqlite'), + }; + for (const key of Object.keys(env)) { + if (/redis|garnet|rabbitmq|^NETSCRIPT_PROJECT_ROOT$|^OTEL_/i.test(key)) delete env[key]; + } + child = new Deno.Command(Deno.execPath(), { + args: [ + 'run', + '--no-lock', + '--allow-all', + '--unstable-kv', + '--config', + config, + fixture, + mode, + ...(override === undefined ? [] : [String(override)]), + ], + cwd: root, + env, + clearEnv: true, + stdout: 'piped', + stderr: 'piped', + }).spawn(); + const timer = setTimeout(() => { + try { + child?.kill('SIGKILL'); + } catch { /* already exited */ } + }, 15_000); + let result: Deno.CommandOutput; + try { + result = await child.output(); + } finally { + clearTimeout(timer); + } + const output = new TextDecoder().decode(result.stdout); + assertEquals(result.code, 0, new TextDecoder().decode(result.stderr)); + const line = output.split('\n').find((line) => line.startsWith('{"probe":"worker-budget"')); + assertEquals(line && JSON.parse(line), { + probe: 'worker-budget', + cleaned: expected, + status: 'timeout', + }, output + new TextDecoder().decode(result.stderr)); + } finally { + await Deno.remove(root, { recursive: true }); + } + }); + } +} From fefc5e3bf66c1bd4a27809d1be6f52e73110195a Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Tue, 17 Nov 2026 00:43:00 +0100 Subject: [PATCH 12/20] feat(workers): configure cleanup budgets in standard runtime --- .../context-pack.md | 2 +- .../worklog.md | 9 +++++ .../src/config/workers-config.ts | 23 ++++++++++++ .../src/runtime/in-process-job-runner.ts | 9 ++++- plugins/workers/bin/runtime.ts | 24 ++++++++++++ .../workers/src/cli/load-workers-config.ts | 37 +------------------ plugins/workers/src/runtime/workers-config.ts | 35 ++++++++++++++++++ .../tests/runtime/_fixtures/budget-process.ts | 8 +++- plugins/workers/worker/job-runner-pool.ts | 2 +- 9 files changed, 109 insertions(+), 40 deletions(-) create mode 100644 plugins/workers/src/runtime/workers-config.ts diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md index 33ccb76247..4dd9ce978e 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md @@ -8,4 +8,4 @@ Use Deno2.9.7 through mise. rtk/rg absent; plain commands. Scoped structured wra Public artifacts/comments/bodies must contain no operator absolute paths, hostnames/ports, session/thread IDs or downstream names. Keep raw logs private, public evidence repo-relative. Evaluator writes plan-eval.md and phase comment, leaves artifact uncommitted. No source changes, commits, pushes, labels/readiness changes or PR watching by evaluator. -Current checkpoint: original and listener-owner addendum Plan-Gates approved; S2/S3 portable and native handback committed and live-tested on all seven backend variants. S4 lifecycle, pause generations, bounded stop, portable shutdown-only refund and first-cause controls implemented; full Worker113/0/0 and static gates green. S5 standard runtime configuration and standalone default, S6 docs, S7 final validation/supported full smoke and independent IMPL-EVAL remain. No acceptance completion claim yet. +Current checkpoint: original and listener-owner addendum Plan-Gates approved; S2/S3 portable and native handback committed and live-tested on all seven backend variants. S4 lifecycle, pause generations, bounded stop, portable shutdown-only refund and first-cause controls implemented; full Worker113/0/0 and static gates green. S5 standard runtime configuration and standalone default now implemented: real runtime/runner/schema12/12 and full core112/0/1 baseline-ignore green. S6 docs and S7 final validation/supported full smoke/independent IMPL-EVAL remain. No acceptance completion claim yet. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index 20f4c7280b..f15ae0cb4b 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -78,3 +78,12 @@ S4 quality:gate (quality scan + architecture fitness) exit0; no new doctrine exe Before changing configuration/runtime forwarding, committed schema default/validation cases and isolated real-runtime subprocess cases. Executable final fixture uses a real project config module, dynamic user handler, native Deno KV queues, KV registry/execution state and both startWorkerProcess/startCombinedProcess. No static pool registry or mocked runner. An owned HTTP acknowledgement sink satisfies the unrelated execution mutation publisher. Handler requires1.25s post-deadline cleanup; the terminal state must remain timeout even when cleanup returns success. Authored50ms versus1800ms and explicit overrides both directions test actual reachability/precedence. Child processes and their temporary project/KV paths are owned and removed; no container started. Final red report:4pass/6fail/0ignored. Both schema tests fail; all four short-grace cases observe cleanup awaited despite requested50ms, proving neither authored nor explicit grace reaches the runtime. Four long-grace controls pass under S4's new Worker-derived default. Initial fixture failures (missing unrelated streams reference and a project-relative dynamic entrypoint mistake) are excluded from product evidence, repaired before this oracle. The original committed core config/default>1s tests still cover the two main regressions. + + +## S5: configuration and standard runtime forwarding + +WorkersConfigData and authoring/schema contracts expose validated total drain/grace, shared defaults derived from the core policy and grace<=total validation. Standalone InProcessJobRunner uses the same5s default with queue/Worker arithmetic in JSDoc. Runtime policy loading moved from CLI to the runtime folder, retaining an identity CLI export. Both standard startup APIs validate project policy and explicit overrides before runtime dependencies, and pass both durations into Worker regardless of static registry presence. Generated glue already delegates to combined startup, so it uses the same resolver. + +Structured contract+runner+real runtime suite12/12green,0ignored: eight real subprocess cases cover both startup paths, authored50/1800ms and explicit precedence both directions with1.25s cleanup. Every terminal state remains timeout, never false success. Full workers core112pass/0fail/1pre-existingignore. Full core/plugin source/test check0; scoped lint0; canonical source/test fmt174files0. Core combined doclint2/plugin2 and all per-entrypoint distributions match measured baseline, config/shutdown0. release:preflight fourPASS and docs:accuracyPASS. Initial red fixture prerequisite failures excluded as recorded above. + +S5 quality:gate exit0 (no new doctrine exemption or dependency/lock change). diff --git a/packages/plugin-workers-core/src/config/workers-config.ts b/packages/plugin-workers-core/src/config/workers-config.ts index a69c38cef6..71a035a74a 100644 --- a/packages/plugin-workers-core/src/config/workers-config.ts +++ b/packages/plugin-workers-core/src/config/workers-config.ts @@ -1,4 +1,8 @@ import { z } from 'zod'; +import { + DEFAULT_WORKER_DRAIN_TIMEOUT_MS, + resolveWorkerDrainBudget, +} from '../shutdown/worker-lifecycle.ts'; import { type JobConfig, JobConfigZodSchema } from './job-config.ts'; import type { ConfigSchema } from './config-schema.ts'; @@ -45,6 +49,10 @@ export interface WorkersConfigData { readonly queueName: string; /** Default worker concurrency. */ readonly concurrency: number; + /** Total worker stop budget, including abort cleanup and settlement; defaults to 25,000 ms. */ + readonly drainTimeoutMs: number; + /** Cleanup after abort; defaults to min(5,000, floor(drainTimeoutMs/5)) within the total. */ + readonly abortGracePeriodMs: number; /** Legacy flat job definitions. */ readonly jobs: JobConfig[]; /** Topic-scoped worker groups. */ @@ -104,15 +112,30 @@ const WorkersConfigObjectSchema = z.object({ queueProvider: QueueProviderZodSchema, queueName: z.string().default('jobs'), concurrency: z.number().default(2), + drainTimeoutMs: z.number().int().min(1).max(2_147_483_647).default( + DEFAULT_WORKER_DRAIN_TIMEOUT_MS, + ), + abortGracePeriodMs: z.number().int().min(0).max(2_147_483_647).optional(), jobs: z.array(JobConfigZodSchema).default([]), groups: z.array(WorkerGroupZodSchema).default([]), enabled: z.boolean().default(true), +}).superRefine((config, context) => { + if ( + config.abortGracePeriodMs !== undefined && config.abortGracePeriodMs > config.drainTimeoutMs + ) { + context.addIssue({ + code: 'custom', + path: ['abortGracePeriodMs'], + message: 'Abort grace must fit within the total drain budget.', + }); + } }); const WorkersConfigZodSchema: z.ZodType = WorkersConfigObjectSchema .transform( (config: z.output) => ({ ...config, + abortGracePeriodMs: resolveWorkerDrainBudget(config).abortGracePeriodMs, groups: config.groups.map((group: WorkerGroupData) => ({ ...group, jobs: group.jobs.map((job: JobConfig) => ({ diff --git a/packages/plugin-workers-core/src/runtime/in-process-job-runner.ts b/packages/plugin-workers-core/src/runtime/in-process-job-runner.ts index c9cc0ff258..500820d4c2 100644 --- a/packages/plugin-workers-core/src/runtime/in-process-job-runner.ts +++ b/packages/plugin-workers-core/src/runtime/in-process-job-runner.ts @@ -1,3 +1,4 @@ +import { DEFAULT_ABORT_GRACE_PERIOD_MS } from '../shutdown/worker-lifecycle.ts'; import { WebPlatformJobRunnerClock } from '../adapters/web-platform-job-runner-clock.ts'; import type { JobRunnerClock } from '../ports/job-runner-clock.ts'; import { InProcessJobDispatcher, type JobDispatcherOptions } from './job-dispatcher.ts'; @@ -15,7 +16,11 @@ export type InProcessJobRunnerOptions = id?: string; /** Injectable time and disposable scheduling. */ clock?: JobRunnerClock; - /** Cleanup wait after abort; bounds waiting, not physical JavaScript termination. */ + /** + * Cleanup wait after any abort; defaults to 5,000 ms, one fifth of the 25,000 ms + * Worker total (19 s work + 5 s cleanup + 1 s settlement, below the 30 s queue drain). + * Bounds waiting, not physical JavaScript termination. + */ abortGracePeriodMs?: number; }>; @@ -39,7 +44,7 @@ export class InProcessJobRunner implements RuntimeWorkerPort { this.id = options.id ?? 'in-process-job-runner'; this.#dispatcher = new InProcessJobDispatcher(options); this.#clock = options.clock ?? new WebPlatformJobRunnerClock(); - this.#graceMs = options.abortGracePeriodMs ?? 1_000; + this.#graceMs = options.abortGracePeriodMs ?? DEFAULT_ABORT_GRACE_PERIOD_MS; validateDuration(this.#graceMs, 'abortGracePeriodMs'); } diff --git a/plugins/workers/bin/runtime.ts b/plugins/workers/bin/runtime.ts index 22a304c627..c30b309bd7 100644 --- a/plugins/workers/bin/runtime.ts +++ b/plugins/workers/bin/runtime.ts @@ -8,6 +8,11 @@ import '@netscript/kv/redis'; import { createQueue } from '@netscript/queue'; import { deadline } from '@std/async'; +import { + resolveWorkerDrainBudget, + type WorkerDrainBudget, +} from '@netscript/plugin-workers-core/shutdown'; +import { loadWorkersConfig } from '../src/runtime/workers-config.ts'; import type { JobMessage } from '@netscript/plugin-workers-core/runtime'; export type { WorkerLifecycleState } from '@netscript/plugin-workers-core/shutdown'; @@ -72,6 +77,10 @@ async function registerProjectJobs( /** Options for starting only the workers job execution process. */ export type StartWorkerProcessOptions = Readonly<{ + /** Total stop budget, including cleanup and settlement; defaults to 25,000 ms. */ + drainTimeoutMs?: number; + /** Handler abort cleanup, within the total; defaults to min(5,000, floor(total/5)). */ + abortGracePeriodMs?: number; /** Number of jobs the worker may execute concurrently. */ concurrency?: number; /** Project-generated static job definitions to register before processing. */ @@ -109,8 +118,18 @@ export type StartCombinedProcessOptions = signal?: AbortSignal; }>; +/** Validate authored policy and explicit overrides before allocating runtime dependencies. */ +async function loadDrainBudget(options: StartWorkerProcessOptions): Promise { + const config = await loadWorkersConfig(Deno.env.get('NETSCRIPT_PROJECT_ROOT') ?? Deno.cwd()); + return resolveWorkerDrainBudget({ + drainTimeoutMs: options.drainTimeoutMs ?? config?.drainTimeoutMs, + abortGracePeriodMs: options.abortGracePeriodMs ?? config?.abortGracePeriodMs, + }); +} + /** Start the plugin worker process. */ export async function startWorkerProcess(options: StartWorkerProcessOptions = {}): Promise { + const budget = await loadDrainBudget(options); const runtime = await createWorkersServiceRuntime(); runtime.executionState.setMutationHook(createStreamMutationHook()); const generated = await registerProjectJobs(runtime, options.definitions); @@ -126,6 +145,8 @@ export async function startWorkerProcess(options: StartWorkerProcessOptions = {} idempotency: runtime.idempotency, jobsDir: options.jobsDir ?? Deno.env.get('NETSCRIPT_JOBS_DIR') ?? './workers/jobs', workerPoolOptions: poolRegistry ? { registry: poolRegistry } : undefined, + drainTimeoutMs: budget.totalMs, + abortGracePeriodMs: budget.abortGracePeriodMs, }); await worker.start(); return worker; @@ -150,6 +171,7 @@ export async function startSchedulerProcess( export async function startCombinedProcess( options: StartCombinedProcessOptions = {}, ): Promise> { + const budget = await loadDrainBudget(options); const runtime = await createWorkersServiceRuntime(); runtime.executionState.setMutationHook(createStreamMutationHook()); const generated = await registerProjectJobs(runtime, options.definitions); @@ -180,6 +202,8 @@ export async function startCombinedProcess( idempotency: runtime.idempotency, jobsDir: options.jobsDir ?? Deno.env.get('NETSCRIPT_JOBS_DIR') ?? './workers/jobs', workerPoolOptions: poolRegistry ? { registry: poolRegistry } : undefined, + drainTimeoutMs: budget.totalMs, + abortGracePeriodMs: budget.abortGracePeriodMs, }); const runtimeHandle = Object.freeze({ scheduler, worker }); const stop = () => { diff --git a/plugins/workers/src/cli/load-workers-config.ts b/plugins/workers/src/cli/load-workers-config.ts index b88a7c75f3..db7b9af239 100644 --- a/plugins/workers/src/cli/load-workers-config.ts +++ b/plugins/workers/src/cli/load-workers-config.ts @@ -1,35 +1,2 @@ -import { loadConfig } from '@netscript/config'; -import { type WorkersConfigData, WorkersConfigSchema } from '@netscript/plugin-workers-core/config'; - -/** Load and normalize project policy; missing project configuration means no policy. */ -export async function loadWorkersConfig( - projectRoot: string, -): Promise { - let workersSection: unknown; - try { - workersSection = (await loadConfig({ cwd: projectRoot })).workers; - } catch (error) { - if (isMissingProjectConfig(error)) return undefined; - throw new Error( - `Failed to load NetScript configuration from ${projectRoot}: ${errorMessage(error)}`, - { cause: error }, - ); - } - - try { - return WorkersConfigSchema.parse(workersSection); - } catch (error) { - throw new Error( - `Invalid workers configuration in ${projectRoot}: ${errorMessage(error)}`, - { cause: error }, - ); - } -} - -function isMissingProjectConfig(error: unknown): boolean { - return error instanceof Error && error.message.startsWith('No config file found.'); -} - -function errorMessage(error: unknown): string { - return error instanceof Error ? error.message : String(error); -} +/** Compatibility export for CLI callers of the shared runtime policy loader. */ +export { loadWorkersConfig } from '../runtime/workers-config.ts'; diff --git a/plugins/workers/src/runtime/workers-config.ts b/plugins/workers/src/runtime/workers-config.ts new file mode 100644 index 0000000000..b88a7c75f3 --- /dev/null +++ b/plugins/workers/src/runtime/workers-config.ts @@ -0,0 +1,35 @@ +import { loadConfig } from '@netscript/config'; +import { type WorkersConfigData, WorkersConfigSchema } from '@netscript/plugin-workers-core/config'; + +/** Load and normalize project policy; missing project configuration means no policy. */ +export async function loadWorkersConfig( + projectRoot: string, +): Promise { + let workersSection: unknown; + try { + workersSection = (await loadConfig({ cwd: projectRoot })).workers; + } catch (error) { + if (isMissingProjectConfig(error)) return undefined; + throw new Error( + `Failed to load NetScript configuration from ${projectRoot}: ${errorMessage(error)}`, + { cause: error }, + ); + } + + try { + return WorkersConfigSchema.parse(workersSection); + } catch (error) { + throw new Error( + `Invalid workers configuration in ${projectRoot}: ${errorMessage(error)}`, + { cause: error }, + ); + } +} + +function isMissingProjectConfig(error: unknown): boolean { + return error instanceof Error && error.message.startsWith('No config file found.'); +} + +function errorMessage(error: unknown): string { + return error instanceof Error ? error.message : String(error); +} diff --git a/plugins/workers/tests/runtime/_fixtures/budget-process.ts b/plugins/workers/tests/runtime/_fixtures/budget-process.ts index d86454c914..dfce8cab51 100644 --- a/plugins/workers/tests/runtime/_fixtures/budget-process.ts +++ b/plugins/workers/tests/runtime/_fixtures/budget-process.ts @@ -26,7 +26,13 @@ const definitions = new Map([['cleanup', { maxRetries: 0, }]]); const controller = new AbortController(); -const options = { definitions, jobsDir: Deno.cwd(), queueName: 'budget-jobs', concurrency: 1, ...budget }; +const options = { + definitions, + jobsDir: Deno.cwd(), + queueName: 'budget-jobs', + concurrency: 1, + ...budget, +}; const running = mode === 'combined' ? startCombinedProcess({ ...options, signal: controller.signal }) : startWorkerProcess(options); diff --git a/plugins/workers/worker/job-runner-pool.ts b/plugins/workers/worker/job-runner-pool.ts index 8782031f4b..434e893e95 100644 --- a/plugins/workers/worker/job-runner-pool.ts +++ b/plugins/workers/worker/job-runner-pool.ts @@ -14,7 +14,7 @@ export type WorkerPoolOptions = Readonly<{ registry?: StaticJobRegistry; /** @deprecated Reserved for a future isolate pool; the in-process runner does not load this URL. */ workerUrl?: string; - /** Cleanup wait after abort; does not physically terminate an in-process handler. */ + /** Cleanup wait after abort; defaults to 5,000 ms within the Worker total stop budget. */ abortGracePeriodMs?: number; }>; From 667ffa8b1fff83a96ff98b720e2adb0e2b89ad99 Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Tue, 17 Nov 2026 00:45:37 +0100 Subject: [PATCH 13/20] docs(workers): describe pause drain budgets and shutdown handback --- .../worklog.md | 5 ++ .../how-to/tune-worker-runtime.md | 79 +++++++++++++++---- packages/plugin-workers-core/README.md | 17 ++-- packages/queue/README.md | 36 +++++++-- plugins/workers/README.md | 66 +++++++++++++--- 5 files changed, 163 insertions(+), 40 deletions(-) diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index f15ae0cb4b..3c71a746dc 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -87,3 +87,8 @@ WorkersConfigData and authoring/schema contracts expose validated total drain/gr Structured contract+runner+real runtime suite12/12green,0ignored: eight real subprocess cases cover both startup paths, authored50/1800ms and explicit precedence both directions with1.25s cleanup. Every terminal state remains timeout, never false success. Full workers core112pass/0fail/1pre-existingignore. Full core/plugin source/test check0; scoped lint0; canonical source/test fmt174files0. Core combined doclint2/plugin2 and all per-entrypoint distributions match measured baseline, config/shutdown0. release:preflight fourPASS and docs:accuracyPASS. Initial red fixture prerequisite failures excluded as recorded above. S5 quality:gate exit0 (no new doctrine exemption or dependency/lock change). + + +## S6: public semantics and options + +Updated the actual tune-worker-runtime.md guide and worker/core/queue READMEs. Documents pause/resume generation joins, healthy paused lifetime, terminal bounded stop,25s=19+5+1 arithmetic and duration validation/precedence, project/generated startup forwarding and standalone5s grace, parent headroom, shutdown-first-cause delivery preservation versus ordinary failures/pause races, at-least-once provider outage limitations and the distinction between listener post-intake wait and Worker total. Public docs contain product concepts only. Preserved unrelated Markdown formatting rather than expanding scope. docs:accuracyPASS. Manual docs committed before refreshing generated prose/export/asset consumers. diff --git a/docs/site/background-processing/how-to/tune-worker-runtime.md b/docs/site/background-processing/how-to/tune-worker-runtime.md index 682eb0a32b..2dabaac6d4 100644 --- a/docs/site/background-processing/how-to/tune-worker-runtime.md +++ b/docs/site/background-processing/how-to/tune-worker-runtime.md @@ -53,6 +53,8 @@ I/O-bound work, then measure throughput and memory under load. caption: "WorkersConfigData — top-level worker config (@netscript/plugin-workers-core/config)", rows: [ { name: "concurrency", type: "number", desc: "Default queue concurrency for in-process jobs. Schema default 2; no isolate pool is allocated." }, + { name: "drainTimeoutMs", type: "number", desc: "Whole Worker stop budget, including abort cleanup and settlement. Integer 1–2147483647 ms; default 25000 ms." }, + { name: "abortGracePeriodMs", type: "number", desc: "Handler cleanup after abort, inside the total stop budget. Integer 0–drainTimeoutMs; default min(5000, floor(drainTimeoutMs / 5)), normally 5000 ms." }, { name: "queueProvider", type: "'auto' | 'deno-kv' | 'redis' | 'postgres' | 'amqp'", desc: "Queue backend. 'auto' resolves one for you. Default 'auto'." }, { name: "queueName", type: "string", desc: "Queue the runner consumes from. Default 'jobs'." }, { name: "jobsDir / tasksDir", type: "string", desc: "Directories scanned for default-exported job and task modules. Defaults ./workers/jobs and ./workers/tasks." }, @@ -422,23 +424,70 @@ the consumer extracts it, preserving exact admission-to-pickup ancestry across c Use those trace/span parents to measure queue wait; native filesystem propagation before admission needs its own upstream correlation and cannot be inferred from nearby timestamps. +## Pause intake and drain work + +Call `worker.pause()` on an embedded `Worker` to stop new jobs, tasks and queue-trigger intake +without aborting admitted work or stopping its lease renewal. `worker.start()` stays pending, and +`worker.isPaused` and `worker.healthStatus.lifecycle` expose the intentional healthy pause. +`worker.resume()` waits for the previous listener generation to finish before claiming again, +including when resume races a pause. Repeated calls are harmless. A claim committed just before +pause may be returned without running the handler; this ordinary requeue still counts. + +`await worker.stop()` is terminal and idempotent for that instance. It pauses intake, allows work +to finish and acknowledge, aborts leftovers, then awaits cleanup and queue settlement within one +monotonic total budget. A replacement uses a new Worker. Pause before start or after stop and +resume during drain or after stop are harmless no-ops. + +Set budgets in the project's workers configuration so generated and packaged startup use them: + +```ts +import { defineWorkers } from '@netscript/plugin-workers-core/config'; + +export const workers = defineWorkers({ + drainTimeoutMs: 25_000, + abortGracePeriodMs: 5_000, +}); +``` + +The default total is **25 seconds: 19 seconds for work, 5 seconds for abort cleanup and 1 second +for settlement**. Grace defaults to `min(5_000, floor(total / 5))`; settlement reserves +`min(1_000, max(1, floor(total / 25)))` milliseconds, leaving the remainder for work. Total must be +an integer from 1 to 2,147,483,647 ms, and grace an integer from zero to total. A hung handler, +executor, storage call or queue close makes stop reject `TimeoutError` within the total bound. + +Both `startWorkerProcess` and `startCombinedProcess` load and validate the authored workers +section before opening runtime dependencies. Explicit duration options override project config, +then defaults apply. Direct `new Worker(...)` takes the same durations; its top-level grace wins +over the legacy `workerPoolOptions.abortGracePeriodMs`. Keep the parent process drain budget at +least five seconds longer than the Worker total. Raising this total also requires raising the +parent budget, including a custom `runChildHealthProcess` host. + +Redis and KV polling also expose `ListenOptions.drainTimeoutMs`, a post-intake callback wait that +defaults to visibility timeout. It accepts a finite positive integer or `Infinity` under an outer +deadline owner. Worker passes `Infinity` to keep pause and settlement barriers alive, and enforces +its own finite total. The two same-named options have different scopes. + +Shutdown handback keeps the live delivery count unchanged on every counting queue adapter; +repeated clean restarts therefore cannot exhaust `maxDeliveries` for a healthy job. An ordinary +handler failure still consumes an attempt. Only a first cause of controlled shutdown gets the +preserved count: an earlier business failure, deadline, operator cancellation or lease loss cannot +be converted into a free retry by a later stop. Delivery remains at least once. A crash or uncertain +persistence/acknowledgement can replay effects; retain idempotent external operations. + ## Abort cleanup budget -`InProcessJobRunner` accepts `abortGracePeriodMs` in its constructor options, with a default of -**1,000 ms**. After a timeout, a shutdown or a caller cancellation, the runner aborts the handler -signal immediately and waits up to this budget for handler and progress cleanup. If cleanup does not -settle, dispatch rejects with the abort reason. Returning success after an abort still rejects. - -The budget bounds waiting; it cannot stop in-process JavaScript. A handler that ignores its signal -can keep causing side effects after dispatch rejects, and a blocking synchronous loop prevents -timers from firing. Use abort-aware I/O, yielding checkpoints and `finally` cleanup. The setting is -not a forced termination delay for a worker or subprocess, and it is not a top-level -`defineWorkers` field. - -The shutdown manager's overall timeout is a separate host drain budget; size both within your -platform's shutdown window. See -[JobContext cancellation](/reference/plugin-workers-core/#job-context-cancellation) for `deadlineAt` -selection and reason names. +`InProcessJobRunner` accepts `abortGracePeriodMs`, default **5,000 ms**, derived from the Worker +budget above and below the default 30-second queue listener drain. An individual timeout or caller +cancellation aborts the handler immediately; Worker shutdown first allows its work window. Once +aborted, the runner waits up to grace for handler and progress cleanup, then rejects with the +original abort reason. Returning success after abort still rejects. Standalone runner stop aborts +immediately and uses that same grace. + +Every dispatched job handler receives `context.signal` and, when configured, `context.deadlineAt` +in epoch milliseconds. The effective deadline is the earlier of the dispatch deadline and dispatch +time plus `job.timeout`. Forward the signal to provider requests and check it at work boundaries. +A handler that ignores cancellation can continue after the wait ends; synchronous CPU work also +delays timers. Use a process boundary when physical termination is required. ## Production notes diff --git a/packages/plugin-workers-core/README.md b/packages/plugin-workers-core/README.md index 644d1bb8ea..75427fa2f8 100644 --- a/packages/plugin-workers-core/README.md +++ b/packages/plugin-workers-core/README.md @@ -61,11 +61,18 @@ expired deadline prevents handler invocation. Timeout and grace values must be f nonnegative; an explicit deadline must be finite epoch milliseconds. `InProcessJobRunner.stop()` stops admission, aborts active handlers immediately and waits for -cleanup. `abortGracePeriodMs` defaults to 1000 milliseconds. After that grace the runtime stops -waiting and rejects the dispatch; a handler cannot turn cancellation into successful completion. -This runner executes JavaScript in process, so it cannot physically terminate a handler that ignores -the signal, and CPU-blocking code can delay timers. Deadlines, grace timers and parent listeners are -disposed when dispatch settles. Late calls to `reportProgress` reject after abort. +cleanup. `abortGracePeriodMs` defaults to **5,000 milliseconds**, derived from the plugin Worker's +25-second total stop budget (19 seconds work, 5 seconds abort cleanup, 1 second settlement), below +the queue's default 30-second listener drain. Standalone runner stop aborts immediately; the plugin +Worker first stops intake and waits for admitted work. Project workers configuration exposes +`drainTimeoutMs` and `abortGracePeriodMs`, and standard/generated startup forwards them to the +actual runner. An explicit startup override wins over project configuration, then defaults apply. +The core `./shutdown` subpath owns the shared budget resolver and monotonic drain contract. After +that grace the runtime stops waiting and rejects the dispatch; a handler cannot turn cancellation +into successful completion. This runner executes JavaScript in process, so it cannot physically +terminate a handler that ignores the signal, and CPU-blocking code can delay timers. Deadlines, +grace timers and parent listeners are disposed when dispatch settles. Late calls to `reportProgress` +reject after abort. Once a release contains this executor-driven signal and deadline, EIS's `createJobAbortScope` deadline/parent-signal workaround can be removed for job handlers. Source qualification alone does diff --git a/packages/queue/README.md b/packages/queue/README.md index 1e6aa754b7..fedb451ddd 100644 --- a/packages/queue/README.md +++ b/packages/queue/README.md @@ -92,6 +92,24 @@ stop consumption or lease recovery. External stores must honor the port's idempo contract. Use the built-in Redis store as the default rather than as an external mirror, since its append operation does not deduplicate repeated writes. Health exposes pending outbox records. +### Controlled shutdown handback + +`context.nack({ requeue: true, preserveDeliveryCount: true })` returns the current live delivery +without consuming its attempt. The option defaults to false and is ignored for `requeue: false`. +Redis/Garnet, Postgres, KV polling, native Deno KV, AMQP and memory preserve message identity, +payload and headers; the next delivery sees the same count. This explicit handback bypasses the +failure limit/backoff for that delivery only. Ordinary nacks, thrown handler failures and expired +receipt recovery retain their counting policy. First-party Workers use preservation only for +controlled shutdown; ordinary pause races and failures remain charged. + +Receipt-based adapters fence stale/duplicate settlements. KV polling uses atomic checks when the +provided KV supports them; without atomic support its existing fallback is single-consumer, +best-effort publish-before-delete. Native Deno KV and AMQP republish the addressed envelope before +acknowledging its old delivery; AMQP waits for broker confirmation. If that persistence fails, the +original delivery stays on its native recovery path. Crash/outage recovery before the new metadata +is committed follows the provider's native retry behavior; it cannot promise exactly-once effects or +exact attempt metadata across an uncertain outcome. Keep handlers idempotent. + ### Upgrading existing Redis and Garnet queues Existing ready lists, delayed entries, and dead-letter records retain their format. Recovery also @@ -208,13 +226,17 @@ See [Tune the worker runtime](https://rickylabs.github.io/netscript/background-p Abort stops admission. A claim already committed still reaches its handler; in-flight handlers and fenced acknowledgements drain for up to `visibilityTimeout` (default 30000 ms), starting at abort, -stop or listener failure. `stop()` awaits that same drain before disconnecting. Claims left when -the budget expires remain durable for reclaim, and the old contexts cannot acknowledge or renew. -Calling and awaiting `stop()` inside its own handler consumes the drain budget and leaves that -handler's claim for recovery. Signal cancellation to the host instead when the handler should -finish and acknowledge normally. Handlers must respect their application's cancellation signal; -the drain cannot terminate JavaScript or undo effects. First-party workers retain their existing -per-delivery renewal and cancellation policy. +stop or listener failure. Redis and KV polling accept `ListenOptions.drainTimeoutMs` to override +this post-intake callback wait: a positive integer through 2,147,483,647 ms or `Infinity` when an +outer owner enforces a deadline. Omission keeps the visibility-derived default. This listener option +differs from the same-named Worker option, which bounds its entire stop. Worker uses `Infinity` for +listeners so pause retains live receipts and its finite Worker budget owns stop. `stop()` awaits +that same drain before disconnecting. Claims left when the budget expires remain durable for +reclaim, and the old contexts cannot acknowledge or renew. Calling and awaiting `stop()` inside its +own handler consumes the drain budget and leaves that handler's claim for recovery. Signal +cancellation to the host instead when the handler should finish and acknowledge normally. Handlers +must respect their application's cancellation signal; the drain cannot terminate JavaScript or undo +effects. First-party workers retain their existing per-delivery renewal and cancellation policy. ## Compatibility diff --git a/plugins/workers/README.md b/plugins/workers/README.md index c0cd2fbcc0..4949d9b48e 100644 --- a/plugins/workers/README.md +++ b/plugins/workers/README.md @@ -101,19 +101,59 @@ flowchart LR W --> S["Durable streams
executions · jobs"] ``` -## Job cancellation - -Job handlers receive an executor-owned `signal` and optional epoch-millisecond `deadlineAt` through -the Deno job path. `Worker.cancel(executionId)` requests local cancellation of an active execution; -it returns false for an absent or already-cancelled execution. `Worker.stop()` aborts jobs at drain -start and waits for listener and runner cleanup. Timeout records `timeout`; shutdown and explicit -cancel record `cancelled`, release the delivery claim and never record a false successful result. - -`workerPoolOptions.abortGracePeriodMs` defaults to 1000 milliseconds and bounds the wait for handler -cleanup. An in-process handler that ignores cancellation may continue running; CPU-blocking work can -delay timers. Forward `context.signal` to cancellable operations. Task-executor cancellation is a -separate path. Optional `queue` and `taskQueue` dependencies use the existing `MessageQueue` ports; -this worker owns and stops supplied queues as it does the defaults. +## Pause, drain and cancellation + +`worker.pause()` stops intake for jobs, tasks and queue triggers while admitted work keeps running +and renewing its leases. `worker.resume()` resumes intake after that listener generation has joined +its admitted work, so overlapping calls cannot exceed concurrency. Pause keeps `worker.start()` +pending and reports a healthy `paused` lifecycle through `worker.healthStatus.lifecycle` and +`worker.isPaused`. Repeated pause/resume calls are harmless. A claim already committed when pause +arrives can be returned without executing its handler; that ordinary requeue still counts. + +`await worker.stop()` stops intake and gives admitted work time to finish normally and acknowledge. +It then aborts leftovers, waits for cleanup, and closes its queues within one total `drainTimeoutMs` +budget. The default is **25,000 ms: 19 s work, 5 s abort cleanup, 1 s settlement**. +`abortGracePeriodMs` defaults to `min(5_000, Math.floor(drainTimeoutMs / 5))`; shorter totals also +reserve `min(1_000, max(1, floor(total / 25)))` milliseconds for settlement. Total must be an +integer from 1 to 2,147,483,647; grace must be an integer from zero to total. A hung handler, +storage call or queue close can make stop reject `TimeoutError` within this total. Stop is +idempotent and terminal for that Worker instance; create a new Worker to start again. + +Set both durations in the project's workers section, including generated workers: + +```ts +import { defineWorkers } from '@netscript/plugin-workers-core/config'; + +export const workers = defineWorkers({ + drainTimeoutMs: 25_000, + abortGracePeriodMs: 5_000, +}); +``` + +`startWorkerProcess` and `startCombinedProcess` accept explicit duration overrides, otherwise use +validated project workers configuration, otherwise the same defaults. Direct `new Worker(...)` +accepts the same options. Its top-level grace overrides the legacy +`workerPoolOptions.abortGracePeriodMs`; pool-only customization remains supported. Keep the parent +process drain at least five seconds longer than the Worker total; raise the parent budget too when +raising this total. Redis/KV-polling listener `drainTimeoutMs` has a different scope: it bounds +callback waiting after intake stops. Worker gives those listeners an infinite wait and owns their +finite total itself, keeping receipts valid through pause and handback. + +Jobs returned because their **first cancellation cause is controlled shutdown** retain their current +delivery count. Repeated clean restarts therefore do not exhaust `maxDeliveries` for healthy jobs. +Ordinary handler failure requeues still count. An earlier failure, deadline, operator cancellation +or lease loss cannot become a free retry merely because shutdown happens later. Delivery remains at +least once: uncertain acknowledgement or persistence can replay an effect, so use idempotent effects +and stable message/domain keys. + +Deno handlers receive an executor-owned `signal` and optional epoch-millisecond `deadlineAt`. +`Worker.cancel(executionId)` immediately requests cancellation of a local active execution and +returns false for absent or already-cancelled work. Timeout records `timeout`; shutdown and explicit +cancel record `cancelled`, release the claim and cannot record success after abort. The standalone +`InProcessJobRunner` also defaults to five seconds of abort cleanup. Cancellation is cooperative; +in-process JavaScript and CPU-blocking work cannot be physically terminated by the runner. Forward +`context.signal` to cancellable operations. This Worker owns and stops supplied `queue` and +`taskQueue` ports as it does defaults. ## Job retries and dead letters From 764271ba630dfd0fe6cd03e98b9ba315c83cb854 Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Tue, 17 Nov 2026 00:49:49 +0100 Subject: [PATCH 14/20] chore(workers): refresh published docs and API catalog --- .../worklog.md | 3 +++ .../docs/agent-docs-prose.generated.ts | 2 +- .../export-surface-corpus.generated.ts | 14 +++++++------- packages/mcp/src/publish-assets.generated.ts | 4 ++-- 4 files changed, 13 insertions(+), 10 deletions(-) diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index 3c71a746dc..008e851c59 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -92,3 +92,6 @@ S5 quality:gate exit0 (no new doctrine exemption or dependency/lock change). ## S6: public semantics and options Updated the actual tune-worker-runtime.md guide and worker/core/queue READMEs. Documents pause/resume generation joins, healthy paused lifetime, terminal bounded stop,25s=19+5+1 arithmetic and duration validation/precedence, project/generated startup forwarding and standalone5s grace, parent headroom, shutdown-first-cause delivery preservation versus ordinary failures/pause races, at-least-once provider outage limitations and the distinction between listener post-intake wait and Worker total. Public docs contain product concepts only. Preserved unrelated Markdown formatting rather than expanding scope. docs:accuracyPASS. Manual docs committed before refreshing generated prose/export/asset consumers. + + +S6 canonical consumer refresh: gen:agent-docs-prose (site build/semantic checks included), gen:mcp-export-corpus with expected dirty prose carrier, gen:assets-barrel and gen:publish-assets all exit0. Only generated prose, MCP export corpus and MCP publish-assets carrier changed; CLI asset generation produced no diff. check:mcp-export-corpus and check:publish-assets exit0. No lock change or saga transport edits. A final S4 coverage supplement exercises actual Worker replacement on the five remaining adapters, complementing the original Redis/Garnet Worker oracle and seven-backend queue contract; it is a test-only coverage extension, not a design change. diff --git a/packages/mcp/src/infrastructure/docs/agent-docs-prose.generated.ts b/packages/mcp/src/infrastructure/docs/agent-docs-prose.generated.ts index c1796f10d8..9ac3ebffd0 100644 --- a/packages/mcp/src/infrastructure/docs/agent-docs-prose.generated.ts +++ b/packages/mcp/src/infrastructure/docs/agent-docs-prose.generated.ts @@ -35,7 +35,7 @@ export const EMBEDDED_AGENT_DOCS_PAGES: readonly AgentDocsPage[] = [ ["pages/background-processing/how-to/run-a-polyglot-task/index.md","H4sIAAAAAAAA/81Z/27bRhL+n08xlXNHSRUpxykuPfkc1E1cNG1qG7EPxcENsityJG5N7bK7S9k6w0Af4p7hHqxPcphZkqLcOChw7eHyTyxyd2f2m1/fDPfgba1BQmXKzbI0Hrx011F0WSgHFjNVIfjaagcSzje+MBqMBVdgWYLLrKo8KO0NSDhFfxEe0AkTsLV2oDz4wpp6WYAvEG6MvUbr4OWb13CjfAHKOzC1r2oPmax8bTGfgNQ5OKnzublFkvsKteFDwx68rUqVKQ8V2pVyThnt0iiSZVXIKPrKWBbF61cmx3LCv5UmIXR0K89ob2UWntGKRV2WIC6lu36Lri69AFfICifgEOHqvI+PezcsvK/cbDq1KrvelHLu0qXyRT1PlZlq9AGb6Vxm10trap0nlTUZOqf0ctpinfBZ01EaRXt7cG7R4k+1csqji6IExBfbg6qyXiqdNAAmmbEoYMh3Hk3gplBZAZU1a5WjA5HjQmmkqwgYinQ6r1WZo3VixLcVmUXp8RUuZF16WnZyi1ntjQ3Lsf01SqMELhk9j7ay6LGHrqvkjXYTUNp5WZaYg9E9M8N4XBjnx+MZiIpd55mAobEgYY16PSFHEtVG0K7vlc7NjRtNQMylK8QERHXjCjEVlblBy/4mgiHT05NLuHj1Le/fmNpCZZEu6GGutLQb0vlYA2pvN5VR2reOaiyEm8l5SWatswKkA5FOwwI3dYRrWm1Ee2+hXYu5IK+dQI60eI45zLE0N2kU7axR5LBarpBUg6Va447jBzPGHAGH/MZlcrEwZQ65QQfaeAjGAeVTeB2QpTAyOsNZFAnBAEU5xUQDPCTLY7uARMNWE/jR2dmj/vPFfrqf/nWalYoOjKLvlS9MTbGK7aEcwIAyKyAzqxW5DWHFcm8hOf6tAuBva7Rz+OXnf78Qwc8vPFbwFH75+V/wiv2086go6nnuUOUjARbb7NP4MNwUxiHkwXdB2Fp7tUJGXsSkXiwOISNYRNq8HPpNhSMBC3I+bXyBFkqpl7VcYgrn0rk2PTiQdlmvUHsXko1IpV26YZqmI8GxIx2gXitrNK2CtbSK3Klbjno9vEvT9H4kUhApaz0cCZhjZlboWDPawAZlJ3HoeV/rrsNK+mJEYAkhvIum09Z9piFfsJcmc+mzIvUuUqvKWA93sMUO7mFhzQrij2eQLi/EhxGJCVEKDWqTNm62qk22yZcBc2mEtyw9M9pRoBmLX5JicNRTZxj3VI5HEUBnmTjIDA97GMS/Dsp2zXp4B9+dvTp58/78+PLrGcTplPN8s9Km1XUZwz2vZuPFSeILi64wZR5PIN5PPw9nkQam9sOnB/vv9/f3RzCdwsodts7lwBt4ts/vaHljy4DVBSe/WYPZM0hq+LXK0JMM++nnIdoeMas1XnpMSrN0v69ZQ7VuEA91mSKulyBrTZFFiQWG2kCSjR4YNuj2xizdA8P2lH5gWJb6Mbv27+sat2jtdY1Ykame7xrqWbDTo8bgGzwmAcKx8PyjVtgNLs5SlcUFUtmqlNaYc/UCs0YLT8gDIVcuo5+b38tm/+OYmk7hLTpT1l4xvcvRtn790uiFWqbhx7n0BSQv+P7t36cnlxcv374+v3x//o/Lr89OOSbpRRMX02pDWqzQy1x6ObzbOXgGd91pHMgp/YrhvgnfzshssbZ2HLBVvrfKh9IRLkWctf27KxvO2zpjWgkkHuYbqKzSXukljMdGI3xzcXYKZv4jEhMMgVFK56GkwmQWIJzPTe3FeJzCCRkZmvrB73X3ntmM8zlay7WopbNcU6A0yxTOSTLkSi61cV5lnF+6PVRccMWEudHKMg1ldSahvFDtItXlgoiY8qFGBESjPXho3dYff3RGT8C4CbiNi6I9eB2qnbVqTQybq94aPqXKBoo4nHKAt3JVlZhG2wx2BIvSSD90G0eRur5q/0iVzvF2N9GO4FN4+m4UcW5+TzUNjsC4tKmeV/E2h8fvSKk0TSE3HV2i3xHfhATfxddY+XgGnx1MIM6tqSrM4xk8m0C8lTmD7u97OvJVD+ulIbgD2oF6NfiqYPU3xxeX9Jp4EBuXDLKqnYc5giR0lV6WjWnOvvzm5OUlJUvqHwhJuRnR+ZWp6pLoWzg8bf6L2OtCzJIBM0PYeownsFAlHhGQQbVRs5Rslub1qnLDAMJo1OUtZoB7n0xrZ6dzpadktvAMPpz6IuIYCdYGKlXhQqoyiigXHg2e3B3Mkuf3gwb/sI2hx6wwMAgPcnJgNwHaQ+o/uaO/7gfw4s8HALC349TJiwZlssAbiqU+qkc7EdfAA6dGJ7vsg3wSmQ8ndZxPwJloDxZl7QoKYgoUDuIuWJrYXSgtS7jh3PBIyDC+C4jv2ssNZuBtjRMYkI8NZvAnd/+DjmHwhG45CLC/gKtPLl+fv4NLAxUxxsqqNfN0jiWSRXes6qAXM6QemWy60h51nEDtEASZXemGYFIOMRWlYXYHpUUKVxeo8wfSjCZMlf5d29A9h/yeBSUsKDE6YUEj4AoXMHbqnwilWikfOjK01ljX9dE/1VhjkitXcdGydYnUn7fZ+xlnbxo6dI0kG+rBfGCnryKnYC2pIXCmpKxF65eo0bJ/hna/qTKsSo4lLqWnlSGpiO/q0qu3wb12m15fSN/vXFuuVDt0TYOwNSUn6l4LIB17YLotBdxjt3ndeYtyRS162ZSret4YgbkYYxjAoLEIe2e4XZskdoYSIeW7IvoAOtCjBPBDBJAkRKiO4qtBLzMPJoP99PPBu7hZgnp9FN8Ntul4MBt8gFIP7tsNDRs7enqwv7+/3zykfBUihSYwLYA0N1hayR7NsGS182YFMpeVR+smTaf26ECCWqdcWcx8uek50WfsRBdhSNQfEQUWwG2q2GoR8DS63ADV/O2oqhszzUCkvXlS28J1bZtIElmW5iYZC1iUcunaWUfoiG2tRdcnU557vHurpHWYLBDzll/KTg3y2nbWNSyRcifFoypxiaM/hF2yNl8h19g+udxq+YBbcofNnRLdvumUDsEV5kZDgRa5y85KaZXfPE5Ad1AIInbgjwAANPoZXMWyUmnLRjKzit9NSHzyAlqbaPRHDxbxfosypwPSKQlytLH5t7uf1h01i3gj148ZLGTpcMJPUK/ppK9OTl69vzz79uS0O2z3KAqm7aIIHhLZfrfSes7H7vFBHR+RBo/Auy1gL4//fvn67PQdHLchwq4WpmdtCdsNhFCYuGkME45WtixL0eZG3c0FJsBum3Ruy+MNcolae1s7ytVcC5XNk0pav+mNdzOTYxrmbj0VBDVgDr2DYf9xSoic6XJDw8JUoyeHD38bjWJCI6cRkwhauElWMkdwXlquz+yQ7jDMdt+i81Zlvk3/rExP1H9XZgtzk3gztY2QJjA5YSc9Ie0Y+NKamgZKhTGkKQ2Cx2OxQyfDsEvXZSlg3oztQv6vswwxx3w2HoMzKwzcZ2HIYlQlW7bU0K9e/8Lj1N3uR4V5pNxhvmFnCheITTdGs9Lx+ORWUU7JEcTTg+eByjw9+IsgTfyDCbJywLfWSxiO26xJohaE3XhEm+nndlr7YN32xXiUwpfGFzTPy4lytUgxKRFbarFQ1vltVQ4Mv6pQ55gfguwU6utJRNlt0bWSOz9fSEr91twovQyXv/DS1w5EUxkFzck6OJKnHQh8DN4GG/HHj27L7kGZ1BnSSJ0q0MJY6kaodQ20gb3YtYdyBbWYoVpTu6lBlsHj5dxYijfh1FLLshFxDCJb5U3khxua2juVYzuFp4PDGuXaZwkVzzY2tyP5rsSGbwG777Zz/cCIeKIPWDq8oULRfvUweZ3x5EEb/vCxk6jGY2oNOEX8qjHoqmY6HjcfpybNsKtHNba+wkSp5xH0vefsYlthHfX7PRZIXy/C16gPpcRCUngALhZNJPkCV4c9blGiJJ7aGS1oy01LIKSmIvJKD9zGeVyR29NXKak0WqhK6RfGrlI4V/2vGcxn12jVQmHe9nuhmVDOUO+Zw1pZX8uyT1Pd9nvHzminHb+I5rsbZrUNPJWuv6tSY4FguFO8pc4OKxdFb0x2DTkxAOQRyTal84C7UFWA/f8o0yZ/7Ce92YOPkF139Oh3yJRUOs5zkE22aSKrocq/CzIyzxMZ8GiOT5rjSWOKDtnJ3Tpy92WKDJlGSRRddZ0HTUZRZ/hb1es2TFtWPIr+A+hmeOGDHgAA","7130db821f40f2c12d27973b20f2084e08d854d5d64ebe0803792ea4cbe7a3ac",7811], - ["pages/background-processing/how-to/tune-worker-runtime/index.md","H4sIAAAAAAAA/+193XIbybHmPZ4iV4owQRhoUpLHZ016Jg5H4rE5ox8ekpoJx6xCXeguADVqdLWrqklhRorw1XmA3X2GfYVzfx7FT7LxZVZ1N0BoLO96N/ZidTGWGt31k5W/X2aWH9JNW2sKK0131r3TjlxbB7PWo9HNynhyujCNptDW2tOfW91qKmxdtM7puthMqdFuFmxjCvKFqky9nNKPdk5OB7chVZdUalXOKh2CdjTXK3VrrJtSUP4dYRrbBs/vNdqtjffG1n7KD7AkNbcuUFFpVbcNzdtyqUNGz3RRKadJq2KF2Q58nK+xlSk2NNfelJpM8LRSdVlpR95SaXxhb3VcFkZf6lo7FXRJTi+NxwiFcm7DP3q11mlAUwdL+r0u2mBsnY1G39i5jw80mXrWOFto78nU/O1cFe+WzrZ1mYjqtbs1hc7oaU88wmoq1XhSflMXK2dr23q6OHp1SiZQabWn2gYqnFZB0/d6Tt/LaMbbSgXtyTpqlFNVpSvzk6bhME8vX/PkGb0KKznWWjta21J7AvWcxqJ0eUq2rjaU99vIyXgy66bSa12DPGOvNf2wzR8y0pvxKoTGnxwdOVO821Rq7rOlCat2nhl7VOvgC2eacNRTJE1i6uWRjOiPHspfZnHoGQ89k4dv40N/eJjRDQ5Gh2DqpafK3Op01uC5eNZ+imPIC1svzPLILhamMKqaNVW7NLU/WtsyCz6ncV7qham1kNSPsyw7zA/5W+ZpMGjaqow1BblNTaomXd8aZ2uQh26VM2peQYZUGAqSroPbNNbUgZxWpc/oX6zrOFK1YWUdCwxT9+ueZ360838QYQ+z0ejhQ7p02uk/t8aboP1oNGM6xndICION+wBOKunOhBXl3RzxjfQC/XM/u/wST8rnssPKFqoCzYIz8zZYB1lqKu2n1HpNealrC9LS7IwaVbxTS+2PisoczU3d72xW6tss+N3JI21nsxrymfYwm8UpcuGR9Pzs8iKJ3pRszSpBeW+WtS6psS5MSdes1fwpKfIa0hS2BDiJdpR2n41mf5O5pjvqZR+r0byyxTu6W2m3o1JzXtCls7em1C5P2nFbzTL381o+eRyzwjp9JEvNp3S3MsWK9Hts2+8sKZ+mB9BsMiW2kMffn6ZB8muZPj7gF/Mb5d+lB2HTCI2YEyChroVGP/ONcZrGeVGSkr//6lfpbz4oF/Io3wPaR51Va116urQ+LB2UV13St9+JkYjDmvpHXQS/LX33RbRb2HXQDT2RY8rrpGp8Tk+fX2R0EaAAFTGLbWxLS+yjHzsx5YHH+6diLgq1WNiqvKe2TcjoIjKvCWTrQp+MRnmez5VfjVgWEm/PlmduQbOa+hXRj96dfPKE//k4O85+B9nBgKPR9yasbCtKKA46ZUljS1nY9Ro0U3z6taX3kMDPnIB+f6vdnP76l//xVS5KhWn4iP76l/9O11rmLPVCtVUYsvNolA/+lUMIh4QU7qRSLPoeD4PGvljpterGzh+DUf4YdT35FWzZXqN74JOhnML+g9DyTz7dCmoKwtnAPFc2ZHSljN+efGEd7PFs3g3LPFPTWivf8sTOtstV0wZmx7VeW7ehti419KAqs9FoMtkSomcqKCZasM2s0re602mRFuPPEujDyWQ0+kAvwaL/8e90s2k0faBnWj40tqYPow80m80o/nf0gbZOIq/b9RzqBV8Jae8TH/sfuDcwTBldbx/I49MhaRtrqy36ZjL1tk7LD1Qb7AH9lw90AFacvbuVfzhdGi9/baK8y7/U+s/NARb7r7xIHLauy4ziQE57W93CI6o1r3pjWziKskR5abgS0C3PfYAB7kcFG0WdU9jat2vtaeHsejASSJBGwt+fGUdH7M7ir8Mhnxmni2Cd0R7aoYbJwcoi3WaiiTWbe1G4cDnWtmyhqNKMnrLOmmM+frN/xBPH5YD5G59Hnf0H/AvM9sMbrOayMyCR2+TtqagGtvgwjvau7iwMZnI66JrZaczy/vgwTqZrKNQyz+fWwj3HHN+vNLuaSbiheVQAK/QEDK7VGAIKK/jR0RH9si0dmTXIRD/TlsGij3wwdPBZwnJwOhoJuflcQ7fCL7dHHf88ImbyZ8ad0ME25Q+mI+oOeutXfsg/b3H5SeS77gcw3UlkITwcSNoJ/WZKR0eRD7d+iFECtQ0FS78Z6CMMxMIHX7+LA/w6GxHF8zlhgmMyOe8T+uHNdPTx8FQMxsOH9HRlrRfWT2JeqY1tAwLAXqUnii2dalZisD1zzB9vbi7ZzwK7TCaQv8Ku5wbsft+NmkySIwK3gOY2rKL5XGkwvutcj04ZQQxdRmcDf6pxFvY+LSMgUI2amPJ0KtF5hz8mJjr6mmVvvaP30Lmq4PPWa085XNG0DYwgrsla1WahPVjaW6qMD/FVmZPjCvaI8KzbkjyGSgza1SrAmehjAz9lb0G/D67XqVBnrSvgsIziAYGwcjCsRRSrDa/DCczAc/nhA10KmbUHrzB56AMrbCGuvau18yvT7FqHzkY8TWc3jms5pA/0anioUUF+YIrsPrV3tXhhafdROT1DeA0S6zKO15FnxuFn4r2man1kNvkhcYCPM/ackj65P+fp4CseJIXqzFQiOKyEnsEtiHwE/3//mlRl6yWHmerehnEWHVwg434NphZEAYbE2bpfGbNHwRGkMHj3bTDFO/ApQm9xIMEOZdtUplAIeKnUTWU37NAq51S95PA8g9rtlXa/cVU4C4ImjpjSO60b5iO/zRC8KIRme+meUf79q6tvz6+u3z599fLp66ur85dP/5STqeHhgtW24Auw5dCFgGlB0NwNuBfcWFvX6R/BNXQFCR+cCVValdAE4II+HsrjzjOABjnNdWXvqGQ/aA4xWCWxOaVipeolWzZR+4YFvluKCV5XC6pUWxcrLMHpYLaWJcG/qhDOb4BW1BjtrGn+aH0YeMWPxSsuVMUrhatnitHojIa2Oae5qUvEGcIJOb+VQ3CTJU57yyMQxX6sopUNlN/p+cradx4fgA5gqmhwh+R/dAx9W4F1V1rdbih3miPA9J0PauPx1aOMcqFhqQsDlOgu2vNBVJW0c38q4oCDq6JEjPMkI/khUxGL8n1s3ZNznJe9WsiBVUwmW/Fl5yvfD3/HO15OFn/4R/rFL9T7npa9CRSRh8nBijJ6YWqzbtf0aMpkGbrG0VlisuYHiS7R7+33zo7tlZCvl/ITpuquwrn1GaDMTpvK495hpH4aVnGTyQ2WeZUcuU9StsM3h07fLpW7n/6X6Pzu9pna+K3QA8zXz7wyPiCAEqY0NX373S/R958ifcv53zsuR6sqqLny+pdm+N3x/3dW9zirj6e/6GKOiIjgS+MPc9cJHSSFxaPhT5TYE/qZseSTAedOt6d7dEwf2T2et86HWbCVdqoOZOql9iEO17EmBhROO6F/mpIwxwn9DoPEd2Go0zqHa8UfU57QQVRSs7joWaM2COe7teMP4KETOoguF/B5vEqXe17t3T0OHD4xehZ64jDlJD9yQk+O3x4fHw9/Wqv3VzogtjyhJ1vfqCV2lqh98Kb/sdt9fMb/3okGUmrlb+dwJGaEHEVEezQ6CwHBZM4fsiFLpmKtZhyxlx38nWQwJmViGLwwld5yoNwmuf0WsFvMWUSwCUtVnJFhn2BPOudWVabkTIkJMGc1gzK6nNJcL+ByJDcNJgWD/mjnMcsAxSYUEDUCn7tpKkTzsNA9KPIMUFoXmvNfSl3pJa8kwF43ttosKxtSPiEbRL/DCPPI67qc6bUy1Z7I9xs7j6jXZwa/B6f9ED913/xkS4TDEXGL8TCmRMD6M2EJ42BPSICMKQEsV9UJnSEVd83/ODxBnLE2Xv/+1pryK/rYx9dJcb6aI0LLBGwfj+jeJsY/ZZbfGf9MmO+nTGYcH2ZMgfEhfTycim9J458pykhaEH08pC+/ioKr7pRJm8h4B51E2fTB4Sm/6nRoXY2NtgWOT7QXdkCYb5R0AWuB/jyiXDJDsMIayN/jKQe6drE4oQOQoYYeUtUB3B6m7xQcoTYvPFTZ8fEUnz9LT0S4RRo/TkdJGPN+ipwK29bB02SiypK9VlWRCkGvm+AnkxNxR/R7zoFIZA2ceUqgi6dHU0yw9qQWkGA8WhjnAy2UqVqnI6YpLz++/7LXha3L/m2OPwOtrQfs67Tu1pIRIGgarP2EjmM6pEvFBlNVXTqFF4rcSxydlnDHfXDKLFch6RDohqESgjXXEFQHN1k7B+3AkffgYBeq8noqv0LtZtkBfcwRWzOuxthDDEoE+WNpj0lmODdJ3OSHhdEVgtcXWrHbv8e/GR7ZB7qog14iRITcHWMrj46PTzsJeUJ7DjM5NMIcGOVr5aHrKoU0NK1NVRk5D98NxRop4u8xqjnwUQ0zm8kJ1B0jxFki12KWfGHe6zLvo/4p5ZWptXI5pz3zAV/nyavtmBgjPNWGHXMbsw3gpsFmj4/j3FNygNlZNQps3+9vwakeYMYvWh9oDsZC2OdDTxL2x5DDkVMRl4/yOpconsM7pFnAp9hTks3I3t04U5L97f5e0yRJay45psHWd19+TJMJjWua0aPDwXcZnXOhAasl46lQTaNLrGtINLjsFfzRddwsH54GHlTXsB8AirxaIJfDnOSRSWKTNjR+nkUnmTp4u0j9aFWSXdBcgyDeVLoO1YbMsrZOIytxjjMSlu6wc7bJBtE3o5eTSRLZZNH/+m//leatqZDdyNc6qDLGBOnHQVi9JQxHW9yYXmXenUyA7aWZZEV3pvakb3XN4Ccn3N8DBzHYg0C6LVwGdjjlm4w4SJndaudbP1tUKiQogt8WygABEdhSvzeemeVuZSs9k5Eapwtd6hqFGlz9wjjfWqsaYbq8w66c5PrtGiMJH3ZVCUD6baAV4KKh2utqU1amLnfCDA8SDGgntQEpE89SEqmOKB9IzU6FimQU9/oM/whnYQs7h1E8h03s0PNv7Hw8tJWHI6IsWuDPMvP8Qe8jjw+ybVdIRmRCjx9P6edP2NvPMrM8FJNz3Pu9D5GibRhw4qRHA3Wk369U62OGxo9GItNSaeIpoF4Jqntol/iwwVXzTTpbBugHmr7j9EotGemLGq8vWxB3RqCrsnVcW3LnTNDRJOP5gqdGku2OFQtbD148g3es7SsdBxF8SRXvantX6XKZ/Gek590G+J7A19jJc96IzyMO/o2dP+ueXsPqXloXokXncQ8GspRDz9Ul6gQAjuN/nU5FRaJM81I3YQWPpg5OAcNvNNI+5icxCZjxhfZeLXWe0aUC/CoHUjDcydlrWe8rxhn89rLZyWdGdW0h1iC9H9MAySQZT/m3tzu7y2PuluUrIVYI9HyjCt0lF05+tPNZWf355PdMg684P2AEuIYeLTXoIDrXLgaH0MTcUAY00OnCujIqJbxjnVnywYLJ1kKFafoLmXJKoJkGkldyoVNEkY+S88SFUtHlEXqrIrQ9180Sp7BDyXVSiQ2e4kkOLIRLiGD0S+M5wh+oJlTU3Wr6STvbVzJF3yXKjvLQ/m3tbBvweZ5JVismuxtrvK1naVPw5PC/DrJe9zBhFyNFrYS6EvqZeu6YpmK4v6nfYnam13BxrDiQpNz2jCLsfS9WktFO4vSnO78KsjAc+vf9mr86HY3ktXhgyEBK8BIzSANeziBC45+pMmsTTujxF4RAXT5faVWF1e7HgKX/yL9cBxVaP47v20pnlV2O06RZpetlWE3jMMNJD09HiErPWNEU1iGFnWLjQrXwp6N2VMRJSHawQrHq8m/Cx6WpVVSWTMZs9MltdhpiPBaB4MCOUymRvePzZFIOYQEiVYAMJTW+UzyUH8n22ONvbO3hRxVVC2Q7HywA4Zy6VaYCs8KENGFFH9k7w9+4BlaVwh5JHSeuRSTE3i7K6zQCPZmC9Q7TMFILeAlUuNQTkKrtIFMO61/bmoVKpo8YPbMJ3ADr3rEvgCTfWr2T7DzxGHeoWeGERDIfbd1tJ6ML0T4cMCXfZC4fTCnvXuSSzxgwJRUN49Sigihps2LD35maS1279yH2kSEluo5J4S+On2R0xUfno3wTAJAqhWiLtqowZMoIlKc9ZDtLkG0PRneJGqfXVkCgdTYa3QhKhgV1NpnPxTCJGckS3OUEKJDwkqc8omuSsY2HI1lVMAesJ84vmVzUMUTdOXCvne6CjsbyOhW21NaFtQ3AKLjwUoV7Hj1YGFb24wpdVeKVdkuKTxGIie7uLLbvjl8Nl5fR9aoNJYJgLpWGDobKx7RpuV2oBAGE+CKgipmxolJmHfW2lEF+ap6zLT9H3JFkddhL/sS4jbONAhbWBaliCHFCTifjwznByJ1DP0UR14RDAaWjjbqAbpyqPavzfhh2zFLyTAJ5KctSG05DDf1xtVSoevUo7Oa8fEWc4tWLBZcSpqDMlHotZztI8D3py94GdcLThNim0hkEPzD0HQDIb6fKY5RNSnSttmsoJcSAcd6pZTjKDwXdQTUB+UbdcR2riE+2PcjKVqUQY3d10wGy21fddxEMp+H6kXin9zb4qzjGuxp45//BkjXY6jznIrFYFrYJK1vHFJoNtQ7y92Id82qNvUONg64q+ffgr6JdoO9Sxo3Lu1m6O0AKTIJzyohj5DUK9BnGTdXgqlQNy0SXcePVRVQkqZVBMuq8y0Ml8gHJuV9J9+QYf2j8Ba1NnQqtBmH0TmoynmKHHO2M9igOMAi8t2r+YJbmOtxpzRk/rlPbvy4EU/1qnn5WslRAyEUEFOpE0k+ssXHGOhO2V2h8w55F+pGO//qX//YIS9kZ5Yu0ukEPSSqDEya+HPzAm68tKpmAqaYCDlnfqzXX6JKKhyoSC0kxcU8cks9mHHrhv4xHXafqX0W1vpPx+csugH/6/CKCcGil2akPT9H9wLJAwybHC1kRRJRQXRIo5qosuUUhJy7L5WCN7TzClJJ/5wXk6AkJ6r0ABH41GpQWp3dQF+GWela0Ptg1F9dHVv8SNBDnCoibgpcy4831wTraF7h2opcP6BUF/5LNcESl830zA9yfibMzmFVEnGazfpYvRUn4o8EXWbNBKXJKWaVDjMqxsE1P2Wl0uziUg8bUXgdRvb7H0qRYfdDxcbInUcPFfkc7JMuYe+7jL1Cjnw/AYJo/cGKTg/i71E/Cu+Ii+BOybZDiv2DpOjjTsEeuwJ+pmpoLylZgHl15ne1mZvqVjQ92thFxFqHBWCRgDzZzb/MRoInabSxQywCzeSKIUC+E45+p1gF5StUYRoQanRV2ffBGdoNfslS8+QZx7i0eXd9cXVyev/32/E8Hb/YiOZ84rSHT/MJJcTdbWuLfd2wHtQ54fBB5i77EBunXWDmf5in5hs8JXhBSGwA3UavMgOeidXBWf/GsBpv4zHPa2vbOGT163B/S9sFk2fBBFjc2PBegvp8+gB+9rYdHkHwXOYrbRxl6CvASexYD4QPfktM84AhZuAcAc42tH5zQA/7sAXJkD3igB7tlBg9Mifd2eJO/4J8h+XjhKb9AT++9ENeJd0DQ/oeeqPhtH/v370byPgDeOEjZP+itOH7qHg8I/QBx6INa49sfHmyLxYM3Uyzjln/qheDBG/oomXwk8kcf5Qi+oh/+08tXN+dvAJagyOobO49O4WFXZ47nA2/xUGCd0iwWms33HnxaIqM+4ZQ8nRP6Lbaa0/i3x+Rliq0E4JMc+T1WYvu/F78nT47P/REe5Rk725yyG6QC0KvnbAmgj9sysfWnZ69vLl69fENn0uWSDwxxbbd9BBJoemDY84Flzzle5Lo6FlwAtEgWABcTn8Bx7K/qDa1QAMhrHCQrpmwGZxIgWx9m0SjtXQPb16gZOhPF2TSObHqXAg1JKOHN6IcrDR4pEqwivw8G/99rIlzZu1mwRy5OEtUeux2zwSRHh7ESWhbKCk3wYYG4A2nOk6TSQdmZH0RTv2FN8OpWO4cIbFi+iOrJuCQBeLo8i5oDA+iSQUOM8l4f2aATc5yn/oKkn3ZLvTnd2Rd0H0rzJk0me8tgkf/rWpMeRQlrlJSLMlp9r6tGljeoIImVfZKciyWreqmKDeVop7k2P+lYVy7Leu2qnKyg3wIUxGYbv9uJk/WdZg97Am+V12/1+uxvXk4B5mgPAb78z7TVz/nLxBUN9Zoh9nyr+Y/Zuozl2OxQbnu89zrvYu1+ykJ2LYD7i5Vj29mwtzCW2Pcuejd0RHQZ9+aDHaPprO9KlxLN2aC/M9P1bZ5ApC43l9oOM7qxUnyMdsAp6dKIY3e/P8Kiay0fhliMA6im6cJzmE5hB6fTEni0WIdMK101INVuG29shO3iDsEebOKKT1V589iDbkronS0vOSZXP1fZWFesoFL4o+TzJy9hFruBDonLwoSYnKCWADqmprfNE1cLAqpAoDQdrnXabU/UzXMUe0HWgCLJXQCM7wk8ioTFUhCblDIi3zYgm4izdMDJ7QUieSxGs+DMEpUeVRy+T2NUUdq7jJQccHrxBYycdLHkW8ZxMnmSwMU+vwJl2NYCtaJQxfkg2fM0nCAZbM2QXzN1a1uPSKLrC3X2DoEBZL1bhKQMNldQy6jl6PTZZPLbVLUxmUAPNpWujV8xj6vQ3ccABBZ5PNRIKwlbTd0tP5VM9CX/nHtSPoLGDM2hoibpxaDdrao6AE7FughqrDccpAlGcEomFkPcqgrIJe+O8ivI2TkyYHmq8hsUsMeWoWGDTTwBSR6jw8neDWs+qo0EAh29LmBmVIWGfbtY7FDsUUcuuDiDc97/+hc9fbkMIlKkI0KEVgVVxzZq/Z7bg3jFEVFlu8eZ6EptpAdfkbin7HfIafmVbSvJNgp0xJUhQ8bnxKUQu8LVHki+7mPVE3rJJ5Bdvrq+uLn47vztxct/uXh5cfOn/DRRUaq8UG/dVb/0BizzwTbjwzyB1mzxEg8LLFlzDiXyDkgTmSCiujL8SkFcFATD8+dwCoFKS4N8HJBTwl3CzRZFy0hLV6ARaX5n6tLeIaGw0l7zFRnbiWa2IjnwHXkc+/iFOf5W9lGSzxseFkKw03/fazAN2BZ3JiRIaX8jO51tnTC8VJzXFuYy/ZSMn1AM/1Km/t4BfxEdjn0MnGop8kFjFHQj68JpX3wrtVMJ+uySSIaFLnHCapCvzDr9iZxd4Hxw0a5bpEluNZBMXp5kqlOlRsdACXlkvSj1b/z618wxr70uYxcfd6yUlBcrU8V8abYztEjdemcFmeS1+Ls+4yW5G2ip9ygsdrpUBZyClB2BY7a0KBsVYYtegtOLFmVw4z5j+OrViyldnZ89e/Xy+Z94fy9fXZ5fvThkA1cjFYsGxT69xwa843PO+ssWu2e+Vo1fWaZHXNBTpHF7ArNuSGvskpbJOkInoBmoRkUTJ+SQySq04PUxie+31vf66rnYRh9UATMZOT4mX8vkUjAtU52urFqcXrk/pNtCXikfoj5PHpi80lO4qqApWApEM8D8Id/oFHR5Wm+JPVe46gN1y9S088p4RsqQS+xNUlpWjCSePr84iu5md+oowEgsJ0lMVKgMyJVoI24Y+1ym6Lkj6PdIKJ31VSaRLQun/GpWWduQaysdq0OYWHHy3gWQ0xr4Cr5vt1ubug16MokpV0XItsanAJjbmo2Ma5vURSTtg1AcuOaErldoT+en0QnDDLOogCU8FZ0JLizQpSqWP2rqlFPo7P58wxaFGS7RsYV3fDZICPdeTwV4LLZCNs5yLgFZ2O4NsOOaU36RX7mw8eFDudHqCjcJUGOKd21DCIj4Mgp5urI1Cob3x0EcvJWIPvZ6xBmdbXfY5S9R2hTLofOXeX/y4HxT06JCVL9VGtCnOKHWT3vLnspikvKRxH46QlxTQbbRCHHP8K+7+72XwBHF66m0QlUnF2RwKhyf+77WYfsyqkeDEhxnVBXTX1upVVUPa5vo1ngzNxVSMyicRIyRBPReuNbViK20cmGuFeCSPddVRfnly3FiZ3cMjTDILaaFbMBM31k5ZOlX7QswUVlmUWhUcV8ofSfuoZyGlGQPy6jFgMNzgz/FWNZkwl3VuYQLnflPFeDn+25/6q/R+FRrdY4VXUTPToqm85fnN9dPry4ub95enT+7uH77r6/PX5+/vXz1/Pnbi5c351ffnT1/++Iar37BRfl9dbguL//2cM/On5/96fzZ3gGR1ZMhR+n+lqGrESPm7tzuwwJdBM2XWnS9yXBKIYACB4l8AUeLF2oM5aCPoCO/sL71feQspyZQB7zh9dzoOhwNZkvxnT/t4t84KhfScvMSXk4xeoQYBC2QEhJ2mKPLNLiaZx/Y8Rv6vAP78vExfe5Z4N2/G0IR+HrnkgQuPha3XdKG7EuWfA8JikJtvPwk+lnjWAoxTVjSYT4oU0n4UocPbPuhEKmY0OziVvRHbTJ6oaqFdWtdHtk2zOxixr3q21Ga2inMZt8lhWuyH56fC5gSbJqwWT7IYaqm1DXXvPoBr3QsNI2mgHOkaa2oNNrZaOQM5NkorHDlwjANEm/Ri6W0UgHgJZgKK21cLyhR73SNcllKni7QKSMqHAa6i+PlKhvxE2Da2CnREVtAr+DaMo16H0v8qxrVYqwJeIzKLgV2TQvYutYi2GhLokVEcXuyWTt6KYvlf6obPlmluS4s7sTRlVka9j+lv6Ebaa9mol/T/RnKMh5LupUqmmikk92tdrhzCpxW8C1pKTEBD4vDChgCcZJiJzpr+rOoxkSWD3xvFD5FF1Ewk8ljCcLjJSa/kX/BrV037JV1IGFioAME71sfPTqO/05KLBaFdpXfHWNAicZrPxQtW4WUL7gOJe/BznRPjYyuVYgygtAisoCfbl31KPm5LUaJNXNQtqlhRI4SUDi8Mqw5co2nud6gwWtL38M8zNaau2v6oVm58P0V7wsRFCjQy10exR0yPxmACD5eQ9I2R93yeQBIFkJ3DCg3FHS83xWcdgyf0fUat9sgcBcu8t39E2TKKrl7wSlcI3Ta3c8l1dwb27pUCR1v5eJ2RriQDUrw70iVqQ6KbYaLsRHj9OLAgZm2nLhY5SYBlF2vTYAlK1tuJmVkY5rc8SHAgVCs95xi9eIAFHpyHBvu/GFGeUJKpCVg67aRVH/Li5XbPRjbg5hsFQ9CZQ/8vuhOIjFTI8R5zR47qxzeUVesGd1q/iY6dsYRMAnmsPeoOJCaPadBQ2i5YfFe162QgMKzBA9222LfK4IxqXAvVeh1V3ANwMbT1P65VUAZcTBGQliXRH8mjRSbPkG92ro1PIlt6my1X8R0TMrakU8llrIEiAt73shdxODttLME0ULwmaTTZ/ZrKhVgFqGV0oAd5MRXuASNC1e59M+n5vop5XJDCpeSA5mIpb6Oy9JWnS0SDSXZUATNYuhjAbUEzslNSD+WUvKZo2goJc3StTR41KVUYmkJTlyuwk32IPYhdF5/NwPfZiSfTlPXNhekvEfLRydr0HbRHuE4BTWSa2sGpWTpXteMXnPnE19XhT6II14lFDT4m9clUi97kA7EeOESUgN+44NedyWp4JzocfTSL3dNJlZsGx+cVmsphY/cFq/wiZXUpl5oBxPBrIyOwvlG+jCCWjcxzXm25x7j0Si/qOPlAd/YuVz/kZMqgFZ4ylmD/AH7vNTO2BIuPi5ClcNI2YTkvoiqkZttWI3YBWxSBy/3Rr2rAlU9IwpiLBp2KFniUCSB6MHarpdXhNGs17o0nO6QampWVnJhGYdwETSAAHb1t5L0wg1/PlGGvb1EpQ43gBHnyvxUEOi08GVnlOWaaGk9Qck5pJ3RbYEVoiZCaXTU+ggx4ygC+KYVim1OaYxhWTI01jBp+426VdcM+A77F1lVpUQu90536kpUBJoopCVzUGss69vdX2zgkViYvxnYfPZ9GseOkOczxc00YMGFgQESYeH9ztQddNXF0aspbRATsBO00sU7uYBMkF7uKKs2eX8Wg3R7fyXVwjpYl1iOHVE1lJLKpWRRhQK/bueRVLIP8RRlkP7qy10YXHo5Y6o/ceda1WrJhpPNUVX1dbRc6pcQNNbWonl7a/ETDGVYfa4upmtcxiwlM6zwhuLwuSlOp6O3urca+iGax6I+nQ2HP5S8HArt0Zt8FnLyfA1W0jqxuYr70dLNzqn8BaRF7hI1MN+fXb28ePmHNzSZ7L1vnFmBmXrfheGTCZ1V0oO2+rxrVge4DHCp7UHjZYVJQiKyI4iUpFNFXffJ00FDCbOp7z0W5t/IUegjZ2dib+JalaXfdzdiD3r1liWlk+DaJOh3q7AiXuzK7uy2a+m0wsJNIY6lSN2A+aUmp7tCBAfMMBcrim1j05VZ7J7h11LwObwtP4GrwS75ji4+ND4yyfrfq63aU5XFSon33MktR6csyXOIF6IiBozjvXx9H4nfm1XjfnquiUo8nJGkXygf95PTr6VdPknxr8m363FMEw7mAB6B0DmkVo+o8tUQeezMWfTAOlcxumng2cHr3KTS//8LSKtM1+NRbKJTf3PzPKNvv+titmFuXtGT41m8E2N35NOuh1dGjg51Msq4A+KLjqbjAV6XePji2fmLy1c34OG3Z085uXpzw5DdIbNurHbisuCuA4Y9El53clHjFbjCbODM0wEeT73LHpukeO0ZodJo0BgV3WoJL5m8W039HiEZz8RItMhyukpQ1ZQPyPqt3uTCQ30/Td94k+wg81R0QdmGsTKOF2CqnRbaz1XGqA6adRniQqe6Ns5Y6JmaSQ1HGvZItHCHe3u9VnUwRVS4L2ETfNCNl8IBKb/jBhLwfV+mF+sG0+W6/6/U6mWj0Ww0+r9O0xHPeoWrALe7of4x227rmZqlcXnXacruLuJkkv9+G979nymM/ic2C/ieLWUAAA==","a55002150a397204a413d5c1a1c8b135d3555783825b94dd9d0528f42ca31d54",25901], + ["pages/background-processing/how-to/tune-worker-runtime/index.md","H4sIAAAAAAAA/+1923Icx7Xle39FDhVhNODuAkCKvgCW4sAkjg2JFxwQtMKhYbCyq7LRKVZXliuzALZERvhpPmDOfMP8wrzPp/hLJtbeO7OqGk2ZnvGZmIfhgwR0V+V1X9e+4At13dVGhZVRd659Z1rVdnWwazOZXK+sV60pbGNU6Grj1V860xlVuLro2tbUxWamGtPOg2tsoXyhK1vfzNQPbqFaE9qN0nWpSqPLeWVCMK1amJW+ta6dqaD9O4VpXBc8PdeYdm29t672M/oAS9IL1wZVVEbXXaMWXXljQqaemqLSrVFGFyvMtudlvsZVttiohfG2NMoGr1a6LivTKu9UaX3hbo0sC6PfmNq0OphStebGeoxQ6Lbd0Jder00c0NbBKfPeFF2wrs4mk2/cwssHRtl63rSuMN4rW9O7C128u2ldV5fxUL1pb21hMvWkPzyF1VS68Ur7TV2sWle7zquLw5enygZVOuNV7YIqWqODUd+ZhfqOR7PeVToYr1yrGt3qqjKV/dGo4TBPLl/T5Jl6GVZ8rbVp1dqVxiucXmuwKFOeKldXG5X328iV9cqum8qsTY3jmXpj1Pdj+uCR3kxXITT+5PCwtcW7TaUXPruxYdUtMusOaxN80domHPYnEiex9c0hj+gPv+Af5jL0nIae84dv5UO/v5+pa1yMCcHWN15V9tbEuwbNyV37Ga4hL1y9tDeHbrm0hdXVvKm6G1v7w7Urs+BzNc1Ls7S14SP10yzL9vN9epdoGgQat8pjzXDctla6Vqa+ta2rcTzqVrdWLyrwkA5DRjJ1aDeNs3VQrdGlz9S/ujZRpO7CyrXEMHS6v+9p5ge3+Ccd7H42mXzxhbpsTWv+0llvg/GTyZzOUZ5RfDDYuA+gpFLd2bBSeZpDnogPqH/pZ+dv5KZ8zjusXKErnFlo7aILrgUvNZXxM9V5o/LS1A5Hq+ZnqtHFO31j/GFR2cOFrfudzUtzmwW/Pbmc7Xxegz/jHuZzmSJnGomfn11eRNabKVeTSNDe25valKpxbZgpU5NU86dKK2/ATWHEwJG1hdt9Npn/XeKabYmXXaSmFpUr3qm7lWm3RGpOC7ps3a0tTZtH6TgWs0T9tJZPXse8cK055KXmM3W3ssVKmffYtt9aUj6LH0Cy8ZTYQi7fP4mD5K94evmAHsyvtX8XPwibhs+IKAEc2naQ6Ge+sa1R07woleaff/GL+JMPug258Pfg7EVm1caUXl06H25aCK+6VN/+iZWEDGvrH0wR/Jj77rNoWtirYBr1iK8pr6Oo8bl68uwiUxcBAlArIrGN69QN9tGPHYlyz+P5U1YXhV4uXVXeE9s2ZOpCiNcG5erCnEwmeZ4vtF9NiBcibc9vztqlmteqX5H6wbcnn7zhfznKjrLfgncw4GTynQ0r17EQkkFnxGmkKQu3XuPMNN1+7dR7cOBnTqB+d2vahfrbX//71zkLFTrDY/W3v/439crwnKVZ6q4KQ3KeTPLBbzmYcHiQTJ2qZI2+w8JQU1+szFqnsfOHIJQ/iqxXfgVdtlPp7vmoKGfQ/zho/pVut4KYAnM2UM+VC5m60taPJ1+6Fvp4vkjDEs3Uam2072ji1nU3q6YLRI5rs3btRnV1aSAHdZlNJgcHIyZ6qoOmQwuumVfm1iSZJmcx/SyG3j84mEw+qBcg0f/5P9T1pjHqg3pq+EXravVh8kHN53Ml/518UKObyOtuvYB4wVt8tPcPH/sfmDdQTJl6Nb6Qh6fDo22cq0bnm/HUZattfc0G33M/nP27latMtGx8cNHOg0Yqqq6EsBvbgThpWAFsn4C9grkxrTr+21///eHxl7/+8jePfvXlr9Xan/aLfHx0dKTWXlZD4/2h1YW5NK115XhFQlz9fEsYr/QOqUkYHKC44IKuhkvul3L0t7/++3jL/VrWtp5iOTO1rJxrp+Pn1KF6vL8/U7Vr17qqNmq88rFuyPd0F9ye+s8f1B5Yev7uln9pTWk9/9iI3OTf9PovzR42+W902WAaU5eZkoFa4111C8uyNnT7G9fB4OaV80PDlYD+8twHGDL9qDgckd2Fq323Nl4tW7cejARSiiPh56e2VYfkFuDH4ZBPbWuK4FprPKRsDdWNlclxzlmjGTKbWHHBdFu7soPAjzN6lSWrCPPRk/1HNLEsB0Kk8bnovj/gNzDt92+wmsukiIVr+ekZi1iynGBkuLs6aWrM1JpgamLLKcnNh/symamhmMo8XzgHcmOOMGSyRyEJCa4DWKo/wNB2BkNA8Ac/OTxUP2+TTOwax6R+UiPFrz7Sxai9zxI6e6eTCR833WtIK/xqPOr0p4kiYfHUtidqb3zye7OJShc9+pY+pK9HVH4idJe+ANGdCAnhw4HEOlFfztThodDh6AvxtlTXqODUlwO5joFIiMFnSv6UX2cTpeR+TujAMRnf94n6/s1s8nH/lBXvF1+oJyvnPJN+FJeV3rguwJHuVWM8sZtWNys2fDxRzB+vry/JXgW5HByA/wq3XliQ+31z9OAgGnQwr9TChZWYISsDwm+TCZeEOtiwzdTZwC5tWge7KS4jwOEXjabyeCviBMGuZVNHbPayt4LECksmP+i888arHCZ93AZGYBNvrWu7NB4k7Z2qrA/yKM9J/hlZlvgsbYk/hmoJpq11gFHW+1geIjMo8z60vW6COOvaAobfRC4IB8sXQ1JEk9jwJpxAnT7jLz6oSz5m40ErdDzqAyk+Plx3V5vWr2yzrWWTrn0S724qa9lXH9TL4aWKgPxAJ7L9qbur2ZqNuxfh9BQwBY7YlDJeOp45ufGR9pqq80Js/EWkAC8z9pQSX7k/5+ngLRokQh5EVMw4JISewrwSOoIftXtNunL1DWlPfW/DuIsEu/C4vwdRMzIDRdK6ul8ZkUdBnjgTeHo32OId6BQQBhviIIeyaypbaAAHqjRN5TbkGOi21fWNmBHfwbCLQrvfuC5ahwONFDFT74xpiI78mCBoUXBxd557pvLvXl59e3716u2Tly+evL66On/x5M85rJzWaJDaCAYCWQ5NMagWgA9pwJ0g0dq1Sf4wPmQqcPjgTlRldAlJACro/cpcdp4BfMnVwlTuTpVkTy7ABqvINqeqWOn6hjQbi31LDJ+WYoM31VJVuquLFZbQmmBHy2IQRVeARTZAfWqMdtY0f3Q+DLyLh+xdFLqilcJktsVkcqaGujlXC1uX8NeYEnJ6KgfjRk0c95YLoEf+gFYrF1R+ZxYr5955vIBzAFGJwh0e//ER5G0F0l0ZfbtReWvIk47v+aA3Hm8dZyrnMyxNYYG23Yk+H3inUTr3t8KODKhKOGKaRx7J9+kUsSjfYxT9cU7zshcLOTCfg4ORn558jvswwnTLysnki3+mf/Fcv+/PsleBzPJQOVhRpp7b2q67tTqe0bEMXQwxluhY8714LmL39nsnw/aKj6/n8hM61W2Bc+szQMJJmvLHvcGo+mlIxB0cXGOZV9GQ++TJJpx4aPRtn3L66n/rnN/dPtWbkcuC3wczr6wPcESZKG2tvv3Tz53vr6OTtvhHxyWvXwe90N783Ay/Pfr/xuoOY/Xh7GdNzIlSSsGWxj+irhO1FwUWjYZ/wrEn6ifC5E8GlDsbT3d8pD6SebzoWh/mwVWm1XVQtr4xPshwiTQxIFPaifr1TDFxnKjfYhB5Foo6rnO4Vvyz5YnaEyE1l0XPG70BLJLWjn+A2U7UnphciHPgUXW549He3CPH4ROjZ6E/HDo59q1P1KOjt/C7B1+t9fsrE+BbnqhHo3f0DXYWT3vvTf9l2r18Rr9veQMxRPX3Y2HsM4KPJDIwmZyFAGcypxdJkUVVsdZz8tjLFEaIPCjBLXGDl7YyIwOq3USz3wG+lNiPgHZYqqbIFtkEO8Jit7qyJUWcbIA6qwncMuVMLcwSJkc006BSMOgPbiHRGgg2PgEWI7C5m6aCNw8N3YNLTwFJJtecfihNZW5oJQH6unHV5qZyIcZlsoH3O/QwD72py7lZa1vt8Hy/cYsI8Hye87t32g/xY3rnR1fCHRbkUvxhTAmH9SeFJUyDO1EMZMwUgg66OlFngJFe0S/7J/Az1tab3906W36tPvb+dRScLxfw0DIOWkwn6t4mpj9mjp6Z/qQw348Zzzjdz+gEpvvq4/6MbUs1/UkJj8QFqY/76quvhXH1nbZxExntIHGUiy/sn9KjrQldW2OjXYHrY+mFHSjMN4mygKRAfx/Cl0QQJLAG/PdwRo6uWy5P1B6OoYYc0tUezB463xkoQm+ee4gyAGhr/f5p/ISZm7nx42wSmTHvp8hV4bo6eHVwoMuSrFZdKR2CWTfBHxycsDli3lMsiT1r4PUzhXPx6njGUJwAgvhoaVsf1FLbqmuNYMP88MP7D3tTuLrsnyb/M6i184DPW2PSWjIFKF8N1n6ijiSslELawVZVCkvRQhHDktHVDcxxH1ptb1YhyhDIhqEQgjY3YNQWZrJpW0gH8rwHF7vUlTcz/hZiN8v21MccvjXhaoQ9iFPCyB9xuyCiMG4iu/EXS2sqOK/PjSazf4d9M7yyDwlTJb47wlaOj456OPWR2nGZ0aBh4sAov9cesq7SCOerta0qy/fRI7MkkSSOIV7NnhcxTGTGN1AnQpBZhGoxS760702Z917/TOWVrY1ucwof5wO6zqNVm4gYIzwxlgxzJ1EbUNNgs0dHMvdMtQhXkGjk8Ee/vyWFzIC9P+98UAsQFtw+H/ojIXsMsTC+FTb5VF7n7MWTe4dwFegUe4q8KeSdxpkp3t/297U6iNyac6xusPXthx+qgwM1rdVcHe8P3svUOSVskFiyXhW6aUyJdQ0PDSZ7BXt0LZulyzPAg+oa+gNAkddLxMSIkjxwelJpQ+XniXWiqoO1ixCa0aVyS7UwOBBvK1OHaqPsTe1ag+jOOe6ISTph56STLbxvQi8PDiLLRo3+t//yX9WisxWiRPnaBF2KTxC/HLjVI2Y4HFFjfJRo9+AA2F6ciVd0Z2uvzK2pCfykxIX3wEEs9sCQbgeTgQxOfidT5KTMb03rOz9fVjpEKIKe5pMBAsKwpXlvPRHLHcI4cx6paU1hSlMj4YWyiAjnWxtdw03nZ8iU45wJt8ZITIcpuwNIvwtqFYMtcWcpx2dlEQYauRkeRzA4O86xiBkNxCVy6vDygdRsZfpwZHanzfDPMBZG2DmU4jl0YkLPv3GL6VBX7k+UykQDf5aapxd6G3m6l41NIR6RDnr6cKZ++oS+/Sw1S0PRcU57u/cLhLobApwo6NFAHJn3K915idD4yYR5mjN2vArI+4LoHuolumxQ1WIT75YA+oGkT5Re6RtC+kTi9ekfbM4wdFV2LeXo3LU2GFHJ+HxJUyNYeUeChbQHLZ7AO5L2lZFBGF/Sxbva3VWmvIn2M9Ic2g3wPYavsZNntBGfCw7+jVs8TZ++gta9dG0QjU7j7g14KYecq0vkWwAcx/9bE5OzWJjmpWnCChZNHVoNDL8xCPvYH1klYMbnxnt9Y/JMXWrAr3whBcGdlAXA631JOIMfL5uMfCLUtitYG8TnJQwQVZL1Kv/2dmt3ucTAib8iYgVHzze6MCm4cPKDW8zL6i8nv6Mz+JriA5aBa8jR0uAcWOa65eASGokNZUADW1O4thShhGdca2/oYkFkaz6FWfxB2XKmcGYGSF5JCWOCIh9G44kSzsTk4fPWReh6qptHSiGDkvLNIhk8wSc5sBBKxYLSL60nD38gmpCZeGvUj6Z1fUaY2C7CO9pD+nd167qA1/OMo1qSNNA46109j5uCJYf/t+D1uocJk48kUgn5Oeon1VPHLIbe/658k+hML+FkLBmIQ247RmHyvucr8WgnMv3p1reMLAyH/l2/5q9PJxN+TC4MEUh2XiSCNKDlDCw0/UlVdm3DiXr4WMFR59dXRldhtf0yYOk/0jevgg6dn8rzrjJZ5W6mcdKsMvVNWM1kmOGk+6cTeKVnJGgK1yKEHX3jQnewp0U6akVBSDKwQrFK8Tem49LWWoQlHWM2+eQ2k4SYTpkhyLGjUIqQt3weVco+NICcCpChKMa3krDyQ94eWfyNq72RxAzItcEC4M7pW20rECtUSBNW6iNZZ/iJcol1yeQRxXGkWnhCZO0iTdH4ce4HnaGcFvASiHDOJ1C6doNIObR/7WpiKp5eMHoiE5gBrn1HtgCCfGv9jqPzisa4Q+4PBSSi+ujqtJ1MXbD0IYcp2iYLfmGm8vQgpc6KwxRFNJRTh0ysKM2KDb1na0oZTs+D7YUg2buWoPDjo0eZuqKr88LfCgBIFV20ZYc8kXT/SOlNkO08QrY9GJ0CNa1ZOwaB1tlkIhkoWFDSyXQvlo6YkCzGXU6AAjEteZULusYRW7kcjqqCOKA9KXtGVC7yGER2Dszr1iSno3G0Tkp96erCuQZgFEx4zmY+FwsWipXsuMJUFVulaUnyKRwxlt1JY/t0/Xq4vEy9WnWhhBNMaT6QwRD5mDYuN7lKYECwLxwqiYwVlbZrkducTvqpec5Gdg6bI1HrkJX8iXGb1jUaWFhyUlkR4oZaE5UPxQSFOod2ilaUWw8BFK9WZIG6bnXtSZz3w5BhFoNn7MhzepveUBhqaI/rG43sYY8EeYrLV4pCvGa5pJTM6JTZ0qz5bgcBvkd9+uAg33oWEduYOgPnB4o+AYD0dMzgRvope9d6nIvKLgaU81Yuw2G+z+gOsgmUb/Qd5QMz+2TjQVauKvkwtlc3GyC7ffVC8mAoDNePRDu9t8FfyBjvauCd/4Gpf9DVeU5JYpIWtgkrV0sIzYXaBP65WEtcrXF3yHEwVcW/D35k6QJ5FyNulCZP3J0AKcqQ0x4sC8m+RqEDwbgxq16XuiGeSBE3Wp2gIlGsDIJR5ykOFY8PSM79jMRHR/inpo+RahcTrQZu9FZoUm4xIUdbox3LAAPHe5Q7CbW0MOHOGIr4UZ7a7nXBmepX8+SzgqUMQi4FUKjjkX5ijU1rXWvDeIXWN2RZxC8pOfEYS9ka5XFc3aAWJ6bBMRFfDr6gzdcOmUzAVGMCB6/v5ZpynZWWS2WOBadY2RO55PM5uV74L+FRr2IWtVa1uePx6c3kwD95diEgHEqStvLso3c/0CyQsNHwQlQEHiVEFzuKuS5LKvXIFaU3k7NGeh5uSknf0wJy1NYE/Z4BAr+aDFK04zPIi2hvzLzofHBrKlIQUv8KZ8DGFRA3DStlTpvrnXWUgVDuRM8fkCsa9iWpYUGl810zA9yfs7EzmJVZXM3n/SxfsZDwh4M3smaDlO4YsoqXKMKxcE1/sjMxu8iVg8Q0Hom62JPvsTRO+h9UzpzsCNRQst/h1pFlRD338ReI0c8HYDDNHyiwSU78XazLoV1RMcGJcl3g5L/g1KvQ2oYscg36jFnplFC2AvGYyptsOzLTr2y6t7UNwVn4DKbMATuwmXubF4BGpNuUoZYBZvOIEaGeCac/qdoExCl1YwkRakxWuPXeG94Nvsli8uYb+Lm3+OjV9dXF5fnbb8//vPdmJ5LzidsaEs3P3BRVBcYl/mPXtlebgI/3hLbUV9ig+iVWTrd5qnxD9wQrCKENgJvIVSbAc9m1MFZ/9q4Gm/jMexpte+uOjh/2lzS+mCwbfpDJxob3AtT30xfwg3f18Aqi7cJXcXucoTYDD5FlMWA+0K1qDQ04QRTuAcBc6+oHJ+oBvfYAMbIHNNCD7TSDB7bEc1u0SW/Q1+B8PPCEHlBP7j0g68QzOND+i/5Q8d0u8u+fleN9ALxxELJ/0GtxfJU+Hhz0A/ihD2qDd79/MGaLB29mWMYtfdUzwYM36iNH8hHIn3zkK/haff+fXry8Pn8DsARJVt+4hRiF+ynPHJ8PrMV9hnVKu1waUt878Gn2jPqAU7R0TtSvsNVcTX91pDxPMQoAPsoR3yMhtvt9tnvyaPjcH+E4z8jYppDdIBSAmsfWlQD6qLwVW39y9vr64uWLN+qMq4XygSKu3dhGUAxNDxR7PtDsOfmLlFdHjAuAlmo+Sis2QUu+v643aoUEQFrjIFgxIzU4ZwfZ+TAXpbRzDaRfRTIkFUXRNPJsepMChV1I4c3U91cGNFJEWIW/Hwz+f1aMuXJ38+AOW5lExB6ZHfPBJIf7kgnNCyWBxvgwQ9xBGYqTxNRB3pkfeFNfkiR4eWvaFh7YMH0R2ZOyJAZ4UpxFL4ABpGDQEKO8V483qGid5rG+IMqn7VRvCnf2Cd37XASrDg52psEi/pdKvI6FwxrN6aKEVt+rTuLlDTJIJLOPg3OSsmpudLFROcqSXtkfjeSV87Jet1WuHKPfDBRI0ZLfrmjK+oq9L/oDHqXXj2qmdheBRwdzsuMAvvqNGtXF/vzhsoR6TRB7PiqiJLIuJR2bDMqxxXuvglFy92MUMpVS7k5WlvK9YY2mpNj3JnoaWhBdwr3pYqco3uur+zlFcz6ok81MfZtHECnF5mL5ZqauHScfo6xypkxp2bC7Xx/hUP2XD10swgF00yT3HKqTyaE1cQk0muQhq5WpGhzVdjm0FBQnv4OxBxep4lNZ3jT2oCoVcmdkJUtw9XOFjWuLFUQKvRRt/mglzKUaaF9RWhgfJgWo2YGW0PRYPVG2IKAKOEqz4VpnaXssbp4h2Qu8BhSJeyoQvsfwKAIWN4zYxJCR8l2DY2N25kpC7gLBnEdsNA+tvUGmRyXD92GMSrg9RaT4guODz6HkuIolHynHg4NHEVzs4ysQhl3NUCsSVVofOHoeh2Mkg7QZ4mu27lzn4Umk+trW3cExAK+nRXDIYHMFsYxcjiTPDg5+FbM2Dg4gB5vK1NaviMZ1SMWDQGARx0OOtGa31dZp+TFlok/5p9iT9gIaEzSHjJooF4Npb3WVADgteRGqcd6Sk8YYwamykgxxqysgl7Q7lV+Bz84RActjlt8ggV1KhoYFNnIDHDxGhZO7G+Z8VBt2BNJ5XUDN6AqND9xyuXVix+m4YOIM7nn344/786U0CDmRdAgCrTKqjm3U5j2VB9GKBVElvUeR6EpvuJeBVmyekt3Bt+VXrqs42sjQEWWGDAmfApd82BVapCD4uotUT9QLuoHs8uWri+uLP52/vXjxrxcvLq7/nJ/GU+QsL+Rbp+yXXoFlqD2d7ucRtCaNF2mYYcmaYihCOzgaIQJBdXn4lQa7aDCGp9dhFAKV5kYDMiCFhFPAzRVFR0hLStCQM7+zdenuEFBYGW+o1cg40ExaJAe+wx9LPwQmjr8XfeTg84aGBRNs9THoJZgBbIveExFS2t0QQJ2NbhhWKu5rhLnMPsXjJ0rcvxipv3fBj8Xg2EXAMZciHxRGQTaSLJz1ybecOxWhzxREssR0kRJWg3hlluQnYnaB4sFFt+4QJrk1QDJpeRypjpkaiYAi8khykfPf6PHfE8W89qaUKj6qWClVXqxsJfHSbGto5rr11goyjmvRe33Ei2M3kFLvkVjcmlIXMApidASG2Y1D2igzm1gJrVl2SIOb9hHDly+fz9TV+dnTly+e/Zn29+Ll5fnV831ScDVCsShQ7MN7pMATnVPUn7eYPvO1bvzK0XnIgp4gjNsfMMmGuMYUtIzaETIBxUA1MpooIIdIVmEYr5cgvh+t7/XVM9aNPugCalIoXoKvZTQp6Cxjni6vmo1e7sOStpBX2geR59EC40f6E64qSAriApYMUH+IN7Yasjyut8SeK7RMQd6yarpFZT0hZYgl9iopLks8iSfPLg7F3Ey3jgSMSHIcxESGyuC44tmwGUY2ly166gjmPQJKZ32WiZBl0Wq/mlfONartKiPZIXRYMnlvAvBtDWwF35fbrW3dBXNwICFXrRBtlU8BMHc1KZm2a2IVEZcPQnCgXYx6tUJ5On0qRhhmmIsAZveUZSaosECVKmt+kdQxppD0/mJDGoUILp5jB+v4bBAQ7q2eCvCYlEI2raNYAqKw6QmQ45pCfkKvlNj4xRfcGewKnQRUY4t3XaPgEFFTD/505WokDO/2g8h5K+F97LSIM3U2rrDLXyC1SdKh8xd5f/OgfFurZQWvfpQa0Ic4IdZPe80e02Ki8OHAfrxCtPtQrjFwcc/w29392kvgiGz1VEYjq5MSMigUjtd9n+swbup1PEjBaa2uJPw1Cq3qepjbpG6ttwtbITSDxEn4GJFB77lrKUdsZXQbFkYDLtnR9kv4l5oMSWW3uEYY5BbTgjegpu8cXzLXq/YJmMgsc0g0qqguVP2JzUO+DU7JHqZRswKH5QZ7irCsgwOqqs7ZXUjqP2aAn+/qotW3I/lUaXWOFV2IZcdJ0/mL8+tXT64uLq/fXp0/vXj19t9en78+f3v58tmztxcvrs+v/nT27O3zV3j0MSXl99nhprz8+8M9PX929ufzpzsHRFSPh5zEPjhDU0M85nRv92GB5EFTU4tUmwyjFAzIcBDzF3A0aagx5IPegxZ6IXnre8+Zb42hDljD64U1dTgczBb9O3+a/F8ZlRJpqXgJD0cfXSAGRgs4hYQMZjGZBi2OdoEdX6rPu7CvHh6pz70LPPsPQygMX281SaDkYzbbOWxItmRJfUiQFOqk+YnYWVNJhZhFLGk/H6SpRHwp4QNjOxQsJQHN5LeiPmqTqee6Wrp2bcpD14W5W86pVn3spemtxGyyXaK7xvuh+SmBKcKmEZulixyGakpTU86rH9BKIqGZqAKKkca1ItNoa6NCGYizqbBCy4VhGES6EUoqLWcAeHamwsrYtmcUkTupUC6LwdMlKmVYhENBJz+eW9mwnQDVRkaJEWwBtYJrR2fU21hsX9XIFiNJQGNU7oZh17iAUVuL4ESXiEZEcnvUWVtyKZP0P52Gj1ppYQqHnjimsjeW7E+ub0gj7ZRM6pfq/gxlKdcSu3uJikY4ub01LXp3gdIK6jYXAxOwsMitgCJgI0kq0UnSn4kYY17e871S+NS5sIA5OHjITrg0MfmSf4NZu27IKksgYSSgPTjvo5eOj+T3KMQkKTRlfifCgBCVth9a3XQaIV9QHVLeg5ub/jQy9UoH4RG4FkICfjZqmcnxuRGhSM4chG0sGOGrBBQOqwxrFqrxamE2KPAayXuoh/naUHVNPzQJF+pf8b5gRoEAvdymUfSQ+dECRPDShqRrDtPyaQBwFlx3DMgdChLtp4TTRPCZeoWmUpQiwVTkU/8JZcsqmnuh1WgjdJr6nHE298Z1bcyElu5mVM5IPbCQgn+ndBnzoEhnoLUVU42t52zAgZhGRpxkubED5dZrG6DJyo6KSaXrlpjjQ4ADrlhvOUn24gAUenQkBXd+P1N5REq4JGDUbSTm39JiubsHYXtgk1HyIET2wO4TcxKBmRouzmuy2Enk0I5SsqaY1fSOGHa2VcAkiMLeI+OAc/ZagzOElBsm76VqhQgUnkV4MG2LbC8BY2LiXszQSy24BmDjaSz/HCVQCg5GSAjJErFn4khS9InT4/Zk26czKr+QcEyM2ikfUyx5CWAXsrwRuxDn7TRpAtEQdCfx9on8mkoHqEVIpThggpyohUswaAxHqX8+FtfPVM4dUiiVHMiEpPq2lJa2SrqIJRRHQ+E0s6KXBGp2nKOZEL8sOeUzR9JQDJrFtjT4KIVUJLUEN84thaM+kDqEZPWnGaibEb86i1XblJDyHiUfidcg7UQf4ToZNeK2NYNUstgfN1OvqfKJ2lWhDuKQVgkBDfqmdTHX8x64AlEaLiE04Dc+mHVKSQXliMXRcz/37Iyk2DU+tEavORVeqE1a+Egmta2XpoWKIFJGReFiw3UYQa8bCXNeAnPB2el3zEpCH659N5k8QQ/NiIc1eBKcQRMpyN8STJ4CEtTXyTUktH4+GiHTpYReSCQSTaWIKvK3ADUG15DowLYpi5cZGk5gj9jqNmBd3EJDAFqp6ZFnrKd9SvbyJxC+pSk2BRLOAbFLJZuN+gqVKmIW0UH086Nqd23uS8KmNbcWSjDhRQPzPziCjf0qWZaQb3QGnP3bS3MCiXkSBdpi3URLuDINOBCXXlVsMKx0u64o5/ZMtECvAH7guAXNRyNQ+jHl3EMKiTGBG4mgy0BUSQeoVEAhadkJ1UIZONDTUTVFFKbDDHhKWUypy3JcmpPXwWvULZaW54VQZlxOxubE4Oi2MtFnEbSvzDJQjE7cAKlaGza7lLWnlpdRIALZWbvaBVfbQtpRJh0xqvai9XFSpRSpCS/JATMSBMUT43g8s9ykKGPmtq2nBteoajd3xKuUZSJiXvS2NJvbi23J/Jbr4t3Aj+Xov/SZE8MnFmyuP1Wl+R/f03DcqhPFRLGhyP2OoidKvhwV1kTrhC/LIh/i4eNoqJyo49/Gn4nU2CJN33PHgHu9UI9TZwEIoUQlsKBpReN0IWo/+nbQf5SXQm1H89MhlUmXEM/vHPM7a/1+erz96sPH+/v7+QgOmhFu1qOesIRgQ8ZtIZgfhuFJ6MlBof8xFvtwdvzlr2df/ubRjBu6spy8oU1tv0BlP7BgMCwFATvAZNE3j91HZqnoCGIIxCzOfwUputbvqG0CUGND0EYu1y3I+cAQFX6D9ZhNJtRsJSdSZYqRfjYixCUswvBD+gpGNH0fG6ywKZLKloUUvYQEhFmBXOJgY5LtKCTUIw8osuJg/zYEEfs+xjzXnfAD913dEaiDkBtkZ8RpPIpPqJmXVCjyNVENOpWfD/NxeGeXzlWx+nRHS95MfRutQTZOUidBFkSxMiX2NlhSob+wCkJqHMnn25IqR6EOtHpm2qT+Y7hICVb9pbMIb6K5QiRemXxHW+IYO8zbrn7Sh8HS/VI6W0TLuW95cl9pPlHeY4Q022qWPKPeNz7MxRJJcRpSXqSOhhw+gJMlNZCUlC4QvvE78gAiGyGZ4YLDDptYwIvyuA7CHmVL6PTArQ6zeJ6SojV4Lzg24llnj3s1q4VuW0umLuBw6QtRLx1SCJK1KOuTq4LUDHeOiI26/peJoO8lWzpqPj9JxVzgfjqpvjaYetInNJ5zKNCoENgaqr4Fb6UvcE0jpOoUSlWMGAjgPl7UhwvEpJWq99ipAtNRCwduOhqtM+qTdFb3hkoqa5eYXArDsQuHsjROY8nUS8D1WlrQUFEp4lEC26PcMflHpIZTziD+6Afv74RMY91WFg2pOs8Viak/TbxyAJvcOmrsLiJoSlZuRZ5GMuULV98aaslMXo4gdtIWBksGKsPmA/Xu4KuIYZPEzuhoQ4YhInmYrKsL01I4MiUqFOZwywWVRFPuPMA1aLg1em1gyKViNakvBNuRi3G240/OTCb5RS1M/Y1bcIfBPHHUrnbiOL37ySn4tLU4f1JZA7kUZRllYxKoRM7xsND+0dFc9DzTZLLUSVwQGdm6tLe2BD4TS6To0gjyGd3dIGMktRJbr01pKdnqNPVjjyTEVCaWLRhVamnJ8VYvayhkDAlXe9D+m21ZbrnM+gD0n2oHOWEP3cmTwSvaiLWvRH1HNf1s/3B9PJwKOAOUgsOxz2G3duEeGStTr4KuSw6IyvrYhOWzGOw/dnIVzIT0HC0fHV+IWkupqpKGaz1CUhhLJpM4+hnjLGQFzFJXBWm8Musfi7x2FihX0DQONXMDe0pwBaJoFmEij6lns0lsLG0S4vr658hdjp+S3UA9gtF9PVYz8B9wudOtNCdliCi41G4h1mdL/92VKd5Rdi5bzWwLaci5YS8a9pckKXdEhGBVSbAbwlyk1oCpne78i0asOSU/CiuPkIZO9oEsBLU/plbNauMtEh3Eq+PmPj4qe4l2X6YMeCDKSF9EGvx3Z1cvLl784Y06ONj5p5suDl8yAr3zby8dHKizittQrD7vL1YMQrMITY8HlX7l8WAluMtBacYwxpKBr0lyBOi2fA9abiisxSo4Wju7c1d1Wfpd7dH7uHcPLsWMMqCbMftjlFstfyODEO0xutwajYXbgrFluVLfLeKtUlp+6iIYHQmmrzHelDKtt+/w91zzNfzDYzG/IrgbatNLl0ZXxom/98ordhRmECXTnnVKkESASvy1LigERihnRFpz96XkfmdiHbXUorKIyMCZ4gwslU/7ydUvuWNWlPa/VL5bTyVTcDAHQpKInoVY7S2SVe+wFqXB9QAtHiijweOs+9OfamOxlFRssRFE5/r6WTa0e4emqh4ote2RT1MbHx5ZMPXegVbHj9OZTgch+0jDF0/Pn1++vAYNvz17QvmV19cUtd8n0pWCB3LlUhE8yTVad4Qv5K9gMLGBMk8HKTnSRwE+kvRJoLVnCsUGg94IgqyzP0/HO+rr5RGVYfcdB828HLuJ61rlg2P91mxypqFd5kwsuyeaEhSaYnqv8EfGpAd+bJUdxfrnppyjQGA+tL2ktIWSlsxczxk4jcMe7ssf6hALz5u1roMtxNJ6ARzcB9N4zh3mChyqIQfd95U6UjoU/77G/yvlOtlkMp9M/q+f6YRmvUI38HFDhH/Otrt6rudxXNp1nDL9ORIjMeTPnTC90P9dusn/Aq1CGDt4cgAA","f01ddb0c5e0c1b5463edc8ac4c4397f9b0204e660ceab1f5a493cfa760da9f77",29304], ["pages/background-processing/index.md","H4sIAAAAAAAA/62UTW/kNgyG7/MrCPTSAPb4np62aYoWBYppEHRRLApElji2dmRRS1Iz8b8vJM+H01OK9DQ2NSYfvSTf7+BHYw8DU44OEpNFER+HzWYV3l3DYOmILHAiPoCORoFzFKCs4h2CjgiM3zKKQjI63kN/y/KVemkgUZiHQApq5CANfMuYsRQ+eocsDYgd0eWArhZpSgH1E4JxJmmpbQooclsSlIfJi3iKsoW/KMOJvSJQRNA5oYPRRBeQf6hsJSPyNaXklIJHqWcVpAFG5blQejvfYLgBfEWb1VOE0YsSzw0o+2FAhk+7XyuVsrEo283mZ2KYclDfimJaqXUyXgX25RhFzIACxGBpShjFaHmNsDc+ZMYGsiB8cZlNHxb2faCT/P39qJrkvuvY28McTC/bweuY+62nLqKKZZ+0O3/YXj/s7sBHUTRuu9l8eaBoMem6/aVD8LkKaM7yfaX+ImEDGJduea3XpV6Qjwho7Fg03b6X7DYU7W3iuqU50t2t4HZvpgWecoTdrCPFBra/Pz4X8WTEEKDoLGAEJPfnnIuYpbVipnN/P4h4Gd46ewvpc1ZibwI8Pu1A5mhBL5FPEfyUiLWo2IBZzT6bKHviqepoFrbWsT9iBMv0fikvxaRDTm2pX6l+oVOrVPValy0b8xPufSyDXs6iAzHR9fRaOl6OfYmSjrc1+aBmY0XpOMfWtG8EXJM+jERSxu6tH8DO27I65/ZVO8Ho6gbNlBkCWRPqPRymQDO6/4ztjJo2IYsXxWjxQmwrUmvapTsXpDX1H4tn/PZnUxFK5+CB0SiumKvIFx+BQJQWL1hZYzWILCgfhl5YD8e2sFTUJ7Q+oaz3nM+h5xx9HJq1iVbaf1tuU/00MfUBp7ru/89MLHh75HID+LzsP/A1Unz19lZ6fvaILTxhMIoOcvQq9/BS7/3SwEu5d/k9GbUjsry8G/Va6Y0TPb6mYKKpxv88IlRX9cHrDBM5DEsQxQ8RuP7NBIQeRx/dusV1RFdW/m4svAEsll6Lt7V4d7f5Bzxb+XDABwAA","2199212417726733df3df2068e5469237388f51713579fa8455506a0e71e4a77",1984], diff --git a/packages/mcp/src/infrastructure/export-surfaces/export-surface-corpus.generated.ts b/packages/mcp/src/infrastructure/export-surfaces/export-surface-corpus.generated.ts index 416930b1b3..d63d6a9a80 100644 --- a/packages/mcp/src/infrastructure/export-surfaces/export-surface-corpus.generated.ts +++ b/packages/mcp/src/infrastructure/export-surfaces/export-surface-corpus.generated.ts @@ -461,7 +461,7 @@ export const EXPORT_SURFACE_ROWS: readonly (readonly [string, string, string, st ["plugin-workers","./jobs/health-check.ts","nJLdagIxEIVfZcj12geQWijeSK+KgjfixWwyuqlxsuSndintsxebKG73B+xdmMn5MnNONhuhaIfRBFGIitCEal6RPLzYUhRi0SrMLQf6CJ36AlkZcp36KzbGohqqr2RFR+x0l+TTNDfltmJbbMQ7Oo2lIVHcHkNT07PR6EfOmgO5Hcq/96+YM15a9gGyM1OIfGB7YlHkRtuo2/6ZCL22wQyWhMqyaR7z/ad+QfYTZuPcbMoA97pmvypZCZ/gsha+fUBW6NR1Hfjqfzgl1Bov2TKU2HRshl+/l5E9hIqgjNqEiWY4WXcg56E2ca85Ow7yTIA3Wz6I4n+iSxo1ek8Kgr2bcInH6z1jiI5gZ93dlDUarTCQgjrnKC37eCQFZdNPS5hJC7PKoUEO1McUukTn9J2snKujEB2PaEe2uvxJn6YZNKb7/Hb7AwAA//8DAA==","Oni6M4y2BzlVG781wHyz4dIOaXyJNzGrpAs0dq/hn+E=",1163,8], - ["plugin-workers","./runtime","rVr9bhs3En+Vgf+JDchK+6+UJnXcpIkvaVzbueJQFRW1O9Iy5pJbkmtblytwD3FPeE9ymCG5X1opTnFAEEv8GA7n8zdD/frr0XkhVf5aeKFeWWvs0SSMvEGhfPHeaOmHg9daVK4wfjDqhcc0dIXOC+vfSOeN3Y6PXhpLJHJ0mZUr/BE1WuExvzCrK9xI53njqwfMai+NJvq1O5ocjS1MrI/N7W78xdhbtK5/0IVZnRvt8YGYujCrH3AttaSTw/fXQqra0g0vzOqN0LlCOvHCrC7FVhmRX2cFliIMXaGrVSR0XWcZOmKAVu25ZmXNJ8z8a6nwo1VHk6PLeqVkdu2FzoVtaVvegnYPmfAR7YVZvdVV7Ts7SAwy612MeLLojLoblX5g5KrWXpZ4ibaUzsVdO4N/F6om2RCjea2Yzlu9Np0hkhbr/9yUK6kxv7QmCuZ6ZPhDlVjkTQ2Vwa7h+GBbUPVgT2+w3TAioSgKYQ9Ob7uznbEbKzcbtDfbikQTTm0+tF7DxhkG3+ZYVsajzrbnSshybCIpdmciOtTO+LWpbcYcfLj626ur698vPrz8/erVj2+vb67+8fvl2c2bo98mvx75bYVnSgriJlOCxdUd2/dZao92LfiEda2z6DIW12hR8/CXyHXX/r8+d1jZw9WXh/es6H7u3j5ds7Ote8uvG37M5zEZdxl6jMq643fCSrFS2NgDDLIDfAdXKHKj1fZZrW+1udfP09mwmzfgM2RGO2/rzBt7LOzmmxnEfRMQdvNt8/VkDt6K7DZ65fFJMzMH9mSpN73BENfs9p0ROea9qRwr1DnqTKIjdrf9jbXWu8RGz1gLqQa0nTdVNRyLCbEzCH9G0cJI4twjxZ3llFHhu0Syt6KfSvcQbNQ8toeiBXwGi6J/mXsrPR4XYdFsbOfJDO6MzPmKyXjhUBo/psSH+QwOZOCTGThvpd50zRp2s38wtb0YYGBxJTonNphoTxqr+WjVDD5evTsh3QfJ7cy1OtwPK3bVc+COe9TUXreHQXrDPTTSm2lxSW+4RSi94SFW6U02qKU32uIX4bY6g0bl++BM389PZnBpTSkdPjsgGpJDQ7gPho4tKuHlHV4KXyRNnrCGeqzuQUwDpg8BqONkATM4hAUm8Dh7fuS12xuMoLc93I+BuUdzn7d7Xhxa272DrssV2p6WDiPH4xENjULJA9MJVLZxbAgv4TN8Mqu3eTKLObi4oh0hmv80ujOCmpJcPoOVMQqFnoPGB39V604YVMINRmytz02t/QyCMDg8hHDUgNBB+DEBW87aBRFtnszhjxpr/EmUHb5aBQZLaXZ1RDsH7MZE3FnbC5k4h8wa3Uym1efdwcjLborsCMh1BO9m8F5Uz1JIDX+ft2vsdSW6gpPuaiTldimyXHdT/k7i7Q2kONawNUr+wqx647XeN2Nx34wPOL43hpqFNlwah0m+w6kN+v3sDiYHNMlch6O5dGPDFkkuQyQyCCNjZdjeqL2TshqscqBs6+XFsdOHddve45uF/WP31H17UuwYB70qcO/xYVX/7LH68RCmOxSO/9q+LpTo+uIwg+yS+gKBFjk803iHdgJdrbexul/ahigYxLIvBIbZNv41sCtHkb9D79G6WYA66fu1NxYJp3ZW38fqths149RoRI1zmdFZbYn3bYrfI8gvcdmLt82q8cA7jLfNci/cbZg0Tdy96YwNlqYDu0tHmJBtXZ9WDloAO8K6NEaltfS5d3Fvt+fKZLcvWPYUqNHyQGfVJ7NyP0i7T+TRFlwvCsUFisxRo30vHmIR4Ubkn1aFumd7hQ79+0ML35I3CPVSZLdmvT649L14OLQsxvaf6SIHb3BdV2jvpDPjFxUZQdTd7HhhVo2FRGz/vLPN1VWlJOZ8/osZvA8FC399dmFW8fvYFjKQ0W000e4b5naytZ+/mO9bx2nr0dlIhd8UxO+imLrCoalzU1YK6fadCbEylDi0t0YxKmmrzwBmBgCiX6GO44lGAbtgoui02gbJUnRq7cHkLgTJhM5QHUYluRVS93EKS+a1oagyPMFUN1376xfiQvrXxp41hjVM/5XwWRFtqjcXIMjIRCFIdASIr9BxV3Aoj8pYn1QZK8shlmjR+G4jMzYUhnH9xWw3yXUtmveOedU+60udlBE/bem09ft4c/VQ6h3vun6xzzIakCkj0nnHkmjM9tDewRxDdp/PoRT29ix4//Etbtu2hvfqPUk5hLcJeHOL+kWnWE6kqXHDolcoHA6IHNx0QJqhwdyDfAwBYG/DeQaLo6lGTz2jyj/dpBryaaXqjdSnIQy4p5/M6jQl56l3iyPuTIZKH6TOZI7aQ4le5MKLOfgCwVi5kVooiM0fEDonE8tuQViEHJVc8WmUIKUvClT5dKG/xwdBoQqWy6V3Cy1L8gXgG3/e6YP+CWtrSnjyfXuJwPrTEGyezBc6yCBxORvppX5eaEhszuAJ5amNNbXOg+HH/iMc5+iFVA4s5iLzmJ9Mn0y4rnVelNUMvpks9J/zhV4ul/zsILJbF0mQf0iNzrEYVkQcaZSjG8Q2H9RO6g0IDVJT54UKC8gIBUwXeqHPrXDFqTKmolWZQmEdiLVHCwLWtVJwL3Vu7sGsodbsDbauPJ0TAvUcBKysxDW40Exa1yrxsNCZ0Np4KMUtgmhYI8ZK2FhEPYUrcQ/4kCGjR0jiYIQKSOeBL6SDMiSl/dr8PNakfrwuw3f4DjTej1A6PpkvdFgzHban25mR7nRnW8ps8UyjcKrM5jhOt53mKQVNPJnD06es423S/suo4ovrDz8lhqvQ9YO1sWA0wmpgaKzlZWKZmVoSZF5LWzp2qsZDu92j2Adjy7oTSua0YArL5oaRFF8l0GLd1hVkBZKNsjlgoqFNE8XZ9GuLIF3M7FNYRtPgBL+kmawu69AbXGi6GxtBsHupnad0PSEHVHVOlltZaSx1F8kE0cF9gXrHFYgCcbupLZJgltQJar2W5eKF1A4C7oO1fGCPCq7ZizpCA1YmKxa6cdYQo2zPntsAFSxaobhDXhd5fVx4GjwyfK1ZjybYFKTY2mbwJD2TPGlb6azjGayFckjNxZ7u0zjR6KqPYhb0JUvZS6leJDtXxmG+Y66g5BqzbaYQ7kwmVrUSdss2fArLxOFyBi+N8VRkV6RUyX3ljUXngg4CqRh8REbKgLy2HPso/8AW/ZRJsn8tZ9BgXE4k0VUnQaGprJbJKyas1q6zh11EK5DNcWNpZY+yIjNoqBnbEWgbyyc00YZK6YB83t6LSDnrBuzlDG6Yw7DYFxZdYchF6DRPfZT5bnQOtRr92UIdo1OkHhH/cgbnVCyAK2qfE97KAtYnetFwyesKQWR4S9gfstpyBmewJu1T1RH1ZOxuBKgs3qH2rkPzq/yBX9Ee7wzR0kfe4Vrjnw+DbRKuyDKybrq2pSSlyXq3w5hLnXNGTCv094g63YxM6pG+Pnj/+wpfP/Cwl5w9BNwZeFvjJIKQxsEjr9SB+XbcreOAm8Gvv02AbfFdMMU2GrQ+fr3VWWGNNrVrYjD9TzGUYroA0rgaFeCTxig4JFCQ5TDwC71husYeYYVrY7FvZ615GUpBbgLOMAIqS8wlaTyRxgfpY5xY6Fy6TNgxjU/hrNUsGT1pnV9Tud9QKZkJgnB/1NIiOKwEJdR4Z4l/SfGTvS+7j7EIhZ7yNrUCxqzhXxxV1mTEjzIfPjeaEIWqGRyfwHfPwxFsKiyNGRzTcwtPfQ6TjKge/Bz+7JnGm7oU+pRIcVA2Gk+V1AiuLktB0XHNQTDGSgq7EzaZBDKU2bjp0eSIfrAkJGUSzviig2Y+mVVLAR+k8w5WdVI3rBBqzkDbEOVD8cw2dkNYQ7p+PcHxLBhCpOkLwQG6suioViHita65d08WltcZZYaFDiVPOiLsC1nJwTJ2qS7MagmZUCqgeU+3ITtmjFZyNbPQy58+3Pz++sPHn35YgvBNo4DrBfjvv/8TkTuRoYRTENYQivMSeUwpPQnGFygtmHu90Kw1NwXqB3CZ4BsJ3yJWASA2eE2z3VdKZEiy/6VAX3Ch0Aq9EU5pqIUP98LBmh1bUH7C9LSYQyV8QVSaR8bYIHJ9xUU6+EDuwvjN1WVQm6OmXRbgWJJtgwCjcez+I9Q+wL3p4MGhnIpD3yToLMI2KqMAozC10WQ03/vCmvuIO2ldw3djd3QFgqbaQLCRhV5+6j3zLuEpQYfO91JUrFeiuDiioMdVI5EjGya/YEpHIbcxmwtNaVWa2nGnJgAR8BwBsaw8gQ36MQBJ6GUtVe4Y6SrkH0akF3nawFCKW+u+ubc1hnd2JRnnnri+9wVLTu/abMch5ZClKVPnitInCKgo9wrF1+k5IBHvlRVAIDo1v10/ucLZmB0GiXTUkBvWgQfUOdQVWXXH/3toTzSHJxco0FK4Y1gJA28kr2NPJHOw6GurGZ1xeWZqn5kSORMl98yEbs1rxDM6P/rs3IwULTN2ko6pUGqmExmVmXsohWZQSRCwh2IHzMX3brPuFX8poiXZJJmE8BJKMaEUMUndRhVLmpjNM6GNlplQbFBmvcfbum7OWn/V0y1qb7eVkdqzB5xdvk0OP4lrJu2z9ITCG79V8uKydn6ho0D7VyA/rTdFuER6NZzCq+Y0sjpBwVyqoJePV+8WmozXobpDB7mVa7qHILOdwXIl9dN0+tS7ZatIipa5tEivZVtwBWnarJlYz59ClSgVak8PN0JzuA+BAi27UhvOmrdkFqDUa0Nz7c8Ukk/wLcj2XPse33plSh0EpazR4GV2y4fwK2gUcOCroRytwYk2ztKO9ErapGeywUigeVfrkqL5lEtaMuFgFg2Dmc7BXzosvPYU2NjO6NadAyKPX02d5N5erLP9vdSyFKqJJd0WJ7g69JF7ztBOc2VpXcBRyUr7+XHc8+EWtyEfsj3kwSlTkbmOu4SiB0bGnKF3lxiEIrwJJwuLL709K4pJ14XXh44V0YNTkBYpM1FN/RmiGMqs5meUEfb3AVEMI7XyMYi1aC4uJCDGwUF4TynMUXbixjslNRHa5iQIIhXa+YztvEnOyOyO7Wi4HxxFlM78KfXS/amhd/EhL+yjLbFTIrapqYAILdPpQp+FEqlpqFosjUfA9Zpcn7u2oXzpdP9TWkAGHMLH1fMFIY34hfIzH5uJ+HstrjyoE5J6ChRJmgdlynr39HCPooQKreNCPGrI3I8IuvMWzVIiHJfiGu06b2J8DGSnDXCg9MVSfUTYP/rtt/8B","huudGfiqOU+7co+SemSK1CP9RDfdpw5UgbckOt7WlDI=",12968,46], + ["plugin-workers","./runtime","rVrrbhs5sn6Vgv/EBmRl96+USSbxJJt4k4nXTs7gYDQYUd0liTGb7CXZtrXZAfYh9gnPkxxUkexmt1qKs1ggiCVeisW6flXUr7+eXGylKt8IL9Rra409mYSRtyiU334wWvrh4I0WtdsaPxj1wmMaukbnhfVvpfPG7sZHr4wlEiW6wsoV/gU1WuGxvDSra9xI53nj6wcsGi+NJvqNO5mcjC1MrI/N7W/8xdhbtK5/0KVZXRjt8YGYujSrn3AttaSTw/c3QqrG0g0vzeqt0KVCOvHSrK7EThlR3hRbrEQYukbXqEjopikKdMQArTpwzdqaL1j4N1LhZ6tOJidXzUrJ4sYLXQrb0ba8Be0BMuEj2kuzeqfrxmc7SAyy6F2MeLLojLoblX5g5LrRXlZ4hbaSzsVde4P/I1RDsiFGy0YxnXd6bbIhkhbr/8JUK6mxvLImCuZmZPhjnVjkTS2Vwa7h+GBbUPVgT2+w2zAioSgKYY9O7/LZbOyTlZsN2k+7mkQTTm0/dF7DxhkG35VY1cajLnYXSshqbCIpdm8iOtTe+I1pbNFx8F6usdgVCpPH/vLx+q+vr29+v/z46vfr1395d/Pp+n9/v3r56e3Jb5NfT/yuxpdKCmKyUIKlmI8d+iy1R7sWfPC60UX0JItrtKh5+Fvk8rX/rc8ZKwe4+vbwgRX55/z26ZrZtvyW3zf8mM9jMs4ZeozKDq25E1aKlcLWNmCQQOAHuEZRGq12zxp9q829fp74gP3UAl+hMNp52xTe2FNhN3+aQdw3AWE3f26/ns3BW1HcRsc9PWtn5sDOLvWmNxhCn929N6LEsjdVYo26RF1IdMTurr+x0Xqf2OgZayHVgLbzpq6HYzFnZoPwRxQtjOTWA1LcW04uDD8kkr0V/Wx7gGCr8rE9FFDgK1gU/cvcW+nxdBsWzcZ2ns3gzsiSr5gMGY5l+lPKjVjO4EiSPpuB81bqTW7isA8QgqkdhAkDi6vQObHBRHvSWs1nq2bw+fr9Gek+SG5vrtPhYeSxr54jdzygpu66PZjSG+4Blt5MB116wx2I6Q0P4UxvsgU2vdEO4gi30wW0Kj+EePp+fjaDK2sq6fDZEdGQHFrCfbx0alEJL+/wSvht0uQZa6jH6gFQNWD6GMY6TRYwg2NwYQKPs+dHXru7wQjAO8D9GN57NPdlt+fFsbX5HXRTrdD2tHQcXJ6OaGgUbR6ZTrizi2NDBApf4YtZvSuTWczBxRXdCNH8h9HZCGpKcuUMVsYoFHoOGh/8daOzMKiEG4zYRl+YRvsZBGFweAjhqMWpg/BjAvycdQsiID2bw98bbPBnUWV8dQoMltLuykQ7B8xjIu6t7YVMnENhjW4n0+qLfDDysp8iMwG5TPBuBh9E/SyF1PD3ebfG3tQiF5x01yMpN6fIct1P+XuJtzeQ4ljL1ij5S7PqjTf60IzFQzM+QP3eGGoW2nBpHCb5Dqc26A+zO5gc0CRzHY6W0o0NWyS5DJHIIIyMVWoHo/ZeymqxypHKrpcXx04flnYHj28X9o89UBoeSLFjHPQKxYPHh1X9s8dKzGOY7lg4/s/25VAi98VhBtkn9Q0CHXJ4pvEO7QRyrXexul/9higYxHIoBIbZLv61sKtEUb5H79G6WYA66fuNNxYJp2ar72MBnEfNODUaUeNcYXTRWOJ9l+L3CPJLXPbibbtqPPAO42273At3GyZNG3c/ZWODpenAfOkIE7Ir/dPKQZdgT1hXxqi0lj73Lu7t7kKZ4vYFy54CNVoeyFZ9MSv3k7SHRB5twfWiUFygyBw12g/iIRYRbkT+aVWoe3bX6NB/OLbwHXmDUK9EcWvW66NLP4iHY8tibP8bXeToDW6aGu2ddGb8oqIgiLqfHS/NqrWQiO2fZ9tcU9dKYsnnv5jBh1Cw8Ndnl2YVv49tIQMZ3UYT3b5hbidb43V5Kk1do2Qm/TZSTMdN3YVFKgBzpkorpI7OnYj8lI3lrthVuLORnkFbYr+Pgs/FTVMXpqoVEtVsQqwMpSLtrVGMc7p6NsCjASTp17zjCEW6K9G4Qa1f09AQNzRVf6g1hn1gs806gwMyIqv7B5P7cKgQukB1HCGxUvqYiWX6xlCE658gykruncDZrtXDEH7UwhfbaNO9uQCBRia2ggRNgPwaHTcuhzKoTXdgrGyHWKarBvZ7rbGhER33G0Y9loFezPbTce57fMqY/x+y6tTzGYkoHZ2u0zDeKT4GEsZbyN/sCI2mDsrddN6pJBqzA7T30NGQ3edzqIS9fRni1Okt7roGjPfqA0k5BOIJeHOL+kVW1u9FGIXC4YDI0U1HpBm65ftNmzED6a1iSAMHm+kzWJxMNXrqgdX+6SbVxE9r1WykPg9ByD39YlbnCWxMvVuccKc1dC5A6kKWqD1U6EUpvJiD3yIYKzdSCwWxmQVCl2SIxS0Ii1Cikis+jRK+9NstqnK60D/ig6BACcvl0ruFlhX5FvB1v+71df+AtTUVPPmxu0Rg/WkIWE/mCx1kkLicjfSGvy40JDZn8ITy7saaRpfBPWI/FU5L9EIqBxZLUXgsz6ZPJlynOy+qegZ/miz0H/OFXi6X/NIiilsXSZAXSY3OsRhWRBxplCMkxLYlNE7qDQgNUlMniQolKAjVTBd6oS+scNtzZUxNqwqFwjoQa48WBKwbpeBe6tLcg1lDo9lnbFN7OiekiTkIWFmJa3ChObZuVOJhoQuhtfFQiVsE0bJGjFWwsYh6CtfiHvChQE6OkMTBiBuQzgO/lQ6qkBIPa/PrWNP98boM3+EH0Hg/Qun0bL7QYc102G7vZka67dm2lFfjmUbhVJnNaZzuOudTCq14NoenT1nHu6T9V1HFlzcff04M16GLCWtjwWiE1cDQWMvLxDIztaQSYC1t5dipWg/Nu2Gxr8eWdSeULGnBFJbtDSMpvkqgFYERFFskG2VzwERDmzbWs+k3FkG6iA6msIymwSBhSTNFUzWh17nQdDc2gmD3UjtPKX9CDqiakiy3ttJY6paSCaKD+y3qPVcgCsTtprFIgllSZ6vzWpaLF1I7CDgW1vKBPSq4Zi/qCA1Ym2K70K2zhhhle/bcBahg0QrFHfK6yOvjwtPg0eR7zXo0DacgxdY2gyfp2edJ9zTAOp7BWiiH1Czt6T6NE41cfRSzoC9ZynFK9SLZhTIOyz1z7aAK3JlCrBol7I5t+ByWicPlDF4Z46lpUJNSJffJNxadCzoIpGLwEQUpA8rGcuyj/AM79FMmyf61nEGL7DiRRFedBIWmNoFMXjFhtebOHnYRrUC2xI2llT3KisygpWZsJtAulk9ooguV0gH5vL0XkXKRB+zlDD4xh2Gx31p0W0MuQqd56gvN96NzqD3pzw6aGJ0i9VhvLGdwQc1OcNvGl4TKilBpEL1ouOR1W0FkeEvYH7LacgYvYU3ap65p1JOx+xGgtniH2ruM5nf5A2OTxztDtPSRd8XO+OfDYJuEK4qCrJuubSlJabLe3TDm0ksA46oV+ntEnW5GJvVIXx+8Z36Hrx95qEzOHgLuDLxtcBJBSOvgkVfqKP153K3jgJvBr79NgG3xfTDFLhp0Pn6z08XWGm0a18Zg+p9iKMV0AaRxNSrAJ61RcEigIMth4Bd6k3WtPcIK18Zi38468zKUgtwEnGEEVFVYStJ4Io0P0sc4sdCldIWwYxqfwstOs2T0pHV+Heb+Sa1kIQjC/b2RFsFhLSihxjtL/I8UPzn4Uv0Yi1DoKW9Ta2PMGv7JUWVNRvwo8+FzowlRqJrB6Rn88DwcwabC0pjBKT0f8dTXMMmI6sHP4Y+eabxtKqHPiRQHZaPxXEmN4JqqEhQd1xwEY6yksDthk0kgQ5mNm55MTug3WkJSJuGMLzI088WsOgr4IJ13sGqSumGFQB0N8mGO8qEYZxv7RFhDun49wfEsGEKk6beCA3Rt0VGtQsQb3fBbBFlY2RSUGRY6lDzpiLAvZCUHy9h1uzSrJRRCqYDmPd2G7JgxWsXVzEIvf/746fc3Hz///NMShG8bD1wvwP/9698RuRMZSjhbwhpCcV4ij6mkJ8H4LUoL5l4vNGvNTYH6C1wm+FbCt4h1AIgtXtNs97USBZLsf9mi33Kh0Am9FU5l6EkC7oWDNTu2oPyE6am0hFr4LVFpH01je8r1FRfp4AO5C+M3ajGx2hw1IYsAx5JsWwQYjWP/H6H2Ae5NBw8O5VQc+jBBZxG2URkFGIWpjSaj+dFvrbmPuJPWtXy3dkdXIGiqDQQbWejll96z9RKeEnTIvleiZr0SxcUJBT2uGokc2TD5BVM6CbmN2VxoSqvSNI77OQGIgOcIiFXtCWzQjxtIQq8aqUrHSFch/9Aj/cKANjCU4qcC397bGsM7c0nGuSeu733BktM7PdtxSDlkaco0paL0CQJqyr1C8XV6DkjEe2UFEIhOzXzXT67wcswOg0QyNZSGdeABdQlNTVad+X8P7Yn28OQCW7QU7hhWwsAbyevYE8kcLPrGakZnXJ6ZxhemQs5EyT0LoTvzGvGM7Heu2c1I0bJgJ8lMhVIznciozNxDJTSDSoKAPRQ7YC6+35t1r/hLES3JJskkhJdQigmliEnqXqpY0sRsXghttCyEYoMy6wPelrs5a/11T7eovd3VRmrPHvDy6l1y+ElcM+me2ScU3vjtlRdXjfMLHQXavwL5abPZhkukV9ApvG5PI6sTFMylCnr5fP1+ocl4Hao7dFBauaZ7CDLbGSxXUj9Np0+9W3aKpGhZSov0+rcDtyVNmzUT6/lTqBKlQu3pIUpoDvchUKBlV+rCWfs2zgKUem1orvvZRfIJvgXZnut+X9B5ZUodBKWs0eBlccuHcJ87Cjjw1VKO1uBEF2dpR3r1bdMz2WAk0L4T5qRoPuWSjkw4mEXDYCY7+FuHhderLba2M7p174DI43dTJ7l3F8u2f5BaVkK1sSRvcYJrQre55wzdNFeW1gUclay0nx/HPR9ucRfyIdtDGZwyFZnruEsoejBlzBl6d4lB2IY37mRh8eW6Z0Ux6brwmpFZET2gBWmRMhPV1J8hiqHMan8WGmF/HxDFMNIoH4NYh+biQgJiHByE95TCHGUnbs9TUhOhuU6CIFKh6c/YzpvkjMzu2I6W+8FRROmlP6eOuz839M4/5IV9tCN2TsQ2DRUQoWU6XeiXoURqG6oWK+MRcL0m1+eubShfsjeClBaQAYfwcfV8QUgjfqH8zMcWIv7+jCsP6oSkngJFkvaBnLLePf0QAUUFNVrHhXjUkLkfEXT2ts5SIhyX4hrtat8HyMQy8SXdt0+X7UjokXj6RZsWnDou2jwRg+F5Cz4oBbJmHpE6Tn777f8B","2YN9oD9jGIGFVDDUoc+1ZnzZ4U7EfsauTxT2lJWqPc4=",13279,47], ["plugin-workers","./scaffold","fFLBbtswDP0VQqcN8Lq7l2Yohg0YsA2Fe9ghCFDaoj0hsihQ1FJj6L8PduLUWbPcKJLv8T1Sm42x1GL2agpz73PnwjfuOpLT86HBtmVvPweVIbILY+ecJPnEQelpmasojXTbYmN+ozisPZnCCLUkFJox1iHSnXeYTGFcUJIWp/xLPIIbDknhqK6EK2pO3PCPg3HQf4FwC2+ag/oSXhl6C7druBfuXaLVubX1UuprJPwBIbQc/AB7ll2K2FDFrCUkFRe6Dy91juo4pBKqY2ZVUcNiV4fOAnLYBd6H9XoBsjJUOZRQM3vCsKj4yXh5tgZ4vij3YGWpNSlqTuU8ckHbCKGS/eI8Xaz3bF3rrjRYVKwx0XfXCU6O76wle8lCdhV1LqkMX5X69PFEB8/Tr/jJsiNJgBajkkA6+gE6HfbGFMYUZnHp5LqAmoXA9dFTT0HJQj1AnBb1jveBLDzevJ/ZHoGeIoumkWu+a8oxencA6i+CH6QPjbio4EJS9J4ElAHPWWdKkpHruHYhzTJW6+Fa/3b7FwAA//8DAA==","lVuTUiHONNOAx6tvx05C897CK0YDKdHVFwwrW16lMb8=",928,5], @@ -473,7 +473,7 @@ export const EXPORT_SURFACE_ROWS: readonly (readonly [string, string, string, st ["plugin-workers","./streams/server","vFZtb9s2EP4rB32xDTgJun1z5yJbEqztMGxIsvZDFEA0dbZYS0eBPCU1svz3gZQsi7LkbsMwwDDo43MvfO740A8P0ZUgTUqK/OYryoqVpjsWXNloHl3pPEfpTNe4VqTcKjDfPCHxe8xLNCH+s+LsYJcGBeMdGxTFrxULh3iv9TaaR1go/qhX97rejeZRW8aVJokld00959Z+i1KbtGtpz9Ba7o3abNDc70qM5tEG+bM2WzS2Tvy70Wkl0UTzyLnincywEP6X265/Bix0Nz5YW2HP9knkKvXV/la6bzu6f4u2yvmwzYJSYdJ+Ca6soILwRPVx2uMeW9p4tf2jXnXXvd2GliDfGGHPXftwnJ61No6MVxdyQ6xYoR0y7waD9th7nD9EBtdokKRj6e+s1xX5qsIl70r8MVfC/ot1J8w/Leb/XHdL/m/wAf1wQmo6oOGpGAL05GcIEgrRvg8wpkjT2QLG1KZ17mnW9IteLaC9SLMFPGmVNvRAX85gCRVtST/vZ2o4Hyxhiu54C7hFkWrKdz80fu9msHw3nKPWQVge+xxBa+ZhCde6EIqOOxLCO1rT+oT609IzJq2O2zEJObQuVN+ufUSHhyF7RR7eHdLmbyFblT4C9vV6ANBT7gMi5PBgP1bz0b2BvF2FH7COegypvp+Dwb6NzNkh6PDDMJJ0dP/kY9GpLnwxvlncifdkGDTwskTzw+fKSwoICu8YQtFca8jcveZMMJTVKlc2QwusgTOEtDJilSNYn+nc/W8pFIOAL3oF6KsbgcYU05XIcxBrRuPxBjfKsvFJL6oydUVY5yy48QKpyVYFGgsWMSYXV1bGIDFYZNDrJggaTF1M60van8u5ey1LlZVGFYoEa+MgXroUPektpk1F2HrZgA4fsua3gzG1glnWLrMi+OVTmNoFqWyYGaZy/7JA6qUJkKpiFnpyfdV6RTv+/BuG6Z7hQ7CXy1zR9ljuXn18mJJmMJi6gcQUnhVnIGJK/nQ0K9okgFaKEiETLLPZOdxnCKXBJ6UrC7n29bbTDFLnuSh9IRlCRUpTTKwhaaLN4TlTMoNnlSI1qIaopDkcpj/tElgrzFMQlMLK6C3GJDWttSmEv5SuSuea9J6NW7SlJouJay8bIdmf8hz+sIo2ziUmt3Kzpysj0Y0Jm4oz2CKWtltP00ZhrdqQH9c9t01o15qfkWGqTfMYz2p/n4A17cUjmB8/u2UjP+cxXeJXUZQ5QpIkbP05LbcAWJ54i97G1AaqSouGp5M21WQOL6DSBXjT2Zvvvp/Mm8lbwMRURIo2k7m7GR8cKkORc3YmM5TbCby64EmSdNXh9Oc6uNOHA3h1W2vjqdkT0oCslyHHY/upZQwKUYKQ7oJiCqtd4ByqB5zgUhWlNlyX8DIssa+wNrqAySUhuztV8kWZVxtFZ026M6kNXtS57OTtvkEtz/s/BGHch04jHmEJLzHBoRvefvZmjP2hPsX0GnTk8fEvAAAA//8DAA==","hSOvD+mncCxrjNZvybngoZiULKkURNn2mxhjDzhiykE=",3832,32], - ["plugin-workers","./worker","tVndUxs5Ev9XVLyst8ok7/YGjjXsBSpOWOD2HkIqJc+0bQVZmpU0Bi91//tVS5rRx8gOXPaqeBh366O79etPPn8+mq0Zr3+jhvILpaQ6GjvKe6DcrOdSMJMTbwVt9FqajGqogY50A9pQZd4zbaTalanXUuER50DrD2AMqBugWoqMVElVJ6RbIxX4vRfizxZa+NQYJoVGwhNULf5AaVqkXH369ev5xdn51w8Xd3cXN18/ns0vbq/PZhfIk4uZFAae8KwruSjcm1Djqy1jyQTD29zv3yjjrUIbXMnFeypqDmi5K7mYg9Z05TnXdMclrW+rNWyoI92Abrk/9aYVAtSMy+rBEW7bqgKNunxg2oAI2vpjgw6e8Dsa5Wh89JFWD2H1dbvgrLo1VNRUhevt4jvFVitQMymWbHU0PrpphWEbuAa1YVr7AwbEPyi39+BZdcuhvpKLS7GUEUnF30EWfB5WWc1XTBsLkTuqHxKTIqF/0LAXycGg+OvW1EzE31cOR16ru12DK/8t1YOVx33M5KbhYKBwg1+ggB5kS3ElF8nvWGtH7bf3iMronds4cnAlC15HvKxh00gDotrNOGWbEuNSNK0pMTxcB/Rb2aoqXJw+haM5uIHyuJ7RVocNuTlyVHuqlDxf2Rtpjw16fkmmgAkblQKxuNL71Zfx5yMFS1AgrMo/8m12DZxxRlEfJgyoJbX0fd/x3i1VjC74oSPj779L5H3ivET8l+z9u+T5f3zvs+2P6P4j+r5Ehh+RM+xNQE+GWT7j5fm+yI4yf5nvHTljDqqBg3wfsVATkhcH5B1pxYOQjyLWO1mGYfaXu7DwhDwTBbSWgu/IxuWNy3pCtFFMrKaBZ0uJj3QDBV7jotuE3EXENdAalJ4Qf6nbNfa7T6KVNXC2BbWbyVaYCRHtZgEq4oOrY+ozU7h7SRnfw1LWKJOBmeKj8blnsobTwn7L9Lm055P/7LFtX/7k5qVNA6IeKWuGSeE1Tn6ekGslN0zDL1vJ6pMp4UybkXSp4XTSnRat685HcRslsQQaKfCWmpCDt5F3J+l9Y3LoKvceJ1NSQ2PWoyEnM0ladlp8GRD447KOzFwDp7vT8NyNYlIxE5NqqNuGs4oONntwne5DlxUp+NGw8K2k0IbsLX8n5P7o0SZKPfkmF8c1//P+KDkyqY6tNxZKZPJu+ACh4D0p7uxxlGwO6Er3JwIl9WHCCcV3Qg5leEIO9WNCzouYhNkX6QEIablOnomQjwif7nm1L2VGFgpz3XHGpKKcL2j1MCEji1ZE6c/xj+x9kz4gCJB0BOSZVFJUrcItCfAk55e4Z0v5XCfw43QH9fU+/pZptmCcmd0d24BsTcTb0KdzF9YYxHvQLzHW/VPRChL0wxJMtbYxMKJrthKUn07I2UIqc2t/Zf6W9jkvjeevCbrn1MBrIjutHkbDmCaQHAJN1H4N18KTAVH/0Rt4tGGcMw2VFHUwZ7RvISUHKvJYFLd8e9OeoIZt4QaMYpRPiD9q2plg5M04IXdxoEzD3FCFiooK+DWImonVSIGWfIuRmdU6RNk4FHeijYkUM7uZA8a8URQ/u4eMvSI3QS/4nIpdJ3y48zU6YBoCMVq7MDEhiSkqB7hJBsDD6SVxyeGF2shmAJ3sTSPg2Fe0KpxG7+by/mkp8Zfy/f40HwLMnulAEGo4JyDPpZrpm1zE7oghcENPJ6TQJk6L8SoTrDiKOMDuhhJhSWE8MWSqRNl8ZBGrmhRSrunstCu1rujqcZ87WJu2wVNSxYOE3nDJeGFKWg1z2Ei1i1d6fGQGLM1aAncwdUlZhfFHuqCUSeOZTIHqpzMRJ5nTBHo/sbE1xOGxDXlHbny46+NMtrM4z9mzL0AhmfbEQZWVkk6X8QsshOlfUpRYIHAsUKce7kM3PJmbVvTBLWJxqvexVCuStJcFmAKkyHOoVoJm44FCYxJipQ2EhaA7rLATI55MSSsKlxUj/QpMuqRX1nYQMcFZ8XsH1ky/ZFkxTk+JFCPYAhq219ulEGuQlBWnsL6kyx8hGxCWAXZ4p40bWPxZgHcFyOQg7Advk8lhLUCVGfXBK0rNpSatUXKlsEcrbBiTBlRlTdPVvz7Nnh48s/K+Xj4z13NPYCicnAXI4ug1ZxeGsAeWdOPYA0v8mOPAin5Em798FMvJs/WQJOnu0ziKhaXR7p7ZSjLuJc/ENY1xii9lxq5kScrTqLHD3qMrXTBklBvEKTFUP/xeOivKPRYppb4nT88vyMp5MjbRvLlbE8+g3Yru1HhFuImFJ52UcWCrJn3OVG5BnxmjoDr1D/AvxaPFj/m0PRRcgYYGMWpne1Vn8qh57UphUHP65AdycVvXcZ2b7G5Ag5mXFlxiJUH5r7R6kMtlccmcPg3ZxRiXlIvkmTRUaRhtscSL04z/ylEe6f7CNL+nKEuj8r5J2f4akDxb3ftCdFQMdy9x5ClpmxqjfLJw7Kl6Qq6pQvOjr4WyLp6/pYw+a+67eJg4v7eyaJXYZ8izdz4YofNMsho0iuylCnSQucI/evZfnYVLvPZV8TLcUQQSztqPxvnfrUF7+WaNmDU1BJ4aTpnQ5HG9I7TLggRQZKiJWYONi8fcxkEXA97Yo1RbmRbXxHw3AyWPihkDgiz8EJvQmjYYWMkCllIBPl1FFbbppJEMpema5jf34h8GNg1HlN+RT4qtGA5gOsn84JugLVCQORNsQzmpW2WVw4k9kAaHeUuphsJ7EV99T+e2eKhv9lH6XmxnYUw6usGn1tBQRQ2u8fUpgac1bTUeQ5ZKbohRVGiU9JjDFriVlThZu/POvVYGu0kU75tcdEZuNR6OL2Sl+Unj/EMKVqE1IrX1mjoVsF3QDVSMcvaXHfASuUwOcHPztxga3vZD7rd2CG0Naw0jlfHSZX84kiNU1Pi6jbS+2VXrKGmroUZEUEGYOPaHO4Vs1PeHxoZ28dmqKRNT+0TbDUSw2NuyGgErO3NrhET0RMTfyMTqzb3AyYnF/QYMramhVnA8TkkEgVnL2snQbacV+haHerUBYS0wQ6R1bAeu4OrWuyjnAfqbVhvCsCa0B9yLaye0JpS0gi0Z1OTs+jJCGErvxdLtJoFbit55AbSZKQWs7Mwt0sPKYS/o4RXjprOMfxc3YmkR0whgqQj1SltVjS8MCLVP+riG4NOEKsW2FoNJOOpbvSo520fbMsiGIPFFuTWWrztCLeWdJfdg25l8dwe2h1YbB6kewegy/l8VfTuq+g09hejWYcG933f29mml828FplUi3xUEPda2Euj88SKjH76884aCkTFMdIXqK06ZKahxZkr5saZLIBU1sJJq56HSlVo/abKR2qCSiD78X2KrLDRcYtuPhS7lUE62lLPaLXDjPEKrChrj5POyeXB6XOYIQCUa3q6YOMZ/OahyTCKNlPz11tXp077CotlOTPP+xeX38BQ/ht33PzxidJ9Doq0wXrr/y5f/Ag==","23UZA/bjtCmRNuTyxAd65+WtmT3xpKc9ZrtcUrLf+SI=",10175,63], + ["plugin-workers","./worker","tVpRc9s4Dv4rnLysd8Zp351tclkne02mbr1Jbu+h7XRoCbbZUKSWpJx4M/ffb0BSIilRTnLbm+mDA4AgAALgR6ifPx/Nt4yXv1FD+aVSUh1NHeU9UG62CymY6RNvBa31Vpoe1VADLekGtKHKvGfaSLXPU5dSoYoLoOUHMAbUDVAtRY9USFUmpFsjFfi1l+LPBhr4VBsmhUbCIxQN/oHWNEi5/vTrt4vL84tvHy7v7i5vvn08X1zeLs/nl8iTq7kUBh5R17VcZfZNqPHWlrFmguFu7u/fKOONwhhcy9V7KkoOGLlruVqA1nTjOUu655KWt8UWKupIN6Ab7rXeNEKAmnNZ3DvCbVMUoNGXD0wbEMFbrzb44Am/Y1COpkcfaXEfpJfNirPi1lBRUhW2t8J3im02oOZSrNnmaHp00wjDKliCqpjWXsGA+Afldh/UVTYcymu5uhJrGZFU/DvYgsfDCuv5hmljU+SO6vskpEjoDjSsRXIIKP51a0om4t/XLo+8V3f7GiX/LdW9tcf9mMuq5mAgs4MXUEAPsqW4lqvk79hrR71QlIn2MCNSX1u3TZd5PXpbXo4cSs4muSNelVDV0oAo9nNOWZVjXIm6MTmGT+sB/VY2qggbp0fmaB/YGop9wSG10WUrKF8Wc9rowOz73y8KT5WS9yW7GI+EpuPnTA0pZZtaIGYlfVl+nX4+UrAGBcJG4u/8Nvsazjmj6A8TBtSaWvrY73jtjipGV/yQyvj3jzJ5zJyXmP+StT/Knv/H77HYvtb3l8TkR8XhJXb+HV/C2qQwyBBI9Hh9SJFlR+Aiz/fF3mMOAMdBvm926Anp4w/yjjTiXsgHEfudiGGH/uUuCJ6SJ6KAllLwPanc1XRVzog2ionNSeBZtPKRVpDh1a4DzshdRNwCLUHpGfGbulVTv/o0kiyBsx2o/Vw2wsyIaKoVqIgPDiqV5yaz95oyPsJSNiizQZhi1Xjcc1nCWWa9ZfrruuOT/4zEtkNY/fDSugZRTpQNwyxzGqc/z8hSyYpp+GUnWXl6QjjTZiLd9XE2a7VFcq1+NLdWElHWRIGP1Iwc3I28O033m5JDW7nzOD0hJdRmOxlyeiFJka3NLwMC/7gqozCXwOn+LBx3rZhUzMSkEsqm5qygg8U+uc7GssuaFOpoiK0LKbQhowh7Rr4cPdjLVM++y9Vxyf/8cpSoTAC4rcYMCifvhgcQMPVpdmWXR8nikF3p+sSgBIImnIDvE3JA+gk5QNSE3Ac6CbN7B4RESF8E5IkI+YDp0x6v9nBnYlNhoVvOlBSU8xUt7mdkYrMVs/Tn+I/e+SZPjWBA8uggT6RE+HrHKpCNWWDytJYUUhSNQl1JRkrOr1DZjvJE3NoL5XKMv2OarRhnZu83i3gVfbxw/Y5BvAYLFpvgPxUtICkLWIMptrY5RnTNNoLysxk5X0llbu1fvUJM31gvbfSv6cYX1MBrWj4t7ifDZieQHDpQ9PQbysKjAVH+0QV4UjHOmYZCijKEM1q3kpIDFf0mFT83R+9DQQ3bwQ0YxSifEa/qpA3BxIdxRu7iDpr2v6ELBRUF8CWIkonNRIGWfIctm5U6tN+4R7emTYkUc7uYAzbDSdRY24OMy6Ufgs7wBRX71viw52t8wPsJxGTr+seMJKEoXMLNegl4+N5JanW4oTayHqRO70yjxLGnaF04i86tVqBB7eAiTvFYwCGGsxxkyCGFcYAQWtPI6CJYPRxikKcc2vouV3G9YvOs6NmMZB6h+YbWMyw7JznAbicmQSQzOxkyVeJsf54Su5pAMPekbb3LPYyxF8Sv6IFs+sg+IUU85egCl8w+TkijYQGVVPtY0udHL4C5QVDgDkZCKSszm0kFcndwPDDKUP3oKOIkQ6RA78ZJFn0cnimRd+TG98OuEfVWZodNI+tCKiSjqLjrstyt1GKFDAvT9C8pciwQOHQo0wr3vR0ezU0juu4XsTjVYyzViORe7HWgTEqRp4BzgmfTgUNTEpqp7ZSZrjzE5kkQT09IIzKbZa+CDZhUpHPWvj1igovicwpLpl8ilm3kJ0SKCewAA9v57e4YG5CUFd9xHRjsH0KYXv544JndK755Wig1hjc7CYrgzUK+JSgmy7xU2fnSta3IvZxFvZlsvrwOr7RdkzyRwpZ3i89mB4t+kJk9O+z5U2UmXeuOkEvucVsruVH4ts0smJIaVGEToz0+j0LODuosfKfL6+z7OdIWM5p710N22t1nZ+beB0TaCfgBET8eOiDRTcX7Jx/dZOTJ9ocEcrzU48FIPboqcnP1kaFVMmv3sF/n68091GNwlMMULRpMkH/0mMZnXQv6sArzj/ITYqi+/z2nK7q1T8eelH1g8wI804cxJvoO0MrE3wacRKs1lgg7sZAOs3wOWbypL5jqR9Bjiug6OvEH8C/FI+GH/leQAFUDDQNi1L5tbenAoH1lgFrQRz8Ejdtjy3Ultr8BDWmXbQWuEINR/ist7uV6nRVZ0MchO9sfE6BNnkhNlYbJDsFxfEH7X1ZFVACR7y8ESCNwNu3oY9PJcfRMnqzvHYSfZFvlS5rACWnqEm+IRHDqqXpGllRh+LHWAiCOZ54po8MbYxsPIcdzktmoxDVDnnzxwQSLZ9ZD79GtkMPug1svfIAb37rXanHb53ttlElhj2wi4feNo2n/363BePlnLjFbagg81pwyocnDdk9oe4MSQJOhJGYLti8ec9sHXQ94Y1WppjANysR8N3cmD4oZA4Ks/IcDQktaY2MlK1hLBXh0BVU4ASG1ZGhNO49480X8w0BVc8zyO/JJsQ3D2VZrmf/YQDAWaMiCCVZRTspGWefwKwmQGgeoa6mGxnsTX71PW7ao1M9R0PrObBdhvHR0jUetoaaKGpTxyJ7A45Y2GtWQtZIVMYoKjZYec9gBt7YSZ2ur78J7ZfAdjuZ9l6s2yI1G5XhC1pqfNI6WpGAFRiNyW2+pcwEfWrqGglHO/rJDdSLXiQL3reIttoa33YeFt3bwbwNrAyOV8db1/iHUJVSUeLq1tLXZYm20tNFQYkZQQZg49sqdQ7bre6VxoF1/tm7KJNT+om1nTQgUd6zEhJVtuDWmRHRExO/IxObNF4FDKZv3FRhaUkOt4ahOSUwCs5Wls6FdTgusLQ7lpgJhIzDHTGvZLrlCqdvqopyH1K8abQhDPGkVfBFLZ7QmlDSCrRmU5Hx5FWUYWu/N0k2VpFuavYtM0vZCKWBjx5mRH9YOu0GXXnHetJHx5+KGUw3mNCawVIR6p62rxgMDQu2RPmwh1DShSrGdzcGkHXWP5CLR7bttPsmGSeIBvQ2Wxx0BS/li6VewfdU8uwIf1tYbl1JdBmPJ+M9D3WNSdQs6CtGNywV3fs+sXUghDZavnWgcqCO03xttX4b20DgTrjLSY/L1/bCVvN3WPsGnhImCN7YB20coKfCqbGq7rwZjfJbG/zOtbTwKTKNE350QwWNtIUrbKC579MNRacs0c/rYv1oE/TotcwUljsopP9Z0DaSgBjZS7X0atzDwJ00qqQ36iZWB35YbZdPWXbrjedpeh5STHeWsdAJuSEtoUUBtnInePF84vmb62Yl+1LzZMHGMX5pUvl+SWkr++gDrNO1eEdTeSoQg/tDlc7ken4dd9z+cY7SfS0aLfl66/uvX/wI=","Cdt3NI0m2qpCPzOXW4LZXs/otN1bu5DreAfC8xKgKGI=",10802,66], ["plugin-workers-core",".","zVldb9s6Ev0rA79UAezs7quTuE1TXDQBem+QBNiHpkBocWyzkUhdkortFP3viyEpifqwnQTtxT5ZooZnhsOZM0P669dRqpFZ/IOJrNR4g6bM7GgcRq/U/E6pzNQDt2WaojEdsf8q/Yja3JTSihzdsJKj8ehCK/kZswI1IXBcCEmQ8fNnJnmGuh66Y+axfiHcRabWo/FISFNgav3s8BJkw1skfKXmH0uRcYd7peafCE1Y4YxqKW1eLpS0uLGtse68S+5/r9k2U4x/YUXr/a9F6/U2XWHO/FDtsMij4fG2YAR/jToXxgglrzUatCRR1GP0dl3OM5HeWiY50w26sUy7xXsvk1OaxdNbaxU04JZBD7eWCxk/Xxkn5NEuOeaFsijT7UXGRD704VIWpR36cK304PitKnVKIVJtV2NrNRLZ+238dbQoZRqMP/j4xLRg84zw7bbA80ww8slrIA4+CmlRL1jaVfL/8BytX+MCNUpn5m+2fhfm8Hi8pzBAPqd3/hfOoJSPUq3lLEGtlZ5W72PgzLL3UwiSR1OoU6yaPYuWHbRUqZekduNmtDPfo1TJ2Znb4r0hC99kUZs3E6aX/64XeTSFG2RcyWx7WmkhZlXSWCB+nUKbXsnD8RCcDQLUJtQUfHp3yQE3FiU3YKwWcjlLBJ/C3SX3Cwk5SoJjuB+57GTZ/Wjc7MheDcHRp3eBGccw4MEiZk2ntsWj9eTZGFYeL97EPvbM296j8gHJvtHEiAf8EhHt2xxT8d0BPR2ifK2upnAm39V8IKh2BFoXgJabWGYe3w5RrSRZh4eXQ9VU0gnHjt/GcHeh5EIsS420v72dhh+A0uptoYS0ScHsahqmDkR67dc+zglIlmPyxLISIwS7EuYEOJpUi4KWvkMiBPDp3Z+4sQEczmI99KHOkUprspBTSFLPV0POg7NZxB/9zA16JwZtK3AidbMTCJnYsm6WkOURBZudeRrP2uvVSDD2LCHzMsMEN4VG1/70HEh8+azkrh2gz6q0SW6mIMt8jrr+pNHqbZKzzQ1aLbAWGINyO2beR469wVRpfloFV+XnWY0WdWgJPZvXTLZsaZLj4+Mu7/uvOVpGVcWVFoIdRIvXhdKFXFjHayxBSV0Dr7w5VypD1phiVSHShCK+5+m/SyzxTovlEvWwxJw2vx09PipiSr4cztafVV3riXcSH84aBmjyqQm1XrnZUR0rZd2qchCsycpeY9FUrjg/+8q64q+2dqDGHbS7b80l3+ff4TnNgYi6naUwVm9rhINheGh59fHq9K5Z4l5DOi1De8lD56hmjyKwnd3eoGrXOR5YSXXe2yfWOwjuEPadYERA0/7U+BwwuO7ReMTMVqZQ1+r4ONnrSK+1yoXB05497RrdaY3eUKR1aInJIz1G2V3Bez3Z3hL+z1ThgT7xl5XhIfi2c3fU2F9b4ve4fYf+NxToX1Nzmd5Xcw1dglQnEb8udy8Shd5T8sS0U9nSFNixFqQWV8jlJ6G7Mfobav+B8j1cgdtXQ4dLcF/+d9Zgf0/1ykpU79eeT3TF1f88fOO1z8Dhq7AdMxpqHLwogx+Qkr5EEMZ0B3bEv8Pmzk4gZ/rxvCgygTx5xG0VdmOwNvtCrW3V7Fr1iPJ9FJYV9JMSfEZZmCEz2AHZO6kJlR23fi23t13SOeC+oWIMM9qu8/OLSPK7mt9aLNxZvDKk31xH2bjjqG4sFhXXD9UhOlX/I4pMhh0tvNSM9HyJafbNy/iFlLanQAzTWf/2mOzuBGVfYB+79DOZ7i4v3NUdMFgwkSGnKAHtfHDs/vJwX0OFB+uawjlLH5HDfAt2hcBSK57QVQiSwAxzKnkQOooIhYHx946LMhvWw2Ch0awCmKn6JlholQNuikykwgLHAiVHmQo0fjI1euGYDatwX7hQujLqu5rvFART+rwdkL+l5hFYNOzvupy7IV0xIUnMbQG25SqPrVfKYJXNoQUBYaA0yIFZ70FiuNQFLsxVSf3s9vhefsANy4sM4eHhwZp7KfLCk2v3IhJ+ege9+yDR+ruafxVZuRRyEhw5SZXGdycRxnM96Vlx+nIvfQdeGX7WU5PcS4DnYzX/jqlNfgBXaZmjtJd8Cs/HPs6SI/h5NCbBpOYw+OlayORHtf1TsLpEf+09raSOGzgPcXRyLx8eHu7lh4JplldyvmeDqu+H8P7EMsGZVRqY5LVmlymstCulhXU+9VjVIs8jz6csyyiyQcgnRfG9Fnbl9idgY71/qrRFaQlOoy21hHM3m7qWGjplWm8pBQnBsLwXBBRvS5ToGcuZzYUpmE1XA7FHpDoYfB256i5yUPbSX1wC64YyLVWV1p+WyGhWpV57WteKl84bsmr/3Bul7KRKTbLW0SQ5YlugsUQW59eXg5KRDo5aPCH3kU4bYVdCQuFOjmEfCOKP6rRoV8wCbjAtLRoyKMZusttxiac3KJihVLaqLRxxgKeeKj52STUBI6ypY6UOa8L4qJnkFJgNzQhOt2QLgU7gk1suMMiERc0cyU4En1g1qQBzVnhvsNhPStdm6HDf4TdwQR5fNbFLRgrP/b0gUjq4dBIKRIAMURqn6yRVecGsIIc0mdsCH+Rct9++7/aZ56X7vOt8Hv6IClULN4UKO7VD3l1lqCJk5Etm+M4wOsIBXfGibRWVVOW5km6uDy2SyxX3xeu6O7dXuRw1UOY5+ahYSu6Tp/ZWUzBLQ5HEccHIV+RtZRoPxiHoUvplydXN/hdmVydsHUo7bj9uKd2Uhqvbv/6EtRbWogRFtzxWgSnnhVZUOPyBdky6JKQZbQ6dETYiL3P4D3wRHwnNgVBtAZamWNgqokjxu4BBnCglZpC41WBoyryvi4wJCb7OGaCW6Wgo8NZkRR2oHDPxhHoLzFrMC2vIdHcMAiZ9iUcOjxgSiw5XZUUcaFRGeSv54AwXEEpiVxUhndsJnWzsxHmra4srYQ3YhMCWJeXhWkiu1sf38hxSzcwK2MKiJsrDXFkEXCwcdUsOc1wojfFZDFImQWOBroEhynTSJ/eSrPQvjoii8qrx71JoNCBk1b/ZbEuB4E9VFrhaS2M1Uq1HbYShs5YrBp/VesDR9dR067y0ZqZyJe/Fbl2BXhbqQwXrReH+7dv/AA==","Ay20xhY/bP/DhM+Zy6w0bmirEq6YFsCq2gx2AgmXfzc=",9348,40], @@ -481,7 +481,7 @@ export const EXPORT_SURFACE_ROWS: readonly (readonly [string, string, string, st ["plugin-workers-core","./builders","tVhRb9s2EP4rhJ8UwA327CQu1qbDYmBbkRjrQ1uglHi2ucikRlJJvKL/fTiKIimJstU0BQKHIo/3HY9335H8+HH2puYlA/Ue1J5rzaXQs/mw829a1hAG1lTfrw8V9rxVUrx7qhRYudl8xmDDBaxk7tso7T8+SHW/KeXjbD5bydzp63zcGWpQ80rm1ziFm0bxSua/U8HKVtx9vJXCwJNp+m5Y8/8WdF26vlswIFDHX5Vx67sFow7hEw0MlkRfrSnY1bEFO4Ix+GWRsXFnGBdxe6XtlHbhAajX04K13R3AttPCtB8rmd8ZqMJC2gHsjT4bU4Lg5/nHGRcG1IYWiGgOFfxacqqPtDe1KJwt6eb3KvyR9o9gKdiAAmHnxu2Xt7/jZjJMNPKVKKBMivJABJjXC5LOu4sgho0pco+KG5giCOJhEm4tpohtNvz1guRSlkBF1M/3lVS4wFrcC/koyDfnqBGV5KqV7Mm1xDMU6PIQuSLaKC62UYwSz0yX6xtG4MmAYNrJLTPOFmR9w84WJFARCs7Jp5nNQ1p+ms1bWN9YDhHQyBMQEcc8D6NN7RM4PYqZiBWCtuuKHtCcrN9KseHbWkEY6xH5nKzf00MpKZuTdUPLS/KVCLqH7AH3euG0nS2I2XF9QRjoQnFLVSMSIIw6VJILk1XU7KLx/tb51Q2NuCC7hsAv13/Ck3Hj5CoWxYFGHPvdxGwjLJDj/8uAEcSXyTBygK80mI7r44kXpGrgO2YtM1y5z5850cUO9tSCOJE729OdddQjkWDsFdTM6hIy8Nnkcf0OGL6H/6QY20IclrXJ9npBRL3PQfkhhaU329MnrMEcvMCcyKY6vV6QuDz7iVUgzgzbOsFHQdrQrc7Oz8+p2v4yNH8PhjJqaIY/iFdIxS7bqG4zIba5OUFkzkbr1v65IgpPmpfAWt84OgymyYoXGSbAwG//1lDDWvHtFlRaIscld4Oh2eRwWmj2OZF1nnR7KTok04G+XuZ7eg04QYlHDF1LckWu5Z5yMcnSyIw2yb4fpj/zmHp3hnwGSn9mEuWGHfNhR18jGk9u1F0eN6Mr1J3ejVJyRW5dWW6Zy4vHedcJilAQeoVrUkXoH6hHSoKqBfJGhqYs+gV/AvcPaupPI//o9D/K/okK/2L0n1IfOX6c21+2tBxx+Aj+M+rCM/mfqmP8r/FeljlvNAuxV7Uoyh6yB6r0oDa4lPWCj1Ldc7G95qofjj+hDp0oLOna0L27ni4O/WwdVoehxhcrD9OMjQ15ZoFITU0DTCXvILuMb5YhuNK97nUgMGziyD6JZVOvCSNMm6aBsevCJGb5p3mKsFeP1kp/WEq/V3i/L8cvKtpA1fJlisWNe9k4jtt7/3gJYF1CD5XViqL6P2LqevayXpA1jpBumjGGD1A2qgNFpEJtSBNpNacTaQS+r3ZqXnblB2pGYrLLGaMnpvjBbYDoXuE6gmOheAoP35KuQcioDBINhtCigMoAI/kBc5BQwWxSNFsLSp/P5rNQIIktHdNm3TbnMVJrYMRIAk9Q1AYIDaIGFZgddCbjXHyNIeECSawPrKL8YGsmKH/RtGB3hipDaDuGRjG//6TYUS4SYhZ2gtyj835SFo+X2oawWwAJZLyRfVustSuZe+FmauukLX58afLq7It1fp2XvOivqFKS1UXjD3QgDkf++6197TE7alrPa0JRzvq3uXaQimoHjArcAVMTBkVJsTg8crOz+iPdbxQVDMeCqznDO8KGg/Kb5JdXW0/YZX8IMxwUUZaycPCdNRJt9tdljFDDxVaP+dFeOEjh6oulz5a9x6ac2q1eSDRzouicul/90OpvWD/iR3cMBRNut/Mn+r35O7XyRJD7XbP93+mBVNJUPS94mfH48iITF9veP3FJlDEkdRv2r3Ja3Du1Vh8WUZvrXGzLyBTsn2RoEgn3ZQTq8+f/AQ==","+v+x0Kazznt5c7gQrZO3jh8hDeHQ+InyIR6IWg5mKgo=",6960,30], - ["plugin-workers-core","./config","vFhfb+M2DP8qQp5aoNcPkEuTFZdhuAJrs7bbHro+0Dad6mJLnv4kzYp994NkW5ZsOUu64p5aiz+RFEX+SOXpafKFs5yuH9IXLGFyEXzeo9SFmlxMMswpwxueSPfxJxcbFOb7hif1Jv//r6zSyl9wBn7TqHEl+JZmKPrfS1ADjNt5jwqZopw5c70Vh3xIoaBs7XDBd2MjWHM7H0Fu3Lbuo5PziqZDR2LLjZ2YyKmro1gvrlCUVErKmRyV/AGFRif9RXBdhV+NUW+lZ+uGJw9ci7TTIt0pgu9AkwxvNVjsGTBB61kwS4/7CifPF08TyhSKHKxY7Su8LiiYA+eapSZC4b9j6C0ICknRXx6B+FpGdo5APmbnmItHaB+Dn7Q1iDvxr232eKdVpdUFebSXS66IZhvGd2xO3kgFQuIZNYJpAzifkmbLZyIhx1UUMmSR1tCc/Nu4dhDk3PDSgXQ0dPaNJ3LaYs6n4+gmVc9Sa2xKhvl8Hlv0I0wciZE3IhAyzoo9odmUSCUoW3/uFpUp9kVEwKDEyHKGMhW0Mv7GdiFTYl9xylREKG2Rtc67uvYR6QtmusCYakVL/IezURk398l0maDwJCW83qMSFGVEWAnKBVX7iEigEvslFhATlvD6hbNUC4EsjQEqFJJKNSUJ5wUCC0UtaS7aUAzo1D8arA2wSRffCVSQgYLFlNw3a7N7TLnIZnV8LlxhzMOD1cRu9wUkH9yjKc/MHaCrgbBhBmmfciZVB6jrJCytmZPOW4VB6yRX4XfD6UOgEUSMR/pwz4GBlnlQOb2Q+PUDIn2hW3zky2QRu9nN1m1ewt5c2iAvsuQ/ISW8/vyKqTaYDmDjXx8x2pl7h2wvvg1dMDoEYetOPhg5/LOnB9O95Bm6FPVcjUws4452nnRTzI+mLxOstijbGeB97NYNFQPI76KI2aZlxYX6FaoROYh110C89XSXxZl4e4AZ6r8BMZxKTFWh15R9jRqnrKAMH2yoY/IfzPP/k8zHu0SfJnu8/YG03e8oXpX1h/5eiXViRwexCX+EFcbeCX5lbraGzKJ05wt8l0efGMfww0hu+i4xNHl34EkSZAdkR4N3gio8Gl0X4ZFuaHY0Ns9ptAXVHNKNC13bPqA3uHkPbB9j5Mr/atpx/yqczL8CS83R2rdNYTEddpyRMWUsBz14MF17iTZ4Vvbyq+e/q5DeiDoWInmwdgavUz88xuMlFTGSA7kZEf3tDy/T4UTUh97Gu+Dhbh7E0ltfmyhFJdFpcSQO9diIrwpZJskKhKJQzO5Kqma3nN3qojC6ZsGeuceJ5M066I/EtWOLKbkWAvaOMwZpIE+YmPpdfCwD2inBk5u36wrSDazxE98xzMwF6FRpAYXtfCXYQYPkXJCd1WJuxOTRFgqagcnyS/tjkXlimlLQwqhJ9gRIFWqu1dlnLwGlsKyU2bq0T0kDR/HJVqKJGgEToMu/2E8VCChtIM283poHrV64yRXCk2+YKmmgtXny+IJEQomdHut/xndMKoFQ9px3HsiqoKp3zpKb7t850izXIR3zJuZMA60Bhx1qlBv3611aONm1M5VzUVo90LpsNiSYc4FttDPMQRdKEhBIoKoKitkhC80+A7EVSxJIN8gyUrWvHokFpoqLkyBONGbMUSaRqBRla+mnnHET3UvjPfiVy60G0lD7ML7vQUYh/uk0M+OgpcuhnsPSgT+u3RzCdqCKFzTdn4TtHG84jyyRcW/qNkEnkKZYqbrWw5LJaYFt1NsdW9u+j9jTZKYlyl5A61y3Tn8YsjvrnaBrygjPw3qyv3TRsDJlXwnaX8WOAkTqN0Y7Nh4nFbNsnjmnHNCmXHjCNhu1xIwo3tSRYUcDvpw8P38HAAD//wMA","kk1GGH7uYCLUMPmY9WdmgmP28P8Cgy2lVJA48AP3zJQ=",6360,32], + ["plugin-workers-core","./config","vVjbcts2EP0VjJ7iGScfoChSM1EnE88kUW21fUjzsASXMiISYHGRrWb67xmAJAiQoCq5njzZxJ69YLF7sNCXL7N3ghdsd0fvsYLZdfR5i8qUenY9y7FgHG9EpvzHn0LuUdrvG5E1SuH/H3htdLjgHfxm0OBGigPLUQ6/16BHGK95ixq5ZoJ7d4MVj7yjUDK+87jou/URrXnNLai9V+s/ermoGR0Hklpu/aRE3lyTxWZxg7JiSjHB1aTkDygNeul7KUwdf7VOg5WBrxuR3QkjaW9F+V1E35ElFZ9qtDhwYJM28GCXtscaZ1+vv8wY1ygLcGJ9rPFtycBuuDCc2gzF/06hDyAZZOVweQISWpnQnIA8j+ZUiGdYn4JfpBrlnYTHtth+Nro2+pps3eGSN8TwPRcPfEm+kxqkwhfMCuYt4GpOWpXXREGBmyRkzCKdoyX5tw3tJMiHEZQD6WnoxTeRqXmHuZpPo9tSfUGdszkZ1/NVajHMMPEkRr4TiZALXh4Jy+dEacn47nW/qG2zrxICDhUmlnNUVLLaxpvSQq7lsRaM64RQuSbrgvd9HSLoPeamxJRpzSr8R/BJmbDnyU2VoQwkFTzeopYMVUJYSyYk08eESKKWxzWWkBJW8PhOcGqkRE5TgBqlYkrPSSZEicBjUUeaqy4VIzoNtwY7C2zLJQwCNeSgYTUnt+3a4hapkPmiyc+1b4xlvLGG2J1eRPLROdr2zP0G+h6IL8yo7KngSveApk/i1lp46bIzGF2d5E383XL6GGgFCeeJe3gQwMjKMuqcQUrC/gFJ79kBt2KdrVInuz945TUc7aGN6iLP/hNSweOvj0iNxfQAl/9mi8mbebDJ7uC71EWjQ5S2fuejkSPcOz1Z7pXI0ZdoEGpiYpkOtI+kn2J+Nn3ZZHVN2c0AT2O3fqgYQX6XZco3q2oh9UeoJ+Qgd/0FEqzThzzNxIcTzND8jYjhUmKqS7Nj/EPSOeMl43jnUp2S/2Se/59kPn1LDGlywNvPSNvDGyXosuHQP2ixXuzpIDXhT7DC1Dsh7Mz9wZJZku5CQRjy5BPjHH6YqM0wJI627k48SaLqgPxs8INkGs9GN014ZhiGn40tCpa8ghoO6ceF/to+YTc6+QDsHmPkTfjVXsfDo/Cy8AgcNSd7310Kq/n4xpkYU6ZqMIBH03VQaKNn5aC+BvH7DhmMqFMpUid7Z/Q6DdNjI14zmSI5UPsJ0d/h8DIfT0RD6Kf0LXj6Ns8lML5tmPZjqq0hE1K/l0Bxg5KJPAmKTiRY39lcJyXJmXMim83wiY8aea7IBqRmUC4+V0wvPgn+yZSltbWIdJYBs5LvLsBwsG4CW83JWynh6JlnVEzqgrlrOAtM1VE3awRy+wLeAN3DDl+KB465PUZDtZFQuvuzAjeukEJI8uCs2HO11XiAkuVge+WV+8nJPlRtQxlpzWRHAqSOLTfm3OOZgNZY1dqqrt2D1MJRvnT9bLNGwCbo1V/8lxokVC6Rdurv3IPR98JWHBHZN6RaWWjjnmzvkSiosLfj4s/FA1daIlSD4H0Eqi6ZHuyzEnaG6ANpl5uUTkWTCqaFNoDTAbXGbfiNlpFe9ta7KoSsnB3oQrYKGRZCYpftHAswpVYEJBKo65JhfspDq2chru9JBnSPPCd193ZSWCLVQl4E8aIpZ554iUKtGd+psORsmOjfK0/Bb3xttZD2ghjn9ynIJCTcneF2qHSkO7ZzWjqKx19ap7A9qBYlo8eLsH3gLeeRNXIRzO426QQoxVo3vR63TMFK7LLeaRzcEHCGTluZjigHCW1q3QX9bMh+r58l2zFORBH3k/u9jMWdqYZG0P22dhYg0b8p2nH5uKiZVftYumSDruTiHXbVaBTmRIu2jyw7WvCr2devPwA=","5Ov8yd+yEEhw1W/KIuynRAtcOYjAGnOvMLFpBcLCYaY=",6430,32], ["plugin-workers-core","./contracts/v1","vFnrTis5En6VUrRSQOoE8TeE6IyYsyPY1Q6C7DnSEKR2uiuJB8fusd0JGcRqH2KfcJ5kZLvb7b5xOYedXwl2uS5flas+h7u7wYXgWpJE3yYb3JJB1Fi4QZUzPYgGiUSi8auQDyhVKTOIBp8fMck1FfzvlGmUyuvxGzeYCJneoMoEV9i/409eieWFtXbJs7xy7Eosf8QV5dQd9eo618NTTc+uxPKaHJggqXOgsbSmSsuDW5xLul6jtI60V0KNxfrPufai/87SriDc8leqN5epX7+9/fx5h1wHX/3enKiHzsi7N2rnmrGbtUZQwZL3vswzyciSMqopqkE02Ley//W1lcttxnCLXKNsb96IXGPP8r/Itm/LBdPh0JfT9olwzZ6Wg/vobkCNRyuSGBv6kOEPjFqNq5wnBsxBNNgRScmSNSWC5W+W6Pv+ESa/3ZMQk3c6/s6Aqu+1XEC990znrh7hHHL+wMWeRzC3VVutzOAJMiIVHlGzMSkEjidQHD4DRVZ43SnS1epKozN4Lhx9Rcy7ElQPdHbL6bzoMQrwUSNPFbT7Dpx3LM6OjifQq21m2rPgSkN3M24GOr1BkgrODlPX/6ZKS8rXkcd0NitD72nVcA5eR3mo7URXe2/50iNdqeuaBt8RUee0COPp0xA61Nd0W351ytZUfVSmmkPtjSG1zvoq7DvdNFRTE46W7yj32qXyULVHbxfeoVSHa76l9Bdw1+j+tsw09IUz/ztSXRKEd1VtnVW0rJfb3kg3t3iXyZfoScuBbuG6so+6K00S1FMNTdkXS6caYh3cCZ5AFmcgY/mackNvJuBcPKs2k5K1oFRUcDUpvQ9kpGExP0mRZ53bSWDX79th5oBscKbWXCkD/7jhBedNZVWi61Osy3DAIKdzs4iP2rvQk2qbI7vDcYcyAvsxc+s6l3x+yHBKS819pi1fnM5NrrzFPq4awQe1u15HrBtvkiwI8l/j+TVNHsqLUGWgr9y+nLYKzjL0rli+nLYrp/6eCE85cl/PcMv4+G/mhuGjnlnmeU2SB7LGkdhzTM1tzBOdS8JAWQRB5e5Kr4QsAvE3VI0H0cBxQZDWIqawrG5wqWJHGE2JKXV3wEiC3iDsTktsqjOcSCn2mIIWQDiQLGM0sYeHChjVaHz7VSwhc3kAWSRivOALPt9g5cqOsByBKmtLmTrYU4mVKaLg6ROj/KGZoeczsH2EagUGxNtE0kwveKxdF7wSyxgskzbavccGooRRM5bWyFG6kBf8Ez4SkzOI41irBbcp8/V2Dk8wxO0S01EqktwkdjiBJyj/uEzLJgnP8Hy24K6ofBTnPUzbN5Wj47MFj+N4wT85aGDeg/2e6k2I8YimHuYl5Snla5NBd7dsuIwqPcKSvypYufFkpK7zJaMJyHJs+rQEleRPgrS9SjXU94h5pU15i4RBqjho6iT1jfbNbnUcaztVF+p1ySL0q1jWsHGdeaQ2JMMKYpVnmZDaXSJTs2WCirJz5esuldHyz1qmRlqMmncCSJJgVmo8GGM+6bZQrTeOAISiYaFHIDITIWHsAEuRc3c1fZ207mE9/kKTSQoJoVsegKZGtqAUYQMxsRfnDHSQSZFgmkts6M4NpXw53V3SpCeF3qVblDuUI2VuMlqaWQaJW6o7EqRQ7mjSdE/1qVFv0POWOtVEPbxeqA2plyvVCNdKtV0cYXas7lp6Xkln+4ALNKpI28H3QbAXpKVrS3hKtJAHiFNUiaRLjB0lNPpcHyyhrJpbgEBfo17wCyKlIavGjETCIsgkJlQhiJvri0qZtSbtlYI//vs/4AJQEpWb8UKUDhJYDUMbS5AX72FxE439H1HSHaaQUomJZgdYSbEtp1Qvg3y2PnifF3yVM3YYUb5CKTHt9n2ZU6aBcoiL6huXEqMKqrFW8diMVarsqBPABDetiMCG8HREcr0RskEcNnQ7AaohCcAsYFzwDOXIeuAG6Imw9XKCUppmcchQRaBEOz1AeLrgsSesR+Px+DgGpckBlO1KhKeQEO6ILqSSrrSDz9j/RaRF57RV/fnRzrvmAPTabbPVmzYDsbgQBhJ/y7EYw/iobfL+gZi5aPNMaYlkC1uapgz3RGJU4B4BI78fHAQyMudXdJ27GCMXg1M5KibugofPGUNa9siY+TRxHqrLZL1rQWqLAHmCHUSEbs28gSdXx/00M2pTWHh22A4/cdTmDmb6xD3rRoXsKBESTzxXPNmdDj15cdFPXnrnPJX3o2I/hmq3HFlwAE9pO0+V/Men3VR+GvBYeykJcLIN5qOHNQLKE5abVJSEdot6I1Jls2VRVu8F1zL1FqwfAOqk+wE3XKO+EsthB4Rjt+UxMu+htApf9dOS3akfWL3h2+Cf+h9z/6+Ii5fisCIyw0YVhHelTKw3U/QuG3JvZXxT0ouH6Qfnvmhtkxffwq8Xwd3wP0JmyfDeY2WeClVXTOEo7E7m3xrHYTmUztl2GPTqaqAcx+UoYI6RuoYZNl6yJtTEZNZLhdGCK1E0vNh1j/HUfs7GZhAxlMVAoAqSDSYPBVNqd0QL2UrILTFN0b6FbLKKd6KZoFC5Hhfs48VRf1Hv2LVoLAcO31t1kEyd1iak4SkkZ3pUAN0z/INW+QyEc6HdBLE4xa1fEe6GZYsc3k8v9OPs6LiMrGadHUbumeAwtgVdTcSNUAhxN+4JYUwBMagU8Ad5tJi7Mr/8eQw3mDGSVHHvqMgVxKWLE5jOZ0fHcD6r/7ARW2phADdJ84Voh3+BbcFtAq9rtOoFNvXl9Nlc9jKFhIPIyG+5KYjapC8qkSQbD7Pz7s7avrcV6AvJMi/fMGqFccSFBrLg8ZF7X1W/VNrIi++G4VTMShvORfnx2PzwyzHRQirrsrFIJYg9t1AnErVHCHaUeJLX8TvQdG7OFjVsCmM8uL//EwAA//8DAA==","DR9J+ZG/Ac4db9f0eXC3wMV9NqmKQ3xIgp57sFHFEtA=",8471,34], @@ -499,7 +499,7 @@ export const EXPORT_SURFACE_ROWS: readonly (readonly [string, string, string, st ["plugin-workers-core","./schemas","pFRNj5swEP0rI067EtkfkA+kraqV2kujTaQeoj0MZth4cWxqm6TRqv3tFYEQIDYg9WaYN2/eG49ntwu+q/grpVxyy5VcF7Hg7Asa2rA9HTAIy/gaz0Jh0vyqULesF07CHx0JbPaYUxPtqPlR2Lyw7liPdWNRJqj7KrZoMi9fN9jL/Kl0lgp18mbfAxqGLrGjp/e5d6C3cBccUXOMRdkde87pWXA0QRhwaUmnyKh3bmPa5xaND+6p1PrtOpYamZLGwuAQzcF980vvlUe1DugP33Jbf8IKCplJdZIRrMA1Aw00avuGgdldbqvqbe5PyFEbejiiKGh+/f84hxq7AIMprd2Y+gR/BhV0i0dAvy3JxHhMXVGfoAkTJcUZTPmCfE2+PK/FqAt14PbhgCabw2vNu3wlpnSyNFZz+R6C1QVF0WMLcO3SAnLOsv9In9jDy0gM2ISVt/rAtUfNtHnnsU18VR2E9ex7l9PcfYWDU98fkwnX/9fUGG+zXFtwgiXXfhz05KrTa65vrU6Q41u4g5J89W7kw8t6ZHf5HFfU42t+hN6vvly9FQZiNATmkgap0mD3XMKHiiFpssxTEAbX/kBVYsbUIUfLY0FwRMETtEoDQ605JRCfAeGkdEa6x1VR6YLZQqNopSJjlFtKAA0g5JW4Wx6k5ZMrs9fIMnynmTpJSsDcuGoTpqheQGnmnqYC1Y6aVBV/ELOzyyqEA+ZQmMrFIMG2bFWN6LoEdWn0CKjVda2UnQk6koBfBWfZzFjUFp7X31zNh1bza+qu+X5hiyYbl9dHmcn62izltZdTN17PhZxec2h+e05G8Q4l5il4e/sHAAD//wMA","SlitkMiorQJbresojIUHVv6E6/zTQDBiOdiVAUm7d/A=",2785,14], - ["plugin-workers-core","./shutdown","io5WCs4oLUnJL8/zTcxLTE8tUtJBF/EvKMnMzytGkghKLcgvKkERKM4vLUpORRIKLkksSVWK1YlWKkpNSy1KzQPLksZG0a2A6VCccggnYyqBOx6bFNwbmJIIDynpIKHYWAAAAAD//wMA","5cFd6xVa8Q3Hgkojzelh+j/ZoO0gqKgHZhlCJRBZ9AU=",324,5], + ["plugin-workers-core","./shutdown","lVXbbhs3EP2Vwb5UBrZun+UmgWIrgdGoMmSleXAMgSJntayomQ0v3ghG/r0g96K1JRUpoAeSM3tmzpxD6uEhu5l+mHz+tFxN3s8Xy9XHxeR6urqbLm7nN6vZfZb38S/zxZ/TxepmMbn9a7W8nU3nn5dNhrJC0xe2W7RZnll0bJ6w2d/E0PugNuizPLsvg1dc00yQ2KTkVyfzymsmNwgssGI7/HaBjoOVODi698LH/aDkRHZAp/oYnF0bltuXR4cmmsNPukC5lwabOo/5Q/YkrBZrE4sOlkWgVPbl0mKBFil1/H/XmjzaQvz02u8rnBgtXOpSMjkP/6nvGCjs1kmKl9ln1B7kC7cnCR1RGJhgxM0Ex3A81RyEPA62cl2M4c7yTjv844m1ejsYJJyz1UjYze9jCLQlruniBWwveD9TOLbg2djBB8cpvS1PhXqDHgc7q/ainZgCPEMlgsNb8mKLo9dDuQKxZuvjedy322veVQY9qhP50rA7xoEf5/poxgbPYFEoJrMHz16YmevUvzpEarbbk4HU1UcrJN6h1axOJjn03uAOyR/C5/tKdxWegbiObDo0J0tUweBIoRH7A1AOUhizFnI7htEFvHmb5nUx3Jyv1Yo/HEJy+FLvkIOfuXc/xflUluq5vBsf0/vRXmM49fjAm87o6YLPCaHQhS+BC/AlgsJCBOMbvXIQxnCtaQPSoKBQgWHaoAVfCgImBIeSSV3GV374JTjPFayTDXJYo+E6wVeWJToHghR8Cxiw4dJmuohzH7/Uyblt/WQR8AyFJu3KPCJRM6oE1HgAau1L3XS1Y2LPpCUoFMpowsuvNEk4v9aaVOymkQEsfgvovAMpSKIxIqp2BYFkdx867r8dvAYW/0Hp3VeKRaGVdGot28t4RxJHrgmtA4u7yNCiq5icXhuEgi1U5d5pKQx4tDtNqWyk/7cwWkWhIrP2yUqNowUVbMpzsMaCLabxSOGjQDZQpJQ+iY9HmuWr3wf9HRVUpXDooNYWFaz3IFr4XxxEzux081wy+4gRXyPzhGpQfTDpRu22vyR6P/Esz2a9Dqm3SElpV7GL/3fdtYvdx4l0JJMhOpRE45qp0Ju2fOfUuuQo+qFyDpqkCSriNeboTHswSRQvIvaXIqJFHi2OJuejD64SYoWqP3EgLHZimcvs8fFf","Yrlck0vJQoL3pIP8pChVHfcbOIbeJs9g9se7cd6o20g=",2298,14], ["plugin-workers-core","./state","rFTBbhoxEP2V0Z5AonxAUqImJFLVtKqU0vQQImHsgbUwM1t7FoLS/HvlZSFrdtP20AusnmfejN+88cNDNuZ14VDw5gl1KZbpaxF/QzbIxh5V58ERGjNpLKQJfSlFxf+PzKsmfoeavWki30RJmdBNvF0u0U92BWaD7HaThEbosw3S6udx8JDJrsBLZ1Wk+1/f2qmQYodK8JZoMII7VIbJ7d6XtCLe0kXNAN1q/injVGYYQR3SimiqDiPo4QZJztrUfRhdwIataTHs5/NP7ewHByO45rWy1B5oGt6Y6jEnnXSlNJzOG55BMwXxpRb2Pd4L1nWpc/A1BtacQRBvaXkOAaWpS69/dtDvHHQ1jQT6WWKZIkGUlwQpPC89hpBy1W5IwCWmmc4GuXSuhV3tPvG8A51wYXUHvhe542DM3qOrbnvSXUmtyhXYLl3D7dr1QUdxg62ra4eKyuLmqbAeTeMIXg7m6FrkTu/FlTucL9gftLa0BAVb9iv0gAci8JWJh9kgzYmz/mvGsZ3oumrdjA3a27UlJexjSLVdlja8QgNqIQlTqDy7rg0XYvyP7moQhD0asAS392npSFKGtDL0tCImq5UDU20PIJXrfpop+406aXpKU6reLjQgDJIjvJI9f3CWVu2NfKn4oUcs4NHgwhIa2FrJQU1p9qtesBlg0KpAyJXovD+ESY5QeNxYLgM4rvqNybZW1TlVVI3kCCVZpikJw6xmG8A2tzqHrTVIdVQt1Ky+HJqr3QwWFp0BRQbmnlc4Jc20YL9WpHHfZUydnbxsdxgKpoCzOF7xSkt1yyF8D9EakuOU4pdDCFx6jcALEF9KDivEIjT7qceoQrBLUnOHR21r6jia2/t3c6WjU049UrHsO8Wnwllt5fWls0yn/o37HXt8w7xhmD0+/gYAAP//AwA=","r/koNoqzR/qIwsGL4SxFd1EYbBo+QMWqLifLCKPaEjA=",1994,9], @@ -515,9 +515,9 @@ export const EXPORT_SURFACE_ROWS: readonly (readonly [string, string, string, st ["prisma-adapter-mysql",".","xFVRj9s2DP4rhJ8SILgf4LTJDu0GZOi29q5v1wOqyHQinCz5SDmZF/S/D5KVRI6d7IA+7CmOSH36SH4kn54yZUqkD6IWa6WVU8jZLPujfXzVo2fWGJROWfPBmlJtsln2mRRXIlj7/+4LUTuk34R0ltq+LeJgEZ1GrcqalSlt37Ziq4U3fcIdXrz4V+0N3D/80uDl6w/IjXaP6PrHX0kYFuHdUVqJ/fjS8+wpKxsTzrJZpoxDKoXEi2+pBXtahCUSmoE9/XZtjfdaCb7h89/fKTEYlHiyQ2JlTQ6NeTF2b6Y5jNX8hDe0wgG4qWtLjh+wq8jvVhnOYW2tRmHgxwjAhXzgAFvLzogKlzmwI2U2c/CgyxxMU62R5tAw0qWHYN5bKpKjYt1DsPpR/YMJilMV2ibFdZqXpwQEtqFMkFQcDiCtYUeN1/BEBtL5eCwzsJ0olnkKEZUynQOhKKzRLdRkd6pAyuFbVrX8qr9liVV00vtTVHgOSHaPTaZ9wic9wa2+OxfhRgfC4acIvvouexD7SfjoZSA04NQf2UoxvhtrxMUc8G+UjcO3YXQ1PN96lKRqN+Hwc2SV+O+sKhZz2KDrzxaf0KuDZw7sBLmk7SeqN336le5PpvGAhyNmMYdCcW0ZJ0O+jSmQdKvM5iOpHdJl/W9VNgbh+1RusRK+XkmLvKF5/SS6HiC8P1K5RiRKHw5QCCfWgtP3Q9tEnr8SWVrmMEGiHMK/KbxfgM/C9TiDJnx4r/oMK2jDpxyFv1/bGjmHeyLRvouGxXXUkyRD7+umMj5xKWZ3Ogo7B7L71FcLdivDSG51HlbXHx/K4yfb8mSNs6kn2OE++//6WNqqUm6kBchqvRbyZWB6UxbPEmwYP2+FcbYKHFKl+1258isSGGmHBDLdcyXZCuK+jHm9y2ZZbxfaEtwWj1MaC78hvnyKcN67+x/tHqhbJQ2FfjpWx3vGsQ2lJZCEwnnBdMFF1FhoUIadMBLDNc/o4vWjH6FryGAB6xYOv2hlXqQtbmyMu0jzRwjzTLlAJ5RmIPSDAwtwNoJ4xyTpcJqSoP2kYBBSYu2vrNuQqB5Bf/s+co2JCNEnBL1L1+68FTUO8Mby4+90ugRG18tCz53EPqhedYk8Z9ElEY3l8vvldvju7x8VJ5itVMID7ZXbghioI8G/y56f/wUAAP//AwA=","u6bGo8FpRhr3rlJ7oVAo8WjWhWoPW8mmd9ZDmC4qMyg=",2955,13], - ["queue",".","1Vptbxs3Ev4rc/qiFU6W0V4/HOTYjs9Or2mSxknUBLgogKnlyGLDJbck145i5L8fhrtcclcrJy6udzggQCS+zuszz1B+/36UG2QOL5lhUqJ8VWGFo2kz2v222JbIw9AFMv4cnUPzGpnVqjeUa8M7Q2+cNnipjRtNR0/U73TKy9IJrexoOnourEMVv79Aa9k1nmvl8JOLA+HyX1j+Ma7uyB6H/ddzrdbiujKMRp8Yo00yozC/Z7h3UmeV/3KuOYaBn5jiEk1nUe+AS6NvBMd2+i2TgnfksmwdRulgb++e5tEH8fSbZstLs9gYfTuajuLRr9FW0nWG3uQbLNhoOroVbhOHRx+m70frSnndR9M9H922xDMpGN0rlEOzZjn+SZ8NrtGg2vn8LWv2rUdVFb3pRLt9kjxkSWpGGEiuRws4hkp9VPpWnWSKFTgH64xQ11Ng5vq7eZiczCF1/qPFSXJNc/J//sQYYI8W/cOsj5w59GPp0eLE3/R9etNO9NbXUQBBHzyi/KlVoQ8oHUXhDgwyrpXcQlFf85QHWY/inIeaXxI9krmSbaVmfA6LZHCDjKOxc2gubbX3/58kKzlKcYNme64r5eagqmKFJpnHGuf4mRu4e82E3DNlvFHmO2ZKjw4AdDqw3082pm/n4cse27bI3DcvK0tUPDPeDPMBb5xM5nBpdCEsPrrRgp8cgRTWZbqGptM0HMK6cD6JWxqdo7WZwcZSc7j3Njg+6d43hfuuqv1xcgQcS7fJdmd6JumWJR9fDhV9ecoTM3OUbHsa3V0aoY1w6RBHXpVS5GxncxNcp/uiqydSpzDCHeRa5ZUh5OpIoKV8SntumHxhO3JItkV+uW/+RlixElK47UIUqCuXzBXs00Ud3wLTPeQgCvp/GpZjxwy4RpdvfDIk41ZcKyZP53C20sa98d96WnbL/bcm9kOy74I5fEiKs/xjthvcioZjxCUsZHctfnKo+NvWwFkhpBQWc614NGeyb6W1RKb6EdBF0D34p5gTN/ganRFMzqE56iiYIGvMOIdFmjHdeN9VIWcqR3mJigt1nRm0Wt5QigpuY7qlORlEm4JW536zRAr+LEmk4Ei/kS4aMEEr+AumtkH4eOdDdCA8QpVtano2h44p8jrg5r0AvB9nOim5e6F1utwJnZ5Pk8DxXvQqnCZ+qwvA6VAFGAL+b8H7IZLcBKmFzuAwyvjjWtoE+8n17poezY4yDRNuuAOOSj+7iaBOFuEiQfkjMGy1Eq74PR0rtXXXxqNVMzYk9R5BU0rfm+qR+54CUeyyYfinc+gw/iNgldMXwnrk7Drame2Zc1iUrguyzmwveoUmby0VLtixHUF+p653Yqit9EfAhWUriWeV0wvDcqGuo1jeaESUu1rAHVx4v0zhNXljCq+9D168msJlY/khe+/2OS3lTBseopv3Esw2c2Ol7/c5NcWM/hnioG3Q98cTPL1Hjl5a7fRj+3KqbdJUA1lpGMTJi10w0KpnwV54t9bs9YEPNmgLiLvsJCA78fqO/gMegDuwVU7ELlGCM8dOPc/2YLUPpYYMfgclMxYzocrKdTqMIx9Al8OzgybqtrxftdBQ2ditDpN0fqgu+lW+MTz3PRawYPu6O/FikZYeoQlGyHpknZyVrOYPs6VaqncbVCk6wwl8N4Vbw0oLbtM57UfkYr0d2xb403BfqtuNyDeAikDAQlFJJ0qJQSwLTsMKgyTI46VObkErYEDySVzWwdzUWTRezMVGWBC1SEQXiwIVRw63bEsH17e2+gpbwFobeHr48mClK8XhVpuPIMVHnC/VAfy0WFzC2eVTyJmUlkYumGMrZr2+xE5p7EchEXSJdTGyJMaP2sD55a/JoT7CrQezd7iCd9p8RGNBKOuQccgsol9Hg8zkG+Ewd5VB4Dq3k9lSPSakluTCRcCP0EQCdbZL9ZjUKoAa5xoC6o+kIH2wpQfbdl3DK5qleVpQaz8m3l6qxwZdZVQHurz0RLagoaZMyq2PiRK5vzcx9GSpHuMnVpCvr66unF2qmHw/61XQ6Q5+06tI1uDLUnHMJTMIbS41wfGzXsXoj0fsMCPyyOEhtBlQx43bMNeGmYUfYgSmEbdU5DfX7DkefFaJV59k49/0yo6nXSozhx/gyyTIUdO45OqqpNhMBGAOGDhR4FKxWyaa22cNo2R2q/I273263y0VQL101zaTIzg8hMUGLYKpFAjV+mWpvFhXV1cD3ilKbaglGlB52qvPX2BtdAHjxwqdzY0o3aEXeex1TuindmLdNKfR4U5HQrnSfJv63lvsHYWjLTGnzS3X+RbXDFx4ko1VMuqdRebr+OvvUxoKN/VI1awmIdF4o+k+eG0zxGcUcbED3pCxGlcvmxYetCFDUtJ0V82X6rtZy3cgE2tgN0xID2Q3gsGZLYXByVJ9P6vZ0X1r/jbzTAqevYWM45pV0k2ByVu2tXHLhHSHA/iVgrNu8II6a5A6Z/LwzavnwmG67tnbA3oMILeJNZ3fMERqwQrt0IPp5L+MY83cV+ArdR+JcEC1fahO/kvzQHWEVt5/gUPaTvY2TSQwxYHXvMrnc12DLOG6f42ki3yBfar8uQkGMUVlkMKAGY58CoW+QU4HXDx/NQVtIGeVxZrU2G8oEJAJtUZjCJopWWv6MXmwvet93hbNR1oZrBXN88cctMObWy/dg1CfWwD6rDlhTg+7IlX+ClLtVBqLij8pmJBZBKlpB6QGq00NTI0aNauDY/g806vfMHeZxxvyyRw+z6jdyd6PkW4ZT2FsCzv+MPH4Q3d+ntX3ZJOZX5LVU7UIcTKg0TAoRgPsgl9HyrZILdo8CLGc5boohcQDH8h/pVLiP0WPT7rlKjwBJdoGLYNy48qiCW6d5boYJ8qNf0Ip9V9o6MuAXA1DflCJPDxsA1VYWFfEV0gyTpNi3e6Y+YQ5Pj4OEk/q/aHIxrBoN+i2q5mR+CQwUAFLasQb5wG5fmKp+Qd+KiUTysLtZpsUD6SKSdm+Qd9WH0jfRTdK+qNM5fkh78zXT9hwa4QjirFqfoMAxlnpiFqucK1NCysE1qUWJE0Anh6QvDTiWtCzaTGAKCTIC6FEwSTwynjlLDX64LOPYGFH+EbEB98T2lk6tAkskr4Vu7ekjgNa4XRnUUDGpt8MJd4DaxNRlpRJDk+6otlS0Uud91iBjlFv6UGejjOaxHcbzWsZwnaWU18mkV8XqBwJcU426hOFwJF8XDApo9OKyjoQlCL+ACINXmgLDCol1gK571KibUj6RixbFR1Dfb1Q9Eyp8LqmAFEPL4e/wKEpvN9SRwfL9J3isciLFjvONqLrf4FiHbRsLz49hWIRV4+mo39QM6aHL4nZUpUUkcgbYwdeF456w9Yot+0rRpKI7Jp8TXQ81rvZUr32xYrsb/zbA9TA3rZzeg2OHkLqkNlbOJ1O7qxvalcHCYKHkrWxWsang0aQe4pk+uI1rRlB/+XhG6vjQHUbeMW4j9dTleye1n/FSEpZY+LjjgZZ5/oWewlsCcfrPbPmNahBbzpOS5xJfZ2N/UHz8bQ5fkYBS7u/AEqL3Q2eY2XjhqQlm/yE3xUw3lfaIe4YymXj+g6NfDLwLtjd1WWLf5Dq7U9HXwPvYbrxrzg6yUEAQGE++NpK9VTUFvtzcqBeE4D8jHNRv0G0Qx4PSZSdhIkuImOSChaerhON/W/k9p5s6r14/l8lVJCdw3Ffjz1p1bTIyR809Glkh+jswJJ3hWdt9KBKqJhQhyYQfAlluQPbonXLXxLHbFCWhNykzrkuSuYE8Y42aikotdugCedKsTKM3upaxqUtwlX9sju5og1LdRWfcydX9WHEvSw1Qo0KFgumnMj7JdTviinfVNC2oUxELwTnEm99g6EDHwqkYzbYSzZ8rWT1o6zTXqxmzx9Nq2Y7ND8uxW7Haf96F9PlXfOWF3Z4w3RavH354dMhxbM9GTLtPYz/DzOm+TuQVOij/Y+PzbL4AHnfZZP9TWKbjMEbxz2LZJR53bT0r317Wpx+kxOT3cN1aHZ6z4U9bdrupe67Bjqsvtgh+T98+Dc=","ky4OD1RhMc0GeQU++ouS7sIY7LAUdddvNLpbHxriyOc=",10713,28], + ["queue",".","1Vptbxs3Ev4rc/qiFU6W0V4/HOTYjs9Or2mSxknUBLgogKnlyGLDJbck145i5L8fhrtcclcrJy6udzggQCS+zuszz1B+/36UG2QOL5lhUqJ8VWGFo2kz2v222JbIw9AFMv4cnUPzGpnVqjeUa8M7Q2+cNnipjRtNR0/U73TKy9IJrexoOnourEMVv79Aa9k1nmvl8JOLA+HyX1j+Ma7uyB6H/ddzrdbiujKMRp8Yo00yozC/Z7h3UmeV/3KuOYaBn5jiEk1nUe+AS6NvBMd2+i2TgnfksmwdRulgb++e5tEH8fSbZstLs9gYfTuajuLRr9FW0nWG3uQbLNhoOroVbhOHRx+m70frSnndR9M9H922xDMpGN0rlEOzZjn+SZ8NrtGg2vn8LWv2rUdVFb3pRLt9kjxkSWpGGEiuRws4hkp9VPpWnWSKFTgH64xQ11Ng5vq7eZiczCF1/qPFSXJNc/J//sQYYI8W/cOsj5w59GPp0eLE3/R9etNO9NbXUQBBHzyi/KlVoQ8oHUXhDgwyrpXcQlFf85QHWY/inIeaXxI9krmSbaVmfA6LZHCDjKOxc2gubbX3/58kKzlKcYNme64r5eagqmKFJpnHGuf4mRu4e82E3DNlvFHmO2ZKjw4AdDqw3082pm/n4cse27bI3DcvK0tUPDPeDPMBb5xM5nBpdCEsPrrRgp8cgRTWZbqGptM0HMK6cD6JWxqdo7WZwcZSc7j3Njg+6d43hfuuqv1xcgQcS7fJdmd6JumWJR9fDhV9ecoTM3OUbHsa3V0aoY1w6RBHXpVS5GxncxNcp/uiqydSpzDCHXDDhFqIAnXlXtjkxlyrvDIEaR3RtJRP6bAbJjvLvQ7IL/fN3wgrVkIKt20uS+YK9umiDnyB6R7yHGXDPw3LsWMfXKPLNz5LknErrhWTp3M4W2nj3vhvPfW7POBbM/4haXnBHD4k91n+MduNekXDMRQTerK7Fj85VPxta+CsEFIKi7lWPJoz2bfSWiJT/dDoQuseYFTMiRt8jc4IJufQHHUUTJA1ZpzDIk2lbiLsqpAzlaO8RMWFus4MWi1vKHcFtzEP02QNok1Bq3O/WSJlRZZkWHCk30gXDZigFfwFU9sgfLzzIToQUKHKNjVvm0PHFHkdcPNeAN4PQJ1c3b3QOl3uhE7Pp0ngeC96FU4Tv5UGLZobvEhDPF1Ql47TodoxVDK+pVIM0esmii10Bu8GYcgf1xIu2E/Ld9f0CHqUaZiqE0Ci0s9uYjkgi3CR1IcjMGy1Eq74PR0rtXXXxsNZMzYk9R5B02agN9VrC3oKRLHLpjc4nUOnVzgCVjl9IayH1q6jndmeOYdF6boo7Mz2olei8tZS4YId21FN6DCCTgy1HOEIuLBsJfGscnphWC7UdRTLG40odlcLuIML75cpvCZvTOG198GLV1O4bCw/ZO/dDqklq2mrRET1XmrapnbkCP0OqSan0T9D7LUN+v54Arj3yNFLq51Obl9Ote2dajAtDYM4ebGLFlr1LNgL79aavQ7ywQZtEXOX1wTop46go/+AB+AObJUTJUyU4MyxU8/QPVjtQ6khg99ByYzFTKiycp3e5MgH0OXw7KCJus3yVy00VFd2y8cknR8qnH6VbynPfXcGLNi+7mu8WKSlR2iCEbIeWSdnJasJxmyplurdBlWKznAC303h1rDSgtt0TvsRuVhvx7YF/jTcl+p2I/INoCIQsFBU0olSYhDLgtOwwiAJ8nipk1vQChiQfBKXdTA3hRiNF3OxERZELRLxyaJAxZHDLdvSwfWtrb7CFrDWBp4evjxY6UpxuNXmI0jxEedLdQA/LRaXcHb5FHImpaWRC+bYilmvL9FXGvtRSARdYl2MLInxozZwfvlrcqiPcOvB7B2u4J02H9FYEMo6ZBwyi+jX0SAz+UY4zF1lELjO7WS2VI8JqSW5cBHwI7SfQD3xUj0mtQqglruGgPojKUgfbOnBtl3XEI9maZ4W1NqPibeX6rFBVxnVgS4vPbExaLgrk3LrY6JE7u9NDD1Zqsf4iRXk66urK2eXKibfz3oVdLqD3/Qqsjn4slQcc8kMQptLTXD8rFcx+uMRO9SJPHJ4CG0G1HHjNsy1YWbhhxiBacQtFfnNNXuOBx9k4tUn2fg3vbLjaZfKzOEH+DIJctQ8L7m6Kik2EwGYAwZOFLhU7JaJ5vZZQzmZ3aq8zXuf7ndLBVAv3bXN5AgOD2GxQYtgKgVCtX5ZKi/W1dXVgHeKUhvqmQZUnvbq8xdYG13A+LFCZ3MjSnfoRR57nRN+qp1YN21tdLjTkVCuNN+mvvcWe0fhaEvMaXPLdb7FNQMXnmRjlYx6Z5H5Ov76+5SGwk09UjWrSUg03mi6D17bDPEZRVzsgDdkrMbVy6b5B23IkJQ03VXzpfpu1vIdyMQa2A0T0gPZjWBwZkthcLJU389qdnTfmr/NPJOCZ28h47hmlXRTYPKWbW3cMiHd4QB+peCsO8Cgzhqkzpk8fPPquXCYrnv29oBeC8htYk3nNwyRerRCO/RgOvkv41gz9xX4St1HIhxQbR+qk//SPFAdoZX3X+CQtpO9TZcJTHHgNa/y+VzXIEu47t8x6SJfYJ8qf26CQUxRGaQwYIYjn0Khb5DTARfPX01BG8hZZbEmNfYbCgRkQq3RGIJmStaafkwebO96n7dF85FWBmtF8/wxB+3w5tZL9yDU5xaAPmtOmNPDrkiVv4JUO5XGouJPCiZkFkFq2gGpwWpTA1OjRs3q4Bg+z/TqN8xd5vGGfDKHzzNqd7L3Y6RbxlMY28KOP0w8/tCdn2f1Pdlk5pdk9VQtQpwMaDQMitEAu+DXkbItUos2D0IsZ7kuSiHxwAfyX6mU+E/R45NuuQpvRIm2Qcug3LiyaIJbZ7kuxoly459QSv0XGvoyIFfDkB9UIg8P20AVFtYV8RWSjNOkWLc7Zj5hjo+Pg8STen8osjEs2g267WpmJD4JDFTAkhrxxnlArp9Yav6Bn0rJhLJwu9kmxQOpYlK2b9C31QfSd9GNkv4oU3l+yDvz9eM33BrhiGKsml8vgHFWOqKWK1xr08IKgXWpBUkTgKcHJC+NuBb0rloMIAoJ8kIoUTAJvDJeOUuNPvjsI1jYEb4R8cH3hHaWDm0Ci6Rvxe4tqeOAVjjdWRSQsek3Q4n3wNpElCVlksOTrmi2VPSU5z1WoGPUW3qQp+OMJvHdRvNahrCd5dSXSeTXBSpHQpyTjfpEIXAkHxdMyui0orIOBKWIP4BIgxfaAoNKibVA7ruUaBuSvhHLVkXHUF8vFD1TKryuKUDUw8vhL3BoCu+31NHBMn2neCzyosWOs43o+l+gWAct24tPT6FYxNWj6egf1Izp4UtitlQlRSTyxtiB14Wj3rA1ym37ipEkIrsmXxMdj/VutlSvfbEi+xv/9gA1sLftnF6Do4eQOmT2Fk6nkzvrm9rVQYLgoWRtrJbx6aAR5J4imb54TWtG0H95+MbqOFDdBl4x7uP1VCW7p/VfMZJS1pj4uKNB1rm+xV4CW8Lxes+seQ1q0JuO0xJnUl9nY3/QfDxtjp9RwNLuL4DSYneD51jZuCFpySY/4XcFjPeVdog7hnLZuL5DI58MvAt2d3XZ4h+kevvT0dfAe5hu/PuPTnIQAFCYD762Uj0VtcX+nByo1wQgP+Nc1G8Q7ZDHQxJlJ2Gii8iYpIKFp+tEY//rur0nm3ovnv9XCRVk53Dc12NPWjUtcvKnEH0a2SE6O7DkXeFZGz2oEiom1KEJBF9CWe7Atmjd8pfEMRuUJSE3qXOui5I5QbyjjVoKSu02aMK5UqwMo7e6lnFpi3BVv+xOrmjDUl3F59zJVX0YcS9LjVCjgsWCKSfyfgn1u2LKNxW0bSgT0QvBucRb32DowIcC6ZgN9pINXytZ/SjrtBer2fNH06rZDs2PS7Hbcdq/3sV0ede85YUd3jCdFm9ffvh0SPFsT4ZMew/j/8OMaf6CJBX6aP/jY7MsPkDed9lkf5PYJmPwxnHPIhllXjct/Wvfnhan3+TEZPdwHZqd3nNhT5u2e6n7roEOqy92SP4PH/4N","qppUFq0F/4ZOrR1Hw0wp0Q88ntFzxmvKJ/bnzBAqjks=",10771,28], - ["queue","./adapters/amqp","lJLBTsMwDIZfxeppkyokru3YmIDbJtjobezgtWaKyJzguLAK8e6oG6PZxgGkHJLfcX7b+RaLZLx59eMKvZIkaXLHrzXVdO/VOA5JmkxMUOLuPKUQcE03jpW22gmzNi1ZpouktBjam0LPJMQl/Wn/kwlRQYMCrqDmF3bvPASz8ZY2xBogNh0UQ/iA0nFQqUt10qvFZhBUDK9TQFlfZodHUqCtt6Y0ektYTUiV5FGd0CiDE+XBiQ6KYT+H3USyXV2xbw5CWDm2TWvOVLYjymBOWgsXjafBtzzMwe6maHidwco5S8g54MqJtnMUZy3JKIPxsZJDdVzSTxuR9X/7iVIZ1bzRnFQM2qgw4lALnaT3+pE77Sn5TZsiN0f6vvVeLAV1/kgohVAPTEUR+IwZgTM4u9Appl3kDNizUIfueDp7gN4cVyuj01l///OAexg7/rB1uXjia6WNt6gExeEt8NhYhxVo41vAu7VcfgEAAP//AwA=","DTIBAPwoCawR6cuFgaEudOJ1cr98Z01HDsIpfg9uvzo=",877,5], + ["queue","./adapters/amqp","lVPBjpswEP2VEadEQpV6BUobZXvbqEnKLc3BgVnWqhl7x0O6qOq/Vw7NYpIeWokDvOeZNzzPOxySVffiVo1ygpykyWd66bHHL060JZ+kyaP2gjR9b9B71eLakuCrTMAulCXH9JDURvlwkvEJGanGf3p/q4RooKKCD9DTd7I/qATdOYMdkniIRYuqhJ9QW/LCfS2WFz2bDLywpjYFxe377NokBXx1RtdaHlA1jyiC/FUs48cMbpCtZSmqcpnDxZHsMlesmwOjaiyZIYgT1sGiDPYoPVM1OCz+wGUO5uKipjaDk7UGFV0x5AdLQX7LttMei7PVTZmDF+ucpvaecf3JaP+MHFFrS0+au/WzIkJT5lAb67GJ1NTJsoRbY2vMpXY1R3Jo5ga8mRb96P+6F5WSEn3GPQprZaLBkHzPeFO+WEbqOO7k37CNomGGj6bOoODkDAir0nfzfg0afUaeYW40Oow8F6kZlVwjEDHwK15puMvSRN2mamLu8nVHTUlbbXZbWOzV6aRls1uOiwpqzM4UFxVU3n2jT4KdM0oQqmsvcGowVjUggwt5nJ7j8Tc=","jhRtuP0w8gyIEIwlgyfvBShVnHNG0r8z8PBehN3DI6c=",1052,5], ["queue","./adapters/deno-kv","jFXbbtswDP0Vwi/dgDTYc9q1G9Y+ZfcFe2kKlLEZW6hMehLtLBj274Pke9oBBfJg8Xp4dKjc3SU3xLJu3mdYKblkMT9/qdQI+2SR3PKvmmoaDR+NV+Lx/Im8x5w+CCv91tHwLaQl94u7JLXoQ6RhJbfHlJJF4mhPjviF30MVmKG83MBbqPmR5cBXYMrKUkmsHqYQLjdX8AdSYa+uTlXcK3T5m1Wf9/oCLKGnwXABmKppqJ2T3MThCDNhe4RIyWcsaQVeneF84qtQi+tn7LWnHwU6ylawE7GE04L0u7ImNbpurldxxuW6mbgbcjsJGP+feUOYfSRVcj9UHMUyM8tXcXq5ubqAbG5/bj7GwMB3UmfQTrp6RTUpHIwW6+bV60kqsa8d3RhfoaYFuZnTkRfb0AZdTvpc2hzRLMJKfpIRyX/O9gn5OM+NVzgzeZVqZshJT0Yxvr16w/POGVnTnEyWOkLttT/xwN+p3ueq7VYH/owymujlRD6Daq4n1/A4U0knjmlA9hIxRIzDjsGTRR9dpys/ep4s/xPX+AwEwLD+2U4N2JIxbi2G6sstb7kPLNCDMHUJFTnIUHGHnpZw25A7DkWEQQsCjyUNMWAYEConKXkPPpAYy225lQVYkQq0QAUntZIHwrSAsoUNKrFkG0sO9uKiofOf+Q4WY0kL8BI/sm1LYmgUoQ9gmBpy8RWqS2o7iRbkznxf0S/h/dD9UIjvBw+FIxcsPfbAWzuyo5RMYzgfJjUeHJ13K5EBwk5qzigDrstd4GoPakryi5DPUSrnNiojwA+rHd4BLwwPNQdmcGfp4QIOhbEELCMlsVMeTi40UkBrF8M44bHoXsps2UkIVLaRic4RwASarKRoR7L2Tspo72ZaAZ6GHPAxsBwo7KMi6/a4ZYkXT85IZlKoxNolPLSL/wAHNOq72zS+V9CZB8Pne2vyQqFAziw5v9zyO6WysqgEm17QUOHRCmagxyr8S/XrHErG5yBcBs5XfpkskvF3f/8PAAD//wMA","HLAfqiFyK/gDd3BqwnKsHP+YNNzPfn4d0ADJASJ3dk0=",1919,6], @@ -525,7 +525,7 @@ export const EXPORT_SURFACE_ROWS: readonly (readonly [string, string, string, st ["queue","./adapters/kv-polling","jFhtb+M4Dv4rRHBAElyazu3e3of0bbrb+VB0u9udFjMfmgJRbCbVRZa8kpzUF+S/HyjJtuy4MwMUiCVRFEk9fEj1+XlwbVXGk99eMdkMJmF0X1hmuZL1xGc0hbCDyeCT/LvAAv/MadkMJoO77Sdpdem+7tD//s6NjSUelBBcrq9TllvUPVOx8CO29j5apXEwGZBOlM3KPRrD1vibkhbfbDPxF5k3mAy+Mpu8sqXAu201+rRFaatBo8kNHzSu+Fs1+TJ5HnBpUa9YQspsmeO14IzE43mNK9Qo3fePyCeCme7ce9+x7ve+f0RP893yCqJ7hz1ssJyBu8Ez2KI2FAbLsnwGhdxItZNwCH5BGyJwUUnEZ0EMG9iD2sxgqZRAJtv6r2ZgrOZy7fTXvsERzhrVAXHnT83Rl10PmChwBk/f9cXJv+NCC8ewh9xBpD5D8IzbqxnIIluiPgNjmaZxWEaZNgONZAdeRUFIlDQO0kl51TbMgQS6KdJ2l2e5wAylNRDj/vyJIkGqrS4Sq/SI6fWHWv34DDbbekSDB60ybjCa04WUXK4jS9lSaUt5ppUQqK9mcN2eIf8StUVdPrZjoJGlSooS3F3+wTKsbjtay5UQtxT0LRNNNOvlLTd8yQW35RPPUBW2RyZjb5/Rao6mZ1Gj1eWvzOANCla+J3DP3t5bTzEtcsETB/evXKZq1yO1U3qD+jbt8XCLeqkoyHVI6yV8I83c3m2vZhBRVo/IDbL0d7QWtaPEqxl0Zh6UtudPl2eQtufjy62USmb5Fl3QKOq1YShNofFuOxpHu7jkljPB/9dd8NIdM1oSQq1b4zVK1MxiAO1t2lrNUaZcru+wbE9rlaAxPSvuarqT6Kmjb+6eybZswmSC4sGf214RjGfBzM6Znplke5olm9ZYdidCjhCK0z8LG1Sb9qEama1qWjuQjitaU8aqvBNc+2iZbavMC91xQGMI6I34O1pw3BOzX2+Jhn2TzDV3n7WSOOLEo+SN1pqkjSbbydpdqJI0mu9Jzmi1SsrI0pCMMRVvjpIv/ZFcOwpY07nAntKWa7yVtTXH4qQI8M2iTA1cm1ImN9zkypAVsKfrbJH+qKlw4xkE5j6vVs/AoI1EJlURDAITUN42R8+NqZGqreKpow+BFvtPCyKvzPSvh5heesC2zQ/nz9qFdTzzrt9a1OT4eV3eLy8dpClnj+JgmprWFwrmuo9RQs1NIzqBLDQtvdvjluWSCEAZlzgd358fy2ypxJQ1F4YvR3KddqbbvTYrR33s0VLV0TbYibBa46fC074heVPkudLWOPEI7zsafyOkMVTiXnkcho9WI8tcqdk1vXNbYatZOlLY6raPENA06wSCTt7Ei99tAKMSgluuCvOF5pt+6wyoC4yHPMPQK94wi+/0j3VJvJqB1QX2mdgwQSNb34Dha+lmXCf16EaUeEtVyARj2mO7Zl/fMa1I1mDoWPAOO5sNz2+7xsHBvRO+eL/BpRAUBlNYliGvIFFZzjSeMJmemB3LQeVU1em0af1krJMtWq0g6bUtAmanIV3HC9od3gwabaElprDSKgMmq7NrZSRLL0QsT9xtAxJtdPYt1mhH4wUwmcKCKCkcQvztaHaDJayUzpilH2BCwN0XYL7mmelczuXTKzdgXlmOwA1Q/CWdz4QofSQsJ007bl9hcYNSTR0I3TnVHZDu2t3IjrsvEMosuDSHUG3p3Numx3clFwxmTFqeGCioGQKN60IwTQY3IZ5Qu7Tl1M/M5QqZLTRCzjS3pTfxM6bcuIB8Zsslt/d/efXk7EeLWS6YRXiq2AdyVgrFUpcpc/kR3xjZBYvFwpq5bOB4X1Y79iGrqofdXKaYCKYRVoVMHCRCB1L1V5n/nTVKjniXIuIeNiEYFyBxd/xKqhVcjqJOZQbD/6qlGcJh7BSdnlYPTB+TlLqKuWQ7xoP+adVFVs4MMWNcDOEwgb2Xn8EvHz58iFR6knd3HRwybZ2hiXOVAyqnJ/ReI+4fw8Ul7OcSwO/pjxGdBnB6CteFVScs2YCSYIqERCfAaFKGWdRa6bl0Bi4Wiw4cXa9J98OOovgecKnD8LilFo1yJ6SesUrTzZMnmiXW5c21EHUiNQ9WsJRODWyMgj/QPiaa5xZylmwobNSXg6aIy7lcsmSDMj1ha6mM5cm0B4UZsQrsgdLvbhvcgIMngeFHida4E04322EEJbIbZ3Xt9Jhq6RjVt/srMzH91Fdr6tA8DwuD2gwnMPzXTz8PXwgp0qPvWvAEA/z80SQKFxDrWPfooA2xTGjO+sRqDBK3+vo7l47U3P6RP9ezpFoFfY6Lmv9tVHpf4DAeezDSNiVwSk85t3m6oYruP9310+GHcP5XSqfqyQoajSp0gpAxydb+/kcUX/hp+vM/x7PKN89nJlE5pt++ihCLRjJEfoMlBXzoDBqSZEiRjFmeeK6mhi6F3StKrwDwjRN6fG7Uf59cEcWUeLXBqeMJjSyhh7PveyBhOXPPG+7Ycy4fHPWiIUFxQs0EqKWhuusL4crVm+SVybXb4UpLc0a3lyOS5gbsK0KuecZ0GQmvKLdZXj9+SI5ZkIhpY2eRp8x6an8vZdZo77bfT5XNtkar29Ekhms2wOSY8BVPyMNAMN7NPgxS+2ZcOLZTF8rR83HyRDP//uU/w5eXCpGkL9ZEirzKINAG7YL+xfePvZMg7B6CZemsniWKPyw8tR4iMDvPqCdwThUyRQ0sJAuMuExEQYXWYZVExu85C5GvvmUbPQ/xDZPCXd2w9q1l+KdKINxjY7Bz46IeOtR7Bw4Vnl2PDJhxS+2WQz2TAY6YtoDY5fvofRGux7P+t4WCXyT68vJ/AAAA//8DAA==","CQZw9bhBXVBiq9bu9YKU59ssoQyDOHgGLaR+Cb+5HEc=",6233,18], - ["queue","./adapters/postgres","jFXBbqMwEP2VEadUiqo90zRp1d1Dpbabprl1e5jAJLVqbHY8JEFR/30FBDCEZitxgPc8M/bzm+H1Nfhl/maU0e9UlDUuGAcPygmZ9vuRnMMN3VkjtJcWeC7CgnEwt042TO42xlSIT5E2VU08Z8T5glympYtmdKcVGQnexq8B05qYTFQU+c57pNEVVZQR4jWW2PB7JzucKNBSfS1a5kSVE6rWp9wW9DSZLOEaMvNh7M5MQSWppoSMuE7wZDmFA0TWOOEsEssj5M2PsI67uAImjK3ROZQneMKEQnDCymw8TnClz3L3MRlRa0U8sCK1Wt8Xym1Rh2CyZEXs0Vvl1EppJflSJWQzGViT4H5BworcAEn7VKtISXXxsxAG7DCw/Cdh/EAixC9imWYh9JC5ZZksp15oxno2cMBC3ywhvo8bZa8gKut6QNzN7jEueqcEF4Rx7qG6dI4ymxBW1mpCcwW4siyFY9hqTTwL4baLeLsyKGpLpWyF7k0Oqsw6uvBqHbFHNHkHr/bQgZzYtBfrMj7KPEC8lIcbIHpqd1ZETChDoZFGlTzRXo4u75Ap24icW9jdQLJjn3UYjD4630ya0HWTttfmwfDpT4N+ax67HQ49wzQt5mFNa3mY3zCz1vAnjeJxbYN4YHS2IeLv+P+Lg3rjd7KwO7iGBUWW40l1hHEzmIrx0ziS7c79T0Jvi3AoFOO8rEB7IRO7r+qc2cGouPZa3DFsUWelSvUMDGHONlGOJl+cbjotLBuPvJVbq+IpfJa/gWDcPvMqw8vzQ3XZgJUj2vmMhTUu/5gboSTVKATLemJDirm2GIPkKV0G46D20dpyZWJlNoC1VCdFipBq16XSwCQZG4phlYO8NxK3cZU/irBHZVSC2l9SkRBhipXp6jFX5jvcaGU++sb/vAze3v4BAAD//wMA","d6HFdcHl5rmXABl8M888TFPvFvw18dijz0/cI9qKZVo=",2066,8], + ["queue","./adapters/postgres","jVXBbqMwEP2VEadUiqo90zRp1d1Dpbabprl1e5jAJLVqbHY8JEFR/30FBDCEZitxgPdmxp7n5+H1Nfhl/maU0e9UlDUuGAcPygmZ9vuRnMMN3VkjtJcWeC7SgnEwt042TO42xlSIT5G2VE08Z8T5glympYtmdKcVGQnexq8B05qYTFQs8p33SKMrVlFGiNdYYsPvnepwokBL9bVomRNVTqhan3Jb0NNksoRryMyHsTszBZWkmhIy4jrJk+UUDhBZ44SzSCyPkDc/wjrv4gqYMLZG51B28IQJheCEldl4nOBKn+XuYzKi1op4ICK1Wt8Xym1Rh2CyZEXs0Vvl1EppJflSJWQzGYhJcL8gYUVugKR9qlWkpDr4WQgDdhgI/0kYP5AI8YtYplkIPWRuWSbLqZeasZ61DUZl5UbMK4i7+R7jondKcEEY5x6qS28oswlhZa0mNFeAK8tSeIKt1sSzEG67iLcdg6K2VApTKNvUoMqOowtvrSP2iCbv4NUeOpATm/ZyXcZHIQeIl7K5AaKnZyciYkIZSo00quSJ9nL0cYdM2Ubk3MLuBoodb1KHweij882kCZ1fFD79i92/ZceLC4fe2Te3xcOaW+JhvvdnrXdPPO9xrdc9MDrr7fg7Vv6iUW+SThZ2B9ewoMhyPKlaGDczppgkjfXY7tz/JPS2CIdCMc7LFWgvZGL31TpndjAqzrcWdwxb1FmpUj3OQpizTZSjyRfdTaeFN+ORF7m1Kp7CZznRg3H7zKsKL88P1WEDVo5oRy0W1rj8Y26EklSjECzr4Qsp5tpiDJKndBmMg9pHa8uVW5XZANZSnSxSpFS7LpUGJsnYUAyrHOS9kbjNq/xRpD0qoxLUfkhFQoQpVqYr/whZUtU73GhlPvrG/7wM3t7+AQ==","merA0mE5Ij3OSxKusqhEwWy34ZoH3esXhaVqUsUCfmY=",2013,8], ["queue","./adapters/postgres-dead-letter-store","fJLBbsIwDIZfxcppSB0PAIwhjcskNGBwYxxMYiAiTTLH1VQh3n1qEaPdBtfvt/Pbv7NaqVlIsmNKY0IzIRHihQQmld1SplFs8Emts5XSDlNSmbJeiLeo6UrhRvtgCU9Q+IMPX34INo+OcvKS4FfdLLAMlkM4gg4+CRdaAj+Es3kP7g/X6QMTmuBdCdpZ8nLtmBdU0EsNG1WfFX3DnHqQhK3fNTTBjburvRryYreW+FqR9J5yfCc0Ze+ybx8wRvLmodNAziZpAabIQVNKLWooyr5FyKeCaVH7tATNhPJXgFPzUnA/wSr3e8H9k9dPTM8XBqf6P4wLriQwhObR1VaQKi/YoD6QgU0JeLFZzCfnl7offiSUR4dCsIQp25316CCnlHBHELF0AQ1IGamrMnUZfBsYjiNn/eHWiqeuWq+/AQAA//8DAA==","YGglyQ1jaQIhyLbyuHstPBjQOqQb40fr8qyAARB5ZvU=",763,2], diff --git a/packages/mcp/src/publish-assets.generated.ts b/packages/mcp/src/publish-assets.generated.ts index b9067f6d4a..89e9f2256d 100644 --- a/packages/mcp/src/publish-assets.generated.ts +++ b/packages/mcp/src/publish-assets.generated.ts @@ -307,7 +307,7 @@ export const MCP_EMBEDDED_DOCS_PROVENANCE: 'pages/web-layer/vite/index.md', 'pages/why/index.md', ], - sourceBytes: 3255674, + sourceBytes: 3259077, documentCount: 286, - sha256: 'fac2e9ae8649238e745dba7653ae153741b6a78ccc5a8ac767e50ff3e8f40467', + sha256: 'a7ee1ed1b2d7f4ce8b5ffb9af5f2f81de2a487e78a1c05394152968764c8e69e', }; From 2d7aa10448700b46cb1c2dbb4665044eafb60dfa Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Tue, 17 Nov 2026 01:01:43 +0100 Subject: [PATCH 15/20] fix(workers): own listener startup before later registry waits --- .../context-pack.md | 2 +- .../drift.md | 3 + .../worklog.md | 13 ++ packages/queue/tests/abort-cleanup_test.ts | 27 ++- .../workers/tests/runtime/seed-sample_test.ts | 9 +- plugins/workers/worker/queue-consumer.ts | 2 + .../worker/worker-intake-lifecycle_test.ts | 63 +++++++ plugins/workers/worker/worker-options.ts | 8 +- .../worker/worker-shutdown-adapters_test.ts | 175 ++++++++++++++++++ plugins/workers/worker/worker.ts | 9 +- 10 files changed, 298 insertions(+), 13 deletions(-) create mode 100644 plugins/workers/worker/worker-shutdown-adapters_test.ts diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md index 4dd9ce978e..b929dc3acc 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md @@ -8,4 +8,4 @@ Use Deno2.9.7 through mise. rtk/rg absent; plain commands. Scoped structured wra Public artifacts/comments/bodies must contain no operator absolute paths, hostnames/ports, session/thread IDs or downstream names. Keep raw logs private, public evidence repo-relative. Evaluator writes plan-eval.md and phase comment, leaves artifact uncommitted. No source changes, commits, pushes, labels/readiness changes or PR watching by evaluator. -Current checkpoint: original and listener-owner addendum Plan-Gates approved; S2/S3 portable and native handback committed and live-tested on all seven backend variants. S4 lifecycle, pause generations, bounded stop, portable shutdown-only refund and first-cause controls implemented; full Worker113/0/0 and static gates green. S5 standard runtime configuration and standalone default now implemented: real runtime/runner/schema12/12 and full core112/0/1 baseline-ignore green. S6 docs and S7 final validation/supported full smoke/independent IMPL-EVAL remain. No acceptance completion claim yet. +Current checkpoint: S2/S3 portable/native handback, S4 lifecycle/drain, S5 configuration/default forwarding and S6 public docs/generated consumers committed. Original and listener-owner addendum Plan-Gates approved. Real runtime/schema/runner12/12, full core112/0/1 existing ignore, full queue215/0/3 existing optional-backend ignores, original Worker/core-cause113/0/0; all seven adapters now have actual eight-Worker-replacement coverage. Final startup pause before/after allocation repair25/25 and corrected fixture13/13; final static/quality/corpus gates pass, public doc diagnostics unchanged. Full plugin rerun in progress. Supported one-pass scaffold.runtime, independent IMPL-EVAL, final acceptance/head checks/threads and cleanup remain. Four exact implementation backend IDs/mounts remain retained privately for final validation/evaluator. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md index 50f206391b..e067996dd2 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/drift.md @@ -16,3 +16,6 @@ Follow-up PLAN-EVAL cycle2 settlement: addendum risk text now pins the shared na S4 test contract alignment: older immediate-abort fixtures now assert the work window before ShutdownError. The held progress sink permits only the documented bounded TimeoutError result in the shutdown case; cause, once-only state/cleanup and claim-release assertions are retained. Relevant lifecycle fixtures moved into their own small file, reducing the pre-existing oversized dispatch test. Redis delivery fixture uses500ms total to include the native adapter's unchanged100ms disconnect grace, retaining all accounting assertions. Core retry catch formerly replaced an observed failure with a later abort; red first-cause cases precede its narrow preservation fix. No budget or acceptance contract changed. + + +Final-gate scope repairs: actual Worker replacement coverage extended to the five remaining adapters (test-only supplement,175lines). Native AMQP lifecycle made its old Fedify-only abort fixture obsolete; replaced its upstream channel model and added cleanup-order assertions. Full plugin validation exposed a pre-existing seed-fixture stale mkdir-event race; require current directory existence without changing the watcher. The approved startup/pause risk received a red held-registry test and immediate per-listener intake-state application, fixing the gap while keeping Worker<500. No architecture/budget re-plan, no unrelated production change. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index 008e851c59..a6a1829edf 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -95,3 +95,16 @@ Updated the actual tune-worker-runtime.md guide and worker/core/queue READMEs. D S6 canonical consumer refresh: gen:agent-docs-prose (site build/semantic checks included), gen:mcp-export-corpus with expected dirty prose carrier, gen:assets-barrel and gen:publish-assets all exit0. Only generated prose, MCP export corpus and MCP publish-assets carrier changed; CLI asset generation produced no diff. check:mcp-export-corpus and check:publish-assets exit0. No lock change or saga transport edits. A final S4 coverage supplement exercises actual Worker replacement on the five remaining adapters, complementing the original Redis/Garnet Worker oracle and seven-backend queue contract; it is a test-only coverage extension, not a design change. + + +## S4 coverage supplement and final-gate repairs + +Added actual eight Worker replacements on memory/KVpoll/nativeDenoKV/Postgres/AMQP, complementing original Redis/Garnet instances. Stable message/header/count1 asserted at every delivery; ninth execution succeeds; final memory queue empty, KVpoll pending/processing/DLQ0, nativeDenoKV no extra delivery, Postgres row count0/DLQ0, AMQP message/consumer/DLQ0. Four cases green immediately; initial Postgres fixture used the wrong private test credential (excluded), then passed using the native configuration of its positively owned backend. No new dependency: test uses the existing locked AMQP version. + +Final full suites initially queue213pass/2fail/3existingignores, plugin263pass/1fail/0ignored. AMQP abort unit fixture still supplied only the obsolete Fedify consume shim and failed before stop; updated to the native upstream channel contract, retaining connection-close assertion and strengthening exact cancel/channel/connection ordering. The reported Redis failure was collateral cancellation from that unhandled sibling failure; unchanged Redis test passes once AMQP fixture is corrected. Plugin seed-directory fixture consumed a queued initial mkdir notification after directory removal; it now requires live recreated-directory state before publishing. Watcher production code unchanged; no assertion removed. Combined corrected fixtures13/13green. + +Startup audit found a real gap in the approved pause-during-initialization risk: first trigger listener could stay live while waiting on a second registry lookup. New test red before fix; runtime context now exposes current intake state so each trigger/task listener pauses immediately after allocation. Worker remains477lines. Intake/supervisor suite24/24green, including that held-lookup boundary. This implements the original startup/pause contract; no design or budget change. Full suites rerun after these repairs. + +The startup regression now covers pause both before and after the first trigger allocation while a second lookup is held. Initial allocation-only state application passed the first window but left the second red (1pass/1fail); immediate transfer of every new listener into the Worker's existing ownership list closes both windows. One narrow onListenerStarted context callback handles ownership and current-state application; no generic lifecycle registry. Final intake/supervisor25/25green. Static/check/fmt and public surface gates rechecked after this final composition fix. + +Final queue suite215pass/0fail/3existingoptional-backend ignores. All newly added accounting/ownership cases execute against provisioned backends. Final90-file check/lint/fmt0; plugin doclint2 at the exact baseline entrypoints; MCP corpus freshness and quality:gatePASS after final ownership repair. Full plugin rerun remains in progress; no full-smoke/evaluator or readiness claim yet. diff --git a/packages/queue/tests/abort-cleanup_test.ts b/packages/queue/tests/abort-cleanup_test.ts index 7edf6417f9..5f11d1464b 100644 --- a/packages/queue/tests/abort-cleanup_test.ts +++ b/packages/queue/tests/abort-cleanup_test.ts @@ -49,24 +49,39 @@ Deno.test('amqp listener closes connection when stopped', async () => { queue['url'] = 'amqp://example'; queue['queueName'] = 'jobs'; queue['listening'] = false; + const cleanup: string[] = []; queue['connection'] = Promise.resolve({ + on() {}, + removeListener() {}, + createChannel: () => + Promise.resolve({ + on() {}, + removeListener() {}, + assertQueue: () => Promise.resolve(), + prefetch: () => Promise.resolve(), + consume: () => Promise.resolve({ consumerTag: 'owned-consumer' }), + cancel: (tag: string) => { + cleanup.push(`cancel:${tag}`); + return Promise.resolve(); + }, + close: () => { + cleanup.push('channel-close'); + return Promise.resolve(); + }, + }), close: () => { closed = true; + cleanup.push('connection-close'); return Promise.resolve(); }, }); - queue['queue'] = { - listen: (_handler: unknown, options: { signal: AbortSignal }) => - new Promise((resolve) => { - options.signal.addEventListener('abort', () => resolve(), { once: true }); - }), - }; const listening = queue.listen(async () => {}); await nextTurn(); await queue.stop(); await listening; + assertEquals(cleanup, ['cancel:owned-consumer', 'channel-close', 'connection-close']); assertEquals(closed, true); assertEquals(queue['listening'], false); }); diff --git a/plugins/workers/tests/runtime/seed-sample_test.ts b/plugins/workers/tests/runtime/seed-sample_test.ts index 9b2ff2b85f..efc188763d 100644 --- a/plugins/workers/tests/runtime/seed-sample_test.ts +++ b/plugins/workers/tests/runtime/seed-sample_test.ts @@ -209,7 +209,14 @@ Deno.test('seed watcher rearms after its parent directory is removed', async () await Deno.remove(directory, { recursive: true }); const recreated = (async () => { for await (const event of changes) { - if (event.kind === 'create' && event.paths.includes(directory)) return true; + if (event.kind === 'create' && event.paths.includes(directory)) { + // The initial mkdir event may still be queued after removal; require live recreation. + try { + if ((await Deno.stat(directory)).isDirectory) return true; + } catch (error) { + if (!(error instanceof Deno.errors.NotFound)) throw error; + } + } } return false; })(); diff --git a/plugins/workers/worker/queue-consumer.ts b/plugins/workers/worker/queue-consumer.ts index ab27e49173..591bd8324f 100644 --- a/plugins/workers/worker/queue-consumer.ts +++ b/plugins/workers/worker/queue-consumer.ts @@ -43,6 +43,7 @@ export async function startQueueTriggerListeners( ); supervisors.push(supervisor); supervisor.start(); + context.onListenerStarted?.(supervisor); } return supervisors; } @@ -72,6 +73,7 @@ export function startTaskQueueListener( ); }); supervisor.start(); + queueContext.onListenerStarted?.(supervisor); return supervisor; } diff --git a/plugins/workers/worker/worker-intake-lifecycle_test.ts b/plugins/workers/worker/worker-intake-lifecycle_test.ts index f9ee004f5f..425f5f2443 100644 --- a/plugins/workers/worker/worker-intake-lifecycle_test.ts +++ b/plugins/workers/worker/worker-intake-lifecycle_test.ts @@ -215,3 +215,66 @@ Deno.test('stop during initialization is bounded and prevents late listener allo worker.pause(); await assertRejects(() => worker.start(), Error, 'stopped Worker'); }); + +for (const beforeAllocation of [true, false]) { + Deno.test(`pause ${beforeAllocation ? 'before' : 'after'} trigger allocation stops intake during registry initialization`, async () => { + await using kv = new MemoryKvAdapter(); + const first = Promise.withResolvers(); + const allowFirst = Promise.withResolvers(); + const second = Promise.withResolvers(); + const allowSecond = Promise.withResolvers(); + let intake: AbortSignal | undefined; + const context = recoveryContext(new KvWorkerIdempotencyStore({ kv }), { + execute: () => Promise.resolve({ success: true, duration: 1 }), + }); + using _listener = stub(DenoKvAdapter.prototype, 'listen', (_handler, options) => { + intake = options?.signal; + return new Promise((resolve) => { + if (intake?.aborted) resolve(); + else intake?.addEventListener('abort', () => resolve(), { once: true }); + }); + }); + const worker = new Worker({ + ...context, + drainTimeoutMs: 500, + abortGracePeriodMs: 100, + queue: new MemoryQueueAdapter(), + taskQueue: new MemoryQueueAdapter(), + registry: { + get: async (id) => { + if (id === 'first') { + first.resolve(); + await allowFirst.promise; + } else { + second.resolve(); + await allowSecond.promise; + } + return context.registry.get(id); + }, + }, + queueTriggers: [{ queueName: 'first', jobId: 'first' }, { + queueName: 'second', + jobId: 'second', + }], + }); + const started = worker.start(); + try { + await bounded(first.promise); + if (beforeAllocation) worker.pause(); + allowFirst.resolve(); + await bounded(second.promise); + if (!beforeAllocation) worker.pause(); + assert(intake); + assertEquals( + intake.aborted, + true, + 'New trigger intake stayed live during the held registry lookup.', + ); + } finally { + allowFirst.resolve(); + allowSecond.resolve(); + await worker.stop(); + await started; + } + }); +} diff --git a/plugins/workers/worker/worker-options.ts b/plugins/workers/worker/worker-options.ts index c7087a52ef..9186c3318a 100644 --- a/plugins/workers/worker/worker-options.ts +++ b/plugins/workers/worker/worker-options.ts @@ -18,7 +18,11 @@ import type { import type { TracedMessageContext } from '@netscript/telemetry/instrumentation'; import type { Span } from '@netscript/telemetry/tracer'; import type { WorkerPool, WorkerPoolOptions } from './job-runner-pool.ts'; -import type { WorkerListenerFailureCause, WorkerListenerSnapshot } from './listener-supervisor.ts'; +import type { + WorkerListenerFailureCause, + WorkerListenerSnapshot, + WorkerListenerSupervisor, +} from './listener-supervisor.ts'; /** Structural validation schema accepted by worker queue triggers. */ export interface WorkerPayloadSchema { @@ -233,6 +237,8 @@ export interface WorkerDispatchContext { /** Context required by queue-consumer helpers. */ export interface WorkerQueueContext { readonly workerId: string; + /** Transfer each newly started listener to its Worker before another initialization await. */ + readonly onListenerStarted?: (listener: WorkerListenerSupervisor) => void; readonly canConsume?: (delivery: MessageContext) => Promise; readonly registry: WorkerJobRegistry; readonly queueTriggers: readonly QueueTriggerConfig[]; diff --git a/plugins/workers/worker/worker-shutdown-adapters_test.ts b/plugins/workers/worker/worker-shutdown-adapters_test.ts new file mode 100644 index 0000000000..eb7025b5e3 --- /dev/null +++ b/plugins/workers/worker/worker-shutdown-adapters_test.ts @@ -0,0 +1,175 @@ +import { assert, assertEquals } from '@std/assert'; +import { MemoryKvAdapter } from '@netscript/kv'; +import { KvWorkerIdempotencyStore } from '@netscript/plugin-workers-core/stores'; +import type { JobMessage } from '@netscript/plugin-workers-core/runtime'; +import type { MessageQueue } from '@netscript/queue'; +import { DenoKvAdapter } from '@netscript/queue/adapters/deno-kv'; +import { KvPollingAdapter } from '@netscript/queue/adapters/kv-polling'; +import { PostgresAdapter } from '@netscript/queue/adapters/postgres'; +import { AmqpAdapter } from '@netscript/queue/adapters/amqp'; +import { MemoryDeadLetterStore, MemoryQueueAdapter } from '@netscript/queue/testing'; +import { Pool } from 'npm:pg@^8.23.1'; +// @ts-types="npm:@types/amqplib@^0.10.7" +import { connect } from 'npm:amqplib@^2.2.0'; +import { bounded } from '../../../packages/queue/tests/_fixtures/redis-lease-harness.ts'; +import { recoveryContext } from './test_utils/redis-worker-context.ts'; +import { Worker } from './worker.ts'; + +async function restartWorker(create: () => MessageQueue): Promise { + await using kv = new MemoryKvAdapter(); + const idempotency = new KvWorkerIdempotencyStore({ kv }); + let identity: string | undefined; + let executions = 0; + for (let restart = 0; restart <= 8; restart++) { + const entered = Promise.withResolvers(); + const processed = Promise.withResolvers(); + const queue = create(); + const context = recoveryContext(idempotency, { + execute: async (_definition, { signal }) => { + assert(signal instanceof AbortSignal); + executions++; + entered.resolve(); + if (restart < 8) { + await new Promise((resolve) => + signal.addEventListener('abort', () => resolve(), { + once: true, + }) + ); + return { success: false, duration: 1 }; + } + return { success: true, duration: 1 }; + }, + }); + const worker = new Worker({ + ...context, + drainTimeoutMs: 1_000, + abortGracePeriodMs: 800, + taskQueue: new MemoryQueueAdapter(), + queue: { + nativeRetrial: queue.nativeRetrial, + enqueue: (message, options) => queue.enqueue(message, options), + stop: () => queue.stop(), + listen: (handler, options) => + queue.listen(async (message, delivery) => { + assertEquals(delivery.deliveryCount, 1, `Shutdown ${restart} charged the attempt.`); + identity ??= delivery.messageId; + assertEquals(delivery.messageId, identity); + assertEquals(delivery.headers, { purpose: 'worker-replacement' }); + await handler(message, delivery); + processed.resolve(); + }, options), + }, + }); + if (restart === 0) { + await queue.enqueue({ jobId: 'healthy', topic: 'jobs', triggeredBy: 'queue' }, { + headers: { purpose: 'worker-replacement' }, + }); + } + const started = worker.start(); + void started.catch(() => {}); + try { + await bounded(entered.promise); + if (restart === 8) await bounded(processed.promise); + } finally { + await bounded(worker.stop()); + await bounded(started); + } + } + assertEquals(executions, 9); +} + +Deno.test('memory survives eight clean Worker replacements and finishes empty', async () => { + const queue = new MemoryQueueAdapter(); + await restartWorker(() => queue); + assertEquals(queue.drain(), []); +}); + +Deno.test('KV polling survives eight clean Worker replacements without dead letters', async () => { + await using kv = new MemoryKvAdapter(); + const queueName = crypto.randomUUID(); + const deadLetterStore = new MemoryDeadLetterStore(); + const create = () => + new KvPollingAdapter({ + kv, + queueName, + pollInterval: 1, + maxRetries: 2, + deadLetterStore, + }); + await restartWorker(create); + const inspector = create(); + assertEquals(await inspector.getStats(), { pending: 0, processing: 0, dlq: 0 }); + assertEquals(await deadLetterStore.depth(), 0); + await inspector.stop(); +}); + +Deno.test('native Deno KV survives eight clean Worker replacements without an extra delivery', async () => { + await using kv = await Deno.openKv(':memory:'); + const queueName = crypto.randomUUID(); + await restartWorker(() => DenoKvAdapter.withKv(kv, queueName)); + const inspector = DenoKvAdapter.withKv(kv, queueName); + let extra = 0; + const lifetime = new AbortController(); + const listening = inspector.listen(() => { + extra++; + return Promise.resolve(); + }, { signal: lifetime.signal }); + try { + await new Promise((resolve) => setTimeout(resolve, 200)); + } finally { + lifetime.abort(); + await listening; + await inspector.stop(); + } + assertEquals(extra, 0); +}); + +Deno.test({ + name: 'Postgres survives eight clean Worker replacements beyond maxRetries with no queue residue', + ignore: !Deno.env.get('NETSCRIPT_TEST_POSTGRES_URL'), + async fn() { + const client = new Pool({ connectionString: Deno.env.get('NETSCRIPT_TEST_POSTGRES_URL') }); + const tableName = `worker_stop_${crypto.randomUUID().replaceAll('-', '')}`; + const deadLetterStore = new MemoryDeadLetterStore(); + try { + await restartWorker(() => + new PostgresAdapter({ + client, + tableName, + pollInterval: 1, + maxRetries: 2, + deadLetterStore, + }) + ); + const result = await client.query(`SELECT COUNT(*)::int AS count FROM "${tableName}"`); + assertEquals(result.rows[0].count, 0); + assertEquals(await deadLetterStore.depth(), 0); + } finally { + await client.query(`DROP TABLE IF EXISTS "${tableName}"`); + await client.end(); + } + }, +}); + +Deno.test({ + name: 'AMQP survives eight clean Worker replacements with no queued or dead-letter residue', + ignore: !Deno.env.get('NETSCRIPT_TEST_AMQP_URL'), + async fn() { + const url = Deno.env.get('NETSCRIPT_TEST_AMQP_URL')!; + const queueName = `worker-stop-${crypto.randomUUID()}`; + const deadLetterStore = new MemoryDeadLetterStore(); + const connection = await connect(url); + const channel = await connection.createChannel(); + try { + await restartWorker(() => new AmqpAdapter(url, queueName, deadLetterStore)); + const state = await channel.checkQueue(queueName); + assertEquals(state.messageCount, 0); + assertEquals(state.consumerCount, 0); + assertEquals(await deadLetterStore.depth(), 0); + } finally { + await channel.deleteQueue(queueName); + await channel.close(); + await connection.close(); + } + }, +}); diff --git a/plugins/workers/worker/worker.ts b/plugins/workers/worker/worker.ts index cb32747aa4..f1dd46888d 100644 --- a/plugins/workers/worker/worker.ts +++ b/plugins/workers/worker/worker.ts @@ -253,10 +253,7 @@ export class Worker { await Promise.all(triggers.map((listener) => listener.stop())); return; } - this.listenerSupervisors.push(...triggers); - this.listenerSupervisors.push( - startTaskQueueListener(this.queueContext(), this.dispatchContext()), - ); + startTaskQueueListener(this.queueContext(), this.dispatchContext()); jobListener = new WorkerListenerSupervisor({ name: `jobs:${this.queueName}`, abortSignal: this.abortController.signal, @@ -435,6 +432,10 @@ export class Worker { queueTriggers: this.queueTriggers, triggerQueues: this.triggerQueues, canConsume: (delivery) => this.admit(delivery), + onListenerStarted: (listener) => { + this.listenerSupervisors.push(listener); + if (this.lifecycle !== 'running') void listener.pause().catch(() => {}); + }, abortController: this.abortController, taskQueue: this.suppliedTaskQueue, listenerMaxRestarts: this.listenerMaxRestarts, From 33c034cb5e0225d08ecf848747c4fd128322c3c1 Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Tue, 17 Nov 2026 01:07:57 +0100 Subject: [PATCH 16/20] docs(harness): record worker final gates and runtime dispatch --- .../context-pack.md | 2 +- .../supervisor.md | 3 +++ .../feat-worker-pause-drain--issues-2633-2635/worklog.md | 9 +++++++++ 3 files changed, 13 insertions(+), 1 deletion(-) diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md index b929dc3acc..4c2371b447 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md @@ -8,4 +8,4 @@ Use Deno2.9.7 through mise. rtk/rg absent; plain commands. Scoped structured wra Public artifacts/comments/bodies must contain no operator absolute paths, hostnames/ports, session/thread IDs or downstream names. Keep raw logs private, public evidence repo-relative. Evaluator writes plan-eval.md and phase comment, leaves artifact uncommitted. No source changes, commits, pushes, labels/readiness changes or PR watching by evaluator. -Current checkpoint: S2/S3 portable/native handback, S4 lifecycle/drain, S5 configuration/default forwarding and S6 public docs/generated consumers committed. Original and listener-owner addendum Plan-Gates approved. Real runtime/schema/runner12/12, full core112/0/1 existing ignore, full queue215/0/3 existing optional-backend ignores, original Worker/core-cause113/0/0; all seven adapters now have actual eight-Worker-replacement coverage. Final startup pause before/after allocation repair25/25 and corrected fixture13/13; final static/quality/corpus gates pass, public doc diagnostics unchanged. Full plugin rerun in progress. Supported one-pass scaffold.runtime, independent IMPL-EVAL, final acceptance/head checks/threads and cleanup remain. Four exact implementation backend IDs/mounts remain retained privately for final validation/evaluator. +Current checkpoint: S2/S3 portable/native handback, S4 lifecycle/drain, S5 configuration/default forwarding and S6 public docs/generated consumers committed. Original and listener-owner addendum Plan-Gates approved. Real runtime/schema/runner12/12, full core112/0/1 existing ignore, full queue215/0/3 existing optional-backend ignores, original Worker/core-cause113/0/0; all seven adapters now have actual eight-Worker-replacement coverage. Final startup pause before/after allocation repair25/25 and corrected fixture13/13; final static/quality/corpus gates pass, public doc diagnostics unchanged. Final full plugin266/0/0green. Supported one-pass scaffold.runtime workflow43984548275 was dispatched once at source2d7aa10448700b46cb1c2dbb4665044eafb60dfa; it is still pending, not green evidence. Independent IMPL-EVAL, final acceptance/head checks/threads and cleanup remain. Four exact implementation backend IDs/mounts remain retained privately for final validation/evaluator. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/supervisor.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/supervisor.md index 717d517d3d..6d56357463 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/supervisor.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/supervisor.md @@ -3,3 +3,6 @@ Issues2633/2635 share branch feat/worker-pause-drain because abort grace and drain ownership overlap. Fresh origin/main b0eb079575968e843b781579673c9fe2ae250e07 confirmed before creating this worktree. Owner-requested implementation lane Codex gpt-6.1-sol/xhigh owns research/design/implementation; no helper/planning agents. Feature tier, separate GLM5.3/provider-default PLAN-EVAL before production, MuseSpark1.3/xhigh final IMPL-EVAL (matrix lookup before dispatch), different session/vendor family. No merge, ready-merge, rebase/force-push or publication. Live matrix lookup confirms feature PLAN-EVAL primary glm_5_3/provider_default, fallback fable_5_1/low. Requested primary via OpenCode Go, provider options{}; generator OpenAI/Codex gpt-6.1-sol/xhigh. No route substitution permitted. Requested/observed identity and exact evaluated head must be recorded in the verdict. + + +Final IMPL-EVAL requested primary MuseSpark1.3/xhigh via OpenCode Go contributor route; live matrix and T3 capability match, fallback Opus5/xhigh unused. Separate session and vendor family from generator mandatory; evaluator attests observed identity and exact head. Final product source2d7aa10448700b46cb1c2dbb4665044eafb60dfa; final-gate artifact follow-up changes run docs only. Supported full runtime workflow43984548275 is pending, not a completed gate. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index a6a1829edf..cbf3fa8e8d 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -108,3 +108,12 @@ Startup audit found a real gap in the approved pause-during-initialization risk: The startup regression now covers pause both before and after the first trigger allocation while a second lookup is held. Initial allocation-only state application passed the first window but left the second red (1pass/1fail); immediate transfer of every new listener into the Worker's existing ownership list closes both windows. One narrow onListenerStarted context callback handles ownership and current-state application; no generic lifecycle registry. Final intake/supervisor25/25green. Static/check/fmt and public surface gates rechecked after this final composition fix. Final queue suite215pass/0fail/3existingoptional-backend ignores. All newly added accounting/ownership cases execute against provisioned backends. Final90-file check/lint/fmt0; plugin doclint2 at the exact baseline entrypoints; MCP corpus freshness and quality:gatePASS after final ownership repair. Full plugin rerun remains in progress; no full-smoke/evaluator or readiness claim yet. + + +## S7 final local gate set and supported runtime dispatch + +Final full plugin266pass/0fail/0ignored, full queue215pass/0fail/3unchanged optional-backend ignores, full core112pass/0fail/1unchanged ignore. All new native/backend/Worker replacement cases executed; no new skip. Final source check/lint/fmt0, quality:gate/architecture fitness0, corpus/publish freshnessPASS, release:preflight fourPASS, docs:accuracyPASS. All three native JSR publish dry-runs succeed; the final plugin dry-run repeated after startup ownership repair succeeds. Audits report existing cardinality warnings and miscount the normal slow-type-check banner as one warning; native publish has no slow-type diagnostic. Existing dynamic-import warnings remain in unchanged import sites. Doc-lint combined core2/queue1/plugin2 and exact per-entrypoint baseline preserved; new config/shutdown entries0. No lock/dependency, forbidden transport or Streams change. + +Full supported one-pass scaffold runtime workflow dispatched once at source2d7aa10448700b46cb1c2dbb4665044eafb60dfa: https://github.com/rickylabs/netscript/actions/runs/43984548275 . Native Aspire guard remains intact; no local guard bypass or bridge. Run was in progress at dispatch verification and is not claimed green. Evaluator and final readiness must distinguish completed local evidence from this pending external gate. + +Fresh matrix confirms feature IMPL-EVAL primary MuseSpark1.3/xhigh, fallback Opus5/xhigh (unused). T3 capabilities confirm OpenCode Go MuseSpark1.3 contributor with xhigh variant available. Generator CodexSol/xhigh is a different family/session. Matrix CLI rejects the operator-user flag; the documented read-only matrix invocation without that flag succeeded. T3 requires a fresh review delegate rather than same-session steering. No helper agents. Four exact owned backend containers remain available for evaluator probes; parent cleanup after evaluation is still required. From 79f35829e679e5ff0d2c62d0a1a9a62d2f1ae03e Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Tue, 17 Nov 2026 01:10:18 +0100 Subject: [PATCH 17/20] docs(harness): map worker acceptance before independent evaluation --- .../feat-worker-pause-drain--issues-2633-2635/context-pack.md | 2 ++ .llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md | 3 +++ 2 files changed, 5 insertions(+) diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md index 4c2371b447..9e06233454 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md @@ -9,3 +9,5 @@ Use Deno2.9.7 through mise. rtk/rg absent; plain commands. Scoped structured wra Public artifacts/comments/bodies must contain no operator absolute paths, hostnames/ports, session/thread IDs or downstream names. Keep raw logs private, public evidence repo-relative. Evaluator writes plan-eval.md and phase comment, leaves artifact uncommitted. No source changes, commits, pushes, labels/readiness changes or PR watching by evaluator. Current checkpoint: S2/S3 portable/native handback, S4 lifecycle/drain, S5 configuration/default forwarding and S6 public docs/generated consumers committed. Original and listener-owner addendum Plan-Gates approved. Real runtime/schema/runner12/12, full core112/0/1 existing ignore, full queue215/0/3 existing optional-backend ignores, original Worker/core-cause113/0/0; all seven adapters now have actual eight-Worker-replacement coverage. Final startup pause before/after allocation repair25/25 and corrected fixture13/13; final static/quality/corpus gates pass, public doc diagnostics unchanged. Final full plugin266/0/0green. Supported one-pass scaffold.runtime workflow43984548275 was dispatched once at source2d7aa10448700b46cb1c2dbb4665044eafb60dfa; it is still pending, not green evidence. Independent IMPL-EVAL, final acceptance/head checks/threads and cleanup remain. Four exact implementation backend IDs/mounts remain retained privately for final validation/evaluator. + +Completion phase comment6933353414. Draft status:impl-eval, all9acceptance boxes checked and both closing lines/mappings present; final DoD pending. Thread gate0/0. Evaluator may reuse the four existing positively owned backend fixtures using private creation receipts; do not stop their containers. Parent owns final cleanup. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index cbf3fa8e8d..c78f6a0127 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -117,3 +117,6 @@ Final full plugin266pass/0fail/0ignored, full queue215pass/0fail/3unchanged opti Full supported one-pass scaffold runtime workflow dispatched once at source2d7aa10448700b46cb1c2dbb4665044eafb60dfa: https://github.com/rickylabs/netscript/actions/runs/43984548275 . Native Aspire guard remains intact; no local guard bypass or bridge. Run was in progress at dispatch verification and is not claimed green. Evaluator and final readiness must distinguish completed local evidence from this pending external gate. Fresh matrix confirms feature IMPL-EVAL primary MuseSpark1.3/xhigh, fallback Opus5/xhigh (unused). T3 capabilities confirm OpenCode Go MuseSpark1.3 contributor with xhigh variant available. Generator CodexSol/xhigh is a different family/session. Matrix CLI rejects the operator-user flag; the documented read-only matrix invocation without that flag succeeded. T3 requires a fresh review delegate rather than same-session steering. No helper agents. Four exact owned backend containers remain available for evaluator probes; parent cleanup after evaluation is still required. + + +Implementation completion comment: https://github.com/rickylabs/netscript/pull/2645#issuecomment-6933353414 . Issues2633(4/4) and2635(5/5) acceptance now checked with linked local executable proof; PR body carries separate closing lines and4+5fenced mappings. Final DoD/readiness gate remains unchecked pending independent verdict, supported runtime and final checks/cleanup. Exactly one status:impl-eval; still draft. Review-thread gate0threads/0unanswered. Current artifact head's lightweight checks were in progress/policy-skipped, not claimed green. Product remains identical to2d7aa10448700b46cb1c2dbb4665044eafb60dfa. From 9147e69249ead612ee8a526238e6cb7442d85437 Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Tue, 17 Nov 2026 01:30:45 +0100 Subject: [PATCH 18/20] docs(harness): retain worker evaluation and exact resource cleanup --- .../context-pack.md | 6 +- .../evaluate.md | 120 ++++++++++++++++++ .../leak-report.md | 5 + .../worklog.md | 12 ++ 4 files changed, 141 insertions(+), 2 deletions(-) create mode 100644 .llm/runs/feat-worker-pause-drain--issues-2633-2635/evaluate.md diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md index 9e06233454..920ed11338 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md @@ -8,6 +8,8 @@ Use Deno2.9.7 through mise. rtk/rg absent; plain commands. Scoped structured wra Public artifacts/comments/bodies must contain no operator absolute paths, hostnames/ports, session/thread IDs or downstream names. Keep raw logs private, public evidence repo-relative. Evaluator writes plan-eval.md and phase comment, leaves artifact uncommitted. No source changes, commits, pushes, labels/readiness changes or PR watching by evaluator. -Current checkpoint: S2/S3 portable/native handback, S4 lifecycle/drain, S5 configuration/default forwarding and S6 public docs/generated consumers committed. Original and listener-owner addendum Plan-Gates approved. Real runtime/schema/runner12/12, full core112/0/1 existing ignore, full queue215/0/3 existing optional-backend ignores, original Worker/core-cause113/0/0; all seven adapters now have actual eight-Worker-replacement coverage. Final startup pause before/after allocation repair25/25 and corrected fixture13/13; final static/quality/corpus gates pass, public doc diagnostics unchanged. Final full plugin266/0/0green. Supported one-pass scaffold.runtime workflow43984548275 was dispatched once at source2d7aa10448700b46cb1c2dbb4665044eafb60dfa; it is still pending, not green evidence. Independent IMPL-EVAL, final acceptance/head checks/threads and cleanup remain. Four exact implementation backend IDs/mounts remain retained privately for final validation/evaluator. +Current checkpoint: S2/S3 portable/native handback, S4 lifecycle/drain, S5 configuration/default forwarding and S6 public docs/generated consumers committed. Original and listener-owner addendum Plan-Gates approved. Real runtime/schema/runner12/12, full core112/0/1 existing ignore, full queue215/0/3 existing optional-backend ignores, original Worker/core-cause113/0/0; all seven adapters now have actual eight-Worker-replacement coverage. Final startup pause before/after allocation repair25/25 and corrected fixture13/13; final static/quality/corpus gates pass, public doc diagnostics unchanged. Final full plugin266/0/0green. Supported one-pass scaffold.runtime workflow43984548275 was dispatched once at source2d7aa10448700b46cb1c2dbb4665044eafb60dfa; it is still pending, not green evidence. Independent IMPL-EVAL, final acceptance/head checks/threads and cleanup remain. All four implementation backend containers and four recorded anonymous volumes are now cleaned with exact identity/mount proof; no survivors. -Completion phase comment6933353414. Draft status:impl-eval, all9acceptance boxes checked and both closing lines/mappings present; final DoD pending. Thread gate0/0. Evaluator may reuse the four existing positively owned backend fixtures using private creation receipts; do not stop their containers. Parent owns final cleanup. +Completion phase comment6933353414. Draft status:impl-eval, all9acceptance boxes checked and both closing lines/mappings present; final DoD pending. Thread gate0/0. Evaluator PASS retained; all four owned implementation fixtures and their four volumes are now removed. Do not reuse stale connection receipts. + +External first attempt: SQLite116+showcase8/static/native green; Postgres failed during documented-scaffold db generate dashboard refresh (same symptom baseline43936107382). Same-source failed-job-only retry requested, gate still mandatory. Independent verdict comment6933679178; product source unchanged. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/evaluate.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/evaluate.md new file mode 100644 index 0000000000..02e03770de --- /dev/null +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/evaluate.md @@ -0,0 +1,120 @@ +# IMPL-EVAL — feat-worker-pause-drain--issues-2633-2635 + +## Metadata + +| Field | Value | +| --- | --- | +| Run ID | `feat-worker-pause-drain--issues-2633-2635` | +| Target | PR 2645, issues #2633 and #2635 | +| Archetype | 3 runtime/behavior, folding integration 2 and plugin 5 overlays | +| Scope overlays | queue integration, workers plugin, core contracts | +| Evaluator | Fresh independent feature IMPL-EVAL, Muse Spark 1.3 xhigh via OpenCode Go contributor, 2026-11-17 | +| Evaluated head | `79f35829e679e5ff0d2c62d0a1a9a62d2f1ae03e`, branch `feat/worker-pause-drain`, base `b0eb079575968e843b781579673c9fe2ae250e07` | +| Product source | `2d7aa10448700b46cb1c2dbb4665044eafb60dfa`; two follow-up commits touch run docs only | +| Working tree | Clean | + +Route attestation: requested pinned feature IMPL-EVAL primary `muse_spark_1_3/xhigh`; live matrix lookup confirms primary `muse_spark_1_3/xhigh`, fallback `opus_5/xhigh`. Observed route matches requested primary in this fresh session; fallback unused; no substitution. Generator is OpenAI Codex `gpt-6.1-sol/xhigh` in a separate session and separate vendor family. No helper or implementation agents started by this evaluator. Deno `2.9.7` via mise; structured wrappers used; raw reports kept private and uncommitted. + +Prior gates: original feature PLAN-EVAL PASS and listener-owner addendum PASS, both in separate prior sessions, preserved. No plan redesign requested. + +## Process Verification + +| Check | Result | Evidence | +| --- | --- | --- | +| Plan-Gate passed before implementation | PASS | `plan-eval.md` cycle 1 PASS plus addendum cycle 2 PASS; S2 started only after cycle 1 | +| Design section exists in worklog | PASS | `worklog.md` Design plus per-slice design deltas | +| Commit slices match design plan | PASS | S1 research/reds, S2 core/queue contracts, S3a portable handback, S3b listener ownership, original S3 AMQP, S4 lifecycle/drain, S5 config/runtime, S6 docs/consumers, S7 gates/dispatch; all groups under file limits | +| Each slice has a passing gate | PASS | Per-slice structured check/lint/fmt, doc-lint, preflight/accuracy, backend suites recorded in `worklog.md`; independently re-verified below | +| No speculative seams | PASS | No generic lifecycle registry, no new dependency or lock change, no saga transport or Streams edits; worker stays under size cap; polling extraction reduces prior oversize file | +| Constants used | PASS | Shared core budget resolver and defaults reused by config, runner, and worker; no duplicated magic totals | + +## Static Gates (independently re-run) + +| Gate | Command or check | Result | Evidence | +| --- | --- | --- | --- | +| Narrow typecheck | structured check wrapper over `packages/queue`, `packages/plugin-workers-core`, `plugins/workers` | PASS | 0 findings | +| Lint | structured lint wrapper over same roots | PASS | 0 findings | +| Format | structured fmt wrapper over same roots | PASS | 381 files selected, 0 findings | +| Quality gate | `deno task quality:gate` | PASS | exit 0; no new `any`/cast or plugin-name coupling | +| Architecture fitness | `deno task arch:check` | PASS | exit 0; only pre-existing WARN/INFO, FAIL 0 | +| Doc lint | structured doc-lint over queue, plugin-workers-core, workers | PASS | queue combined 1, core combined 2, plugin combined 2; exact per-entrypoint baseline unchanged; new `shutdown` and `config` entries 0 | +| Publish dry-run | native `deno publish --dry-run --allow-dirty` in each of the three packages | PASS | all Success; no slow-type diagnostic | +| JSR audit | fitness audit helper for each package | PASS | exit 0; only existing cardinality warnings plus helper banner miscount of normal slow-type check, disclosed; no new file-list or metadata issue | +| Release preflight | `deno task release:preflight` | PASS | four PASS | +| Docs accuracy | `deno task docs:accuracy` | PASS | PASS | +| Consumer freshness | generated prose/corpus/asset carriers and freshness checks | PASS | generation exit 0; freshness checks PASS; no lock change | +| Review threads | `deno task check:review-threads -- --repo rickylabs/netscript --pr 2645 --pretty` | PASS | 0 threads, 0 unanswered | + +## Fitness Gates (Archetype 3 plus integration/plugin overlays) + +Required F-1 through F-19 reviewed. File-size: worker 477 lines, polling adapter reduced to 737 lines, Postgres reduced to 568 lines; focused test files under caps. No helper-reinvention, layering, inheritance, public-surface, JSR, doc-score, permission, test-shape, forbidden-folder, naming, runtime-invariant, console-log, re-export, cardinality, co-location, or sub-barrel violation introduced. Existing cardinality and public-type-closure WARNs are pre-existing and unchanged; no new debt entry required. + +## Runtime Gates (independently re-run with retained owned backends) + +Backends reused by UUID-scoped tests only; no owned resource stopped or deleted; foreign resources untouched. + +| Gate | Validation | Result | Evidence | +| --- | --- | --- | --- | +| Queue full | full `packages/queue/tests/` | PASS | 215 passed, 0 failed, 3 existing optional-backend ignores | +| Workers full | full `plugins/workers/` | PASS | 266 passed, 0 failed, 0 ignored | +| Core full | full `packages/plugin-workers-core/` | PASS | 113 passed, 0 failed, 0 ignored; prior lane reported 112 plus 1 pre-existing ignore under a narrower selection | +| Config/runtime/real subprocess | schema defaults/validation plus real project config and handler with 1.25s cleanup across both startup paths and both explicit override directions | PASS | focused budget/runtime cases green; terminal state remains timeout, never false success | +| Intake/supervisor/startup ownership | pause/resume generation joins for jobs/tasks/triggers, healthy paused lifetime, pending start, repeated/racing calls, concurrency ceiling, pause before/after allocation during held registry lookup | PASS | focused intake/supervisor suites green; final startup repair covers both initialization windows | +| Shutdown deliveries/adapters | eight Worker replacements on all seven adapters with final-depth, identity, header, and count controls, plus ordinary explicit/thrown failure charging | PASS | focused shutdown suites green; eight-restart oracle and failure controls verified in source | +| Adjacent native/seed fixtures | AMQP/policy plus seed-sample corrections | PASS | corrected fixtures green | + +Genuine red-before-green: research and worklog record executable reds on unchanged main including config stripping, short-grace cutoff, missing pause, immediate drain abort, memory/KV-polling counts, Deno KV explicit nack loss, live Redis/Garnet/Postgres count, real Worker replacement count, and AMQP accounting blockage with honest prerequisite handling. Excluded harness mistakes are recorded and not counted as product evidence. First-cause preservation is narrowly pinned by red retry cases; shutdown-only preservation cannot be reclassified by earlier business failure, deadline, cancel, backoff, lease loss, duplicate/stale receipt, or operator cancel. + +## Consumer Gates + +Real standard runtime uses project config, native queue/state, dynamic handler, and both startup paths with no static registry or fake runner. Explicit direct option wins over legacy pool-only value. Generated glue delegates to the same combined startup. Public contracts use explicit owned types and annotations; no exported Zod implementation, broad casts, new dependency, or root/sub-barrel mirror. + +## Boundaries checked + +- Healthy pause and generation joining for jobs, tasks, and triggers; no intake during pause; resume waits prior generation. +- First-cause shutdown-only refund; earlier business failure, deadline, cancel, backoff, lease loss, duplicate/stale/expired receipt, DLQ/outage paths remain charged or terminal. +- Bounded 25s total with derived grace and monotonic deadline; work/abort/settlement arithmetic validated; standalone runner default documented against worker and queue budgets. +- Terminal and idempotent stop; concurrent callers share completion; startup owns allocation before and after trigger lookup during held registry; no queue close or runner shutdown during pause. +- Durable stale/duplicate/expired receipt fencing; one refund per live claim; no newer-claim refund or duplicate enqueue. +- Native Deno KV enqueue-before-ack and AMQP confirmed publish with native cancel; no new transport seam. +- Real standard runtime configuration without static registry; explicit override precedence; shared schema and defaults. +- JSR public type closure and unchanged per-entrypoint diagnostic baseline. +- No generic lifecycle registry, new dependency or lock change, saga transport edits, or Streams dependency. + +## External and readiness gates (not green; parent-owned) + +- Supported full one-pass `scaffold.runtime` workflow dispatched once at product source: live read shows overall `in_progress`, with the Postgres leg completed `failure` and the sqlite leg still `in_progress`; detailed logs unavailable while the run remains in progress. This evaluator launched no additional smoke run and performed no wait or poll. +- Current PR checks on the evaluated head are policy-skipped under draft labels except one passing command check; no reviews. PR remains draft with `status:impl-eval`. +- Issues #2633 4/4 and #2635 5/5 are checked with linked local evidence; PR carries closing lines and complete fenced acceptance-evidence mappings. Final Definition-of-Done box remains intentionally unchecked pending independent verdict, external runtime, current-head checks, threads, and cleanup. Local acceptance-mirror dry-run is policy-skipped because `ready-merge` is forbidden. +- No merge, rebase, push, label, checkbox, readiness, watch, release, or publication change was made by this evaluator. Only this `evaluate.md` was written and left uncommitted. + +## Arch-Debt Delta + +| Metric | Count | Evidence | +| --- | --- | --- | +| New entries | 0 | No new doctrine violation introduced | +| Resolved entries | 0 | Existing cardinality and public-type-closure debt unchanged | +| Deepened violations | 0 | Reductions only; no broadened exemption | +| Unrecorded violations | 0 | All WARNs trace to pre-existing entries or non-blocking INFO | + +## Findings + +| Severity | Finding | Evidence | Required action | +| --- | --- | --- | --- | +| High | External `scaffold.runtime` gate is not green: Postgres leg reports failure while sqlite leg remains in progress; logs unavailable until completion | Live workflow read at evaluation time; PR body S7 and DoD remain unchecked | Parent triage required; no ready-merge; do not certify this gate; generator must not re-run the expensive smoke without owner direction | +| None | No implementation blocking finding | All local static, fitness, runtime, consumer, JSR, docs, and thread gates independently PASS with evidence above | None; parent owns final runtime, required-head CI, DoD, and cleanup transitions | + +## Lessons for Promotion + +| Lesson | Pattern | Applies to | Confidence | +| --- | --- | --- | --- | +| None | No new reusable pattern beyond the approved contracts | N/A | N/A | + +## Verdict + +| Field | Value | +| --- | --- | +| Verdict | `PASS` | +| Rationale | Approved scope is complete; every local static, fitness, runtime, consumer, JSR, docs, and thread gate independently passes with evidence; no unrecorded doctrine violation; public contracts, docs, and all nine acceptance criteria map to executable proof with genuine reds and failure controls; forbidden surfaces absent. External `scaffold.runtime` is explicitly not certified here and remains parent-owned; its current non-green state blocks ready-merge but does not invalidate the implementation verdict. DoD remains incomplete by design pending parent-owned runtime, current-head checks, and cleanup. | + +PR https://github.com/rickylabs/netscript/pull/2645, head 79f35829e679e5ff0d2c62d0a1a9a62d2f1ae03e, checks and reviews: PR checks policy-skipped under draft labels except one passing command check; no reviews; external scaffold runtime in progress with one leg failed and one leg in progress. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md index acb39ed740..6425635f7d 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/leak-report.md @@ -7,3 +7,8 @@ Commands: maint:leak-check and maint:teardown preview against this run and workt Three run-created Redis/Garnet/Postgres backend containers are intentionally retained for S2/S3 validation. Immutable IDs and native mount identities recorded privately at creation, with the dedicated implementation test-owner label. No AppHost or foreign container/process started or stopped. Final cleanup will preview the repo teardown tool and remove only those exact proven identities/mounts; the research zero-survivor result above is historical, not a claim that these active implementation resources are absent. AMQP validation adds one positively owned RabbitMQ backend; active implementation set is four containers with recorded immutable IDs/mounts. No foreign resources touched. + + +## Final implementation cleanup + +After independent evaluation, maint:leak-check and maint:teardown preview completed read-only. Native immutable creation receipts, exact test-owner label and current native mount identity verified for each of the four implementation backends. Reporter classifies plain test fixtures as unproven; no broad teardown was applied. Every other foreign/unproven entry was reviewed and retained. Removed only the four positively proven container identities, then their four recorded anonymous volumes after a full reference scan proved no other container used them. Final exact-owner scan: zero implementation containers; recorded-volume scan: zero survivors. Maint final leak report captured privately and scrubbed from the public artifact. No AppHost, foreign process, container, network or volume changed. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index c78f6a0127..90202d62ee 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -120,3 +120,15 @@ Fresh matrix confirms feature IMPL-EVAL primary MuseSpark1.3/xhigh, fallback Opu Implementation completion comment: https://github.com/rickylabs/netscript/pull/2645#issuecomment-6933353414 . Issues2633(4/4) and2635(5/5) acceptance now checked with linked local executable proof; PR body carries separate closing lines and4+5fenced mappings. Final DoD/readiness gate remains unchecked pending independent verdict, supported runtime and final checks/cleanup. Exactly one status:impl-eval; still draft. Review-thread gate0threads/0unanswered. Current artifact head's lightweight checks were in progress/policy-skipped, not claimed green. Product remains identical to2d7aa10448700b46cb1c2dbb4665044eafb60dfa. + + +## Independent implementation evaluation and external-gate triage + +Fresh MuseSpark1.3/xhigh independent IMPL-EVAL PASS at79f35829e679e5ff0d2c62d0a1a9a62d2f1ae03e. Evaluator reran full queue215/0/3, plugin266/0/0, core113/0/0 (its broader environment executed the formerly optional case), static/fitness/JSR/docs/consumer gates and all9acceptance. Artifact evaluate.md retained unchanged. No implementation findings; external runtime explicitly uncertified and final DoD still pending. + +Supported workflow43984548275 completed with static/native/SQLite runtime SUCCESS, Postgres FAILURE. Postgres failure is behavior.documented-scaffold's restored variant during db generate refreshing dashboard, after users refresh; first two documented variants and every owned cleanup assertion pass. The bounded error capture truncates native restart detail, so exact root cause remains unknown. This stage/symptom also exists on unchanged main: [baseline43936107382](https://github.com/rickylabs/netscript/actions/runs/43936107382), recorded with crossed candidate/main evidence in the runtime shutdown PR. That comparison predates this worker implementation and prevents inferring a worker regression from the refresh symptom alone. No bypass/assertion removal or speculative product fix. + +A single failed-job-only rerun at the same immutable source is justified by the crossed baseline evidence and completed green alternate runtime/static/native legs. It retains the full one-pass command and cleanup assertions, rather than repeating every lane. The original failed attempt remains evidence, not a PASS. Owner authorization to complete required validation already covers necessary failure diagnosis/retry; the evaluator's suggestion to request fresh permission does not override that authorization. Final supported Postgres PASS remains mandatory before readiness. + + +Final resource cleanup completed after independent evaluation: four exact native container IDs and four recorded anonymous volumes removed after native label/mount verification, maintained read-only leak/teardown preview, and a full unreferenced-volume scan. Final owner/recorded-volume scans zero; every foreign/unproven resource retained. Sanitized leak-report.md records proof; raw logs/identities remain private. Supported SQLite116/116 +showcase8/8, static/native SUCCESS; Postgres-only unchanged-source retry remains pending. No worker source change after independent PASS. From 4ed91e2dde396f5baf21fb52851a7ee698a7b78a Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Tue, 17 Nov 2026 02:11:30 +0100 Subject: [PATCH 19/20] docs(harness): record worker runtime completion --- .../context-pack.md | 2 ++ .../feat-worker-pause-drain--issues-2633-2635/worklog.md | 7 +++++++ 2 files changed, 9 insertions(+) diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md index 920ed11338..a7500beec1 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md @@ -13,3 +13,5 @@ Current checkpoint: S2/S3 portable/native handback, S4 lifecycle/drain, S5 confi Completion phase comment6933353414. Draft status:impl-eval, all9acceptance boxes checked and both closing lines/mappings present; final DoD pending. Thread gate0/0. Evaluator PASS retained; all four owned implementation fixtures and their four volumes are now removed. Do not reuse stale connection receipts. External first attempt: SQLite116+showcase8/static/native green; Postgres failed during documented-scaffold db generate dashboard refresh (same symptom baseline43936107382). Same-source failed-job-only retry requested, gate still mandatory. Independent verdict comment6933679178; product source unchanged. + +Latest checkpoint: independent IMPL-EVAL PASS and supported workflow43984548275 attempt2 SUCCESS (Postgres124, SQLite116+showcase8, static/native/cleanup) at unchanged source2d7aa104. Four owned containers/four volumes gone. PR2645 ready for review; required non-draft head CI pending at transition, not yet a final green-head claim. No further implementation rework identified. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index 90202d62ee..8a4668c78f 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -132,3 +132,10 @@ A single failed-job-only rerun at the same immutable source is justified by the Final resource cleanup completed after independent evaluation: four exact native container IDs and four recorded anonymous volumes removed after native label/mount verification, maintained read-only leak/teardown preview, and a full unreferenced-volume scan. Final owner/recorded-volume scans zero; every foreign/unproven resource retained. Sanitized leak-report.md records proof; raw logs/identities remain private. Supported SQLite116/116 +showcase8/8, static/native SUCCESS; Postgres-only unchanged-source retry remains pending. No worker source change after independent PASS. + + +## Supported runtime completion and ready transition + +Supported full one-pass workflow43984548275 attempt2 SUCCESS at unchanged product source2d7aa10448700b46cb1c2dbb4665044eafb60dfa: Postgres124/124, SQLite116/116 plus showcase8/8, static/native and cleanup assertions intact. The original failed Postgres refresh attempt remains recorded above; exactly one failed-job-only retry passed without a code change or weakened gate. Independent Muse IMPL-EVAL PASS retained, no product delta. Final exact resource scan zero owned containers/recorded volumes; foreign objects retained. + +PR2645 now ready for review, status:impl-eval, all9acceptance and final DoD boxes complete. Current non-draft head CI still has pending quality/check-test lanes, so overall task completion is not claimed. Close-gate and browser/command/docs/build lanes passed, other lanes skipped per workflow policy. Final required head checks and review-thread gate remain mandatory; no ready-merge or merge authorization. From 445002122406a9accc96495c0dc80c72d6f6cc9d Mon Sep 17 00:00:00 2001 From: Rickylabs Date: Tue, 17 Nov 2026 02:38:17 +0100 Subject: [PATCH 20/20] fix(workers): preserve the in-process runner startup log --- .../context-pack.md | 2 ++ .../feat-worker-pause-drain--issues-2633-2635/worklog.md | 7 +++++++ plugins/workers/worker/worker.ts | 3 +++ 3 files changed, 12 insertions(+) diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md index a7500beec1..8e5170d3dc 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/context-pack.md @@ -15,3 +15,5 @@ Completion phase comment6933353414. Draft status:impl-eval, all9acceptance boxes External first attempt: SQLite116+showcase8/static/native green; Postgres failed during documented-scaffold db generate dashboard refresh (same symptom baseline43936107382). Same-source failed-job-only retry requested, gate still mandatory. Independent verdict comment6933679178; product source unchanged. Latest checkpoint: independent IMPL-EVAL PASS and supported workflow43984548275 attempt2 SUCCESS (Postgres124, SQLite116+showcase8, static/native/cleanup) at unchanged source2d7aa104. Four owned containers/four volumes gone. PR2645 ready for review; required non-draft head CI pending at transition, not yet a final green-head claim. No further implementation rework identified. + +Ready head CI found a genuine accidentally omitted startup-log guard; exact main message restored, unchanged guard/intake/supervisor27/27 and source gates green. Only three console.log lines change after independent runtime PASS. Required new-head CI still mandatory; earlier head had17success/13skip/2failure. diff --git a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md index 8a4668c78f..2ba8ace239 100644 --- a/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md +++ b/.llm/runs/feat-worker-pause-drain--issues-2633-2635/worklog.md @@ -139,3 +139,10 @@ Final resource cleanup completed after independent evaluation: four exact native Supported full one-pass workflow43984548275 attempt2 SUCCESS at unchanged product source2d7aa10448700b46cb1c2dbb4665044eafb60dfa: Postgres124/124, SQLite116/116 plus showcase8/8, static/native and cleanup assertions intact. The original failed Postgres refresh attempt remains recorded above; exactly one failed-job-only retry passed without a code change or weakened gate. Independent Muse IMPL-EVAL PASS retained, no product delta. Final exact resource scan zero owned containers/recorded volumes; foreign objects retained. PR2645 now ready for review, status:impl-eval, all9acceptance and final DoD boxes complete. Current non-draft head CI still has pending quality/check-test lanes, so overall task completion is not claimed. Close-gate and browser/command/docs/build lanes passed, other lanes skipped per workflow policy. Final required head checks and review-thread gate remain mandatory; no ready-merge or merge authorization. + + +## Required-head CI startup-log correction + +Ready head4ed91e2d required CI found one failure in1753pass/1fail/21existingignores: the maintained worker-runner-docs guard expects the existing in-process startup log, inadvertently omitted during lifecycle refactoring. Reproduced locally1pass/1fail before correction. Restored the exact existing startup message before running-state initialization; no branch, wait, dependency, runtime wiring or drain change. Guard retained unchanged. Prior independent behavior PASS and supported smoke remain valid for unchanged control flow; focused validation and required-head CI must pass before completion. + +Startup log correction gates: unchanged documentation guard plus intake/supervisor27/27green (including former1/1red), scoped source check/lint/fmt0, quality:gate0, release:preflight fourPASS and docs:accuracyPASS. One mistyped local test filename was corrected before any behavioral run; no empty run claimed green. Required-head checks rerun on the pushed correction, not inferred from local results. No new resource started. diff --git a/plugins/workers/worker/worker.ts b/plugins/workers/worker/worker.ts index f1dd46888d..7f1099d27e 100644 --- a/plugins/workers/worker/worker.ts +++ b/plugins/workers/worker/worker.ts @@ -232,6 +232,9 @@ export class Worker { if (this.lifecycle === 'stopped') { throw new Error('A stopped Worker cannot be restarted; create a replacement.'); } + console.log( + `[Worker ${this.workerId}] Starting in-process job runner (queue concurrency: ${this.concurrency})...`, + ); this.lifecycle = 'running'; this.childHealth.registryLoaded(); this.abortController = new AbortController();