Repository navigation
feat(workers): pause intake and drain without charging shutdown deliveries - #2645
Conversation
|
[PHASE: RESEARCH] Current main findings reproduced before product edits. Core config/default cleanup tests fail; Worker pause is absent and drain aborts immediately. Memory/KV polling/real Redis/Garnet/Postgres shutdown handback increments count1→2, ordinary explicit failure controls count correctly. Actual Redis/Garnet Worker replacement also fails on the first restart. Native Deno KV explicit nack is a no-op, and an actual thrown-handler retry retains count1 instead of2. AMQP baseline cannot reach accounting because its upstream non-native async path double-acks and the broker rejects the tag; native channel/confirmed-handback prerequisites are recorded. Four new executable test files; product unchanged. Corrected Postgres schema-startup harness race excluded. All five run-created containers (including initial Garnet replacement) and five volumes proven by native mount events removed by exact identity; no foreign cleanup. Research and exact result interpretation. |
|
[PHASE: PLAN] Locked feature plan: core-owned lifecycle/clock/budgets,25s total default with19s graceful work/5s cleanup/1s settlement; per-attempt pause barriers retain worker start lifetime, admitted jobs/tasks, lease renewal and concurrency. Explicit shutdown-only count preservation is fenced/idempotent and keeps ordinary failures charged. Shared schema/native project loader and both standard constructors forward the actual runner grace. Confirmed AMQP publication precedes old acknowledgement. Seven ordered slices, each<30files, explicit risks/publication/debt baselines and backend/runtime/consumer gates. Four new test files are red on unchanged product source and pass lint/fmt. No product implementation before independent feature PLAN-EVAL PASS. |
|
[PHASE: PLAN-EVAL] [VERDICT: APPROVED] Independent feature-tier Plan-Gate cycle 1 (of max 2) — PASS at head Route attestation: requested pinned primary Plan-Gate checklist (8/8 PASS)
Findings
Verification performedRepository diff at head is exactly four red test files plus run-dir documents — no product source changed. The red tests are executable against unchanged main contracts (verified compile-safety and failure mechanisms: schema strips the budget fields; dispatch settles within 1s before 1250ms cleanup; pause/resume missing; drain aborts at entry; counted shutdown handback on memory/KV polling/Redis/Garnet/Postgres and real Worker first replacement; Deno KV explicit nack no-op and thrown-retry undercount; AMQP broker-406 recorded as a baseline prerequisite, not an accounting result). Ordinary failure controls read green where claimed. All nine acceptance boxes across #2633/#2635 are mapped to concrete, meetable planned evidence. Acceptance coverage9/9 meetable — #2633 boxes 1–4 (config + real standard runtime forwarding, documented default derived from the drain budget, >1s cleanup awaited, docs) and #2635 boxes 1–5 (pause/resume, bounded drain with in-budget ack, shutdown requeue preserving attempts on every counting adapter including 8 restarts beyond limit 5, failures still charging, docs). Next
PR #2645, head 89effd6, checks and reviews: required CI policy-skipped (command-postgres green), no reviews yet. |
|
[PHASE: IMPL] S2: added core lifecycle/clock/budget contracts and the bounded work→abort→settlement primitive, plus the opt-in shutdown-only nack contract. Redis/Garnet refunds use the existing atomic receipt fence; Postgres atomically refunds/unlocks using per-claim identity and prevents stale ack/release or duplicate settlement. Postgres adapter shrank581→568lines. Validation: real Redis/Garnet/Postgres28/28 (including repeated handback, ordinary failure/expiry charging, terminal nack, duplicate and stale replacement fences); core budget/Postgres8/8; scoped check/lint/fmt0, new shutdown doclint0, release preflight/docs accuracyPASS. Other adapter and Worker/config red tests remain intentionally pending their slices. Run evidence. |
|
[PHASE: IMPL] S3a implements memory/DenoKV/KV-polling handback and once-only settlement. Native DenoKV now enqueues the retained/counted envelope before native ack; ordinary thrown handlers charge. KV polling uses opaque claim identity + CAS move, guards automatic settlement, reports handback storage failure through its listener and retains the durable processing entry. Adapter811→727lines. Validation: local retry9/9, local stale/duplicate/terminal/late-context9/9, adjacent lifecycle/DLQ/outage22/22; final combined filtered24/24 after the per-generation controller fix. Check/lint/fmt0, exact per-entrypoint doc-lint baseline unchanged (combined1), preflight/accuracyPASS. The outage test reproduced an uncaught settlement rejection before the fix. Three owned implementation backends retained with recorded identities/mounts; no foreign cleanup. A separate new red test exposes native post-intake drain expiry:3owner cases fail on Redis/Garnet/KV polling;3default controls pass. This is an explicit contract checkpoint, not a green implementation claim. Proposed ownership addendum awaits independent plan review; added-option code is not implemented. |
|
[PHASE: PLAN] Implementation discovery requires a narrow addition to the approved pause plan: Redis and KV polling resolve their native listener after a visibility-based wait even with an admitted callback still running. Real/scaled50msdrain vs500msheld callback yields3owner reds and3default-control greens. This would end a pause barrier early. The locked addendum proposes optional provider-specific |
|
[PHASE: IMPL] [VERDICT: IN_PROGRESS] At2d7aa1044, S4–S6 are implemented: pause/resume generation barriers for every listener family, one25s total stop with19/5/1 defaults, shutdown-first-cause portable handback, validated project config + explicit runtime overrides, standalone5s grace, public docs and generated consumers. The actual Worker replacement oracle now covers all seven counting adapters beyond the delivery limit, with identity/header/count and final-depth controls. Queue full215/0/3existingoptional ignores; core112/0/1existingignore; original Worker/core113/0/0; final startup/supervisor25/25 and fixture controls13/13. Real startup/config/runner12/12 includes1.25s handler cleanup and both override directions with no static pool registry. A held registry lookup exposed the startup/pause ownership gap: red before/after-allocation tests precede immediate ownership transfer. AMQP abort fixture now models the actual native channel and strengthens cleanup ordering; seed fixture requires live directory recreation rather than a stale notification. Full plugin rerun in progress. Check/lint/fmt, quality:gate, preflight/accuracy, all three native dry-runs/audits and consumer freshness pass; exact public doc-lint baseline retained. Remaining S7: full supported one-pass scaffold runtime, independent feature IMPL-EVAL, cleanup, current required head checks and thread gate. No readiness/merge/publication action. |
|
[PHASE: IMPL] Implemented worker pause/resume, graceful work-before-abort stop with one monotonic total, shutdown-only delivery-count preservation on every counting backend, and authored/explicit grace forwarding through standard/generated startup. Default total25s=19s work+5s cleanup+1s settlement; standalone runner also uses5s grace. Stop is terminal for one Worker instance; parent budget needs at least5s headroom. Public docs and canonical consumers refreshed. Product source2d7aa10448700b46cb1c2dbb4665044eafb60dfa; follow-up33c034cb5 changes run artifacts only. Completed evidence: structured full plugin266pass/0fail/0ignored; queue215pass/0fail/3unchanged optional-backend ignores; core112pass/0fail/1unchanged ignore. All new backend cases execute.
Check/lint/fmt0; quality:gate/architecture fitnessPASS; release:preflight/docs:accuracyPASS; all three native JSR dry-runs/auditsPASS, no new slow types; doc-lint exact baseline core2/queue1/plugin2 unchanged, new config/shutdown0. Canonical site/prose/corpus/asset generation and consumer freshnessPASS. Docs: worker/core/queue READMEs and tune-worker-runtime.md list options, defaults, lifecycle, count rule, parent headroom and provider uncertainty. Pending mandatory gates: supported full one-pass scaffold runtime dispatched once at the product source, independent feature IMPL-EVAL, final head checks/thread verification and positively owned backend cleanup. This is still a draft; no merge, auto-merge or publication. |
|
[PHASE: IMPL-EVAL] [VERDICT: PASS] Independent feature IMPL-EVAL PASS at head Findings
External gate (not certified)Supported one-pass Next
Run dir: |
|
[PHASE: IMPL] Supported full runtime workflow43984548275 SUCCESS, attempt2, product source2d7aa10448700b46cb1c2dbb4665044eafb60dfa: Postgres124/124, SQLite116/116 +showcase8/8, static/native and all cleanup assertions intact. The first Postgres attempt failed during documented-scaffold db generate dashboard refresh; unchanged-main crossed evidence already records this symptom. Exactly one failed-job retry at unchanged source passed. No guard bypass, assertion removal or inferred worker regression/fix. Independent feature IMPL-EVAL PASS retained; only run docs changed since evaluated code. All nine acceptance mappings/issue boxes complete. Exact4owned containers and4native recorded volumes removed after maintained preview plus identity/mount/reference proof; zero survivors, foreign resources untouched. Pre-transition head4SUCCESS/21SKIPPED,0failures; thread gate0/0. Body/DoD updated truthfully. Transitioning ready so required non-draft head lanes run; their final results still must be confirmed before task completion. |
|
[PHASE: IMPL] Required ready-head CI found one failure among1753pass/1fail/21existingignores: maintained startup-log documentation guard. Locally reproduced1pass/1fail, then restored the exact existing in-process runner startup message accidentally omitted during lifecycle refactoring. Guard unchanged; documentation/intake/supervisor27/27green and scoped check/lint/fmt0, quality:gate/preflight/accuracy PASS. Three logging lines only; no control-flow, budget, generated wiring or API change. Existing independent implementation PASS and supported runtime remain behavior evidence; new-head required CI still must finish green. No resources started. |
|
[PHASE: IMPL] Final ready-for-review head445002122406a9accc96495c0dc80c72d6f6cc9d has19SUCCESS/13SKIPPED/0failures, including the formerly failing test-4 shard; review-thread gate0threads/0unanswered. Independent implementation PASS retained for unchanged lifecycle/drain/config behavior; subsequent delta restores only the exact existing three-line startup log, proven by the unchanged guard and adjacent27/27. Supported full runtime43984548275 attempt2 SUCCESS (Postgres124, SQLite116+showcase8, static/native/cleanup), all4owned containers/4volumes removed. All9acceptance mappings and DoD complete. Ready for independent review; no ready-merge, merge or publication. |
|
[PHASE: IMPL] Old head: Semantic review: worker stop remains 25 seconds within the child wrapper’s 30-second work window plus five-second cleanup reserve. Signal forwarding, health hostname, host disposal and singleton KV closure remain intact; the six composition-boundary files hash-match their respective parents. No behavior repair was made. Local gates PASS: check/lint/fmt (381 files); full workers (266 passed), queue (215 passed, three pre-existing ignores), core plus #2641 drain (136 passed); zero failures; quality/architecture, carrier chain/freshness, release:preflight, docs:accuracy, README fences; zero unanswered threads. Initial AMQP cookie/Garnet Lua fixture errors were corrected only in owned containers. One 50 ms Garnet lease-extension assertion failed under concurrent suite load; all six isolated cases and the final serial full queue run passed unchanged. All 6 owned containers and 5 original unshared volumes removed, zero survivors. Labels, PR body and issue checkboxes unchanged. |
Summary
Add worker pause/resume, a bounded graceful drain and configurable abort cleanup grace. Jobs returned during a controlled shutdown retain their delivery-attempt budget; ordinary handler failures continue to charge attempts.
Scope
Closes #2633
Closes #2635
NackOptions.preserveDeliveryCountdefaults false; atomic/fenced handback for counting adapters, confirmed publish before AMQP acknowledgement. Full decisions in the plan.Slices
Validation
Harness
.llm/runs/feat-worker-pause-drain--issues-2633-2635/.Drift / Debt
Task callbacks are covered by listener-generation barriers rather than the job-only active map. Portable native nack and AMQP acknowledgement/cancel behavior are necessary for shutdown handback. Focused KV polling extraction reduces the existing811-line adapter. Existing public documentation baseline measured; new diagnostics prohibited. No saga transport/Redis Streams, dependency/lock change or unrelated migration.
Definition of Done
Acceptance evidence
🤖 Generated with Codex