Repository navigation
fix(scheduler): preserve newer attempts on dispatch rollback - #1615
Naveen Chatlapalli (1aifanatic) wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are recorded for the attempt-scoped rollback change.
No open issues 🎉
What changed in this PR
This PR protects post-processing state when the scheduler’s queue send fails, complementing the worker-side attempt isolation in #1539.
Changes:
- Limits rollback to the claimed run attempt while its post-processing status remains
queued. - Adds regression tests for newer attempts and worker-advanced statuses.
- Documents the rollback contract and adjusts logging to avoid claiming a rollback definitely occurred.
| File | Description |
|---|---|
docs/architecture/post-processing.md |
Documents attempt-scoped rollback and preservation of advanced statuses. |
apps/scheduler/src/post-processor-dispatcher.ts |
Adds rollback predicates and more accurate failure logging. |
apps/scheduler/src/post-processor-dispatcher.test.ts |
Verifies rollback filters and preservation of concurrent state changes. |
Review effort: Balanced
Wassim Chegham (manekinekko)
left a comment
There was a problem hiding this comment.
Thank you Naveen Chatlapalli (@1aifanatic) for this focused rollback fix and for complementing the worker-side attempt isolation in #1539.
I reviewed e37040b, including the scheduler's claim/send/failure flow, eligibility for the next poll, concurrent worker progress, tests, and documentation. I did not find a blocking code defect in this diff. Restricting rollback to the captured attempt while its status is still queued correctly preserves a replacement attempt and a worker that has already advanced the status; stopping the batch and saying "attempted" rollback also match the actual behavior.
Validation using Node 22.22.0:
pnpm exec vitest run apps/scheduler/src --reporter=dot— 3 files / 38 tests passed, including the 8 dispatcher cases.pnpm --filter scheduler exec tsc --noEmit— passed after dependency-ordered telemetry/shared builds.
There is a separate merge-readiness issue in the current GitHub check results: License Headers failed for website/src/styles/contributions.css and website/src/styles/contributors.css, with CI Summary failing as a consequence. Neither file belongs to this PR's three-file diff, so I am not attributing that failure to this patch or asking you to bundle an unrelated fix. The current base/CI result needs reconciliation or a fresh run before treating the PR as fully green.
The concurrency tests use mocked MongoDB/queue clients; I did not run live lost-acknowledgement tests against Azure Queue/CosmosDB. No UI/API/schema change is introduced here, so additional UI stories or migration documentation are not needed. I appreciate the narrow scope and the regression cases.
Summary
When a post-processing queue send fails, the scheduler rolls back
run.postProcessorStatususing only the request ID. If a retry has replaced the run during that send, the rollback erases the newer attempt's status. A lost send acknowledgement can also erase a status already advanced by the worker.Restrict rollback to the claimed
run._idand a still-queuedstatus. The batch still stops after the failed send, and an undelivered queued attempt remains eligible for the next poll. Document the rollback contract and avoid claiming in the log that a rollback definitely happened when the filter may match nothing.Related to #1051; #1539 addresses worker/ATIF writes, while this change addresses the scheduler's send-failure path.
Demos
N/A — internal scheduler state handling; no Portal or CLI interface changes.
Testing
pnpm exec vitest run apps/scheduler/src— 38 tests passed.pnpm --filter scheduler exec tsc --noEmit— passed.git diff --check— passed.Documentation and compatibility
Updated
docs/architecture/post-processing.md. No schema, API, CLI, dependency, or migration changes.Checklist