Skip to content

fix(post-processor): isolate updates to the queued run attempt - #1539

Open
Naveen Chatlapalli (1aifanatic) wants to merge 1 commit into
microsoft:mainfrom
1aifanatic:fix/1051-postprocessor-attempt-gate
Open

Naveen Chatlapalli (1aifanatic) wants to merge 1 commit into
microsoft:mainfrom
1aifanatic:fix/1051-postprocessor-attempt-gate

Conversation

@1aifanatic

Copy link
Copy Markdown

Fixes #1051.

Post-processing messages already carry a run ID, but the worker and ATIF handler previously updated the request by document ID alone. A stale message could therefore mark a retried attempt as post-processed or write its ATIF URL onto the new attempt.

The worker now claims and updates only the completed run._id named in the message. The ATIF handler reads turns and writes atifUrl under the same attempt filter. Stale messages are discarded before their handler runs; if a retry replaces the run during processing, the worker skips the completion stamp and report trigger.

Validation: 9 focused post-processor/ATIF tests passed; pnpm --filter telemetry build and pnpm --filter post-processor build passed; git diff --check passed. The repository's post-processor lint script could not run because eslint is not installed in this checkout.

@manekinekko Wassim Chegham (manekinekko) added area: post-processing ATIF generation and the run-output post-processing pipeline. area: worker Coding agent worker runtime, task execution, and lifecycle. community-contribution Pull requests submitted by external community contributors. language: javascript Work involving JavaScript code, tooling, or dependencies. topic: reliability Availability, fault tolerance, recovery, and consistent operation. topic: testing Test coverage, test infrastructure, and validation quality. type: bug Incorrect behavior or a regression that needs fixing. type: documentation Documentation additions, corrections, and improvements. labels Nov 6, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you Naveen Chatlapalli (@1aifanatic) for fixing the stale-attempt database writes. The shared attempt filter on the worker claim/completion/failure paths and the ATIF read/write paths is a clear improvement.

I reviewed 6868869 and reproduced one remaining attempt-isolation gap at the report-trigger boundary, described inline. The guarded completion write prevents a retry that has already happened, but the subsequent awaited logging/HTTP steps leave another retry window before report creation. This is a remaining part of the pre-existing race, not a new regression introduced by adding the filters.

Validation using Node 22.22.0:

  • pnpm exec vitest run apps/workers/post-processor/src/post-processor.test.ts apps/workers/post-processor/src/handlers/atif-handler.test.ts --reporter=default — 8 tests passed.
  • pnpm exec vitest run apps/workers/post-processor/src/handlers/atif-handler.websocket.test.ts --reporter=default — 1 test passed, including the real HAR-to-ATIF fixture conversion.
  • pnpm --filter telemetry build, pnpm --filter shared build, and pnpm --filter post-processor build — all passed.
  • A scratch test of the actual worker replaced the attempt during the awaited completion log, after the completion update matched. The worker still posted {requestId} while the current attempt was the newer pending run. The API trigger route looks up only requestId and does not check the completed attempt; an always template remains eligible.

Please carry the originating run identity across that API boundary and add the retry-after-completion-write case, rather than relying on another client-side read before the POST. Keep the docs' report-isolation guarantee aligned with the actual endpoint contract. This is internal worker/API behavior, so Portal screenshots, wording, and Storybook tests are not applicable. Live Azure queue/database race testing was not performed.

Thanks again for addressing the data-corruption risk and for documenting the intended invariant.


// Trigger report generation now that enrichment is done (best-effort).
try {
await this.triggerReportGeneration(doc._id, log);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Carry the originating attempt through report triggering too. There is still a retry window after completed.matchedCount succeeds: this branch awaits logging before calling triggerReportGeneration, and that request sends only { requestId }. /api/v1/reports/trigger then fetches the latest run by request ID without checking the originating run ID or completed status.

I reproduced a retry replacing run A with pending run B during the awaited "Post-processing complete" log. Both guarded updates for A matched, but A's worker still posted the trigger while B was current. An always template can create a report for that incomplete/new attempt, and HTTP retries can hit the same gap.

Please include the message's run ID in the trigger contract and enforce attempt identity/readiness where the API resolves the run and creates the report; a fresh worker-side check alone would leave another TOCTOU window. Add a regression for retry after the successful completion stamp and before the trigger is handled. The database filters here are good; this is the remaining part of the attempt-isolation fix, not a claim that those new filters caused the race.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: post-processing ATIF generation and the run-output post-processing pipeline. area: worker Coding agent worker runtime, task execution, and lifecycle. community-contribution Pull requests submitted by external community contributors. language: javascript Work involving JavaScript code, tooling, or dependencies. topic: reliability Availability, fault tolerance, recovery, and consistent operation. topic: testing Test coverage, test infrastructure, and validation quality. type: bug Incorrect behavior or a regression that needs fixing. type: documentation Documentation additions, corrections, and improvements.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Post-processor updates run by _id only, not gated on run._id (retry race)

2 participants