Repository navigation
fix(post-processor): isolate updates to the queued run attempt - #1539
Naveen Chatlapalli (1aifanatic) wants to merge 1 commit into
Conversation
Wassim Chegham (manekinekko)
left a comment
There was a problem hiding this comment.
Thank you Naveen Chatlapalli (@1aifanatic) for fixing the stale-attempt database writes. The shared attempt filter on the worker claim/completion/failure paths and the ATIF read/write paths is a clear improvement.
I reviewed 6868869 and reproduced one remaining attempt-isolation gap at the report-trigger boundary, described inline. The guarded completion write prevents a retry that has already happened, but the subsequent awaited logging/HTTP steps leave another retry window before report creation. This is a remaining part of the pre-existing race, not a new regression introduced by adding the filters.
Validation using Node 22.22.0:
pnpm exec vitest run apps/workers/post-processor/src/post-processor.test.ts apps/workers/post-processor/src/handlers/atif-handler.test.ts --reporter=default— 8 tests passed.pnpm exec vitest run apps/workers/post-processor/src/handlers/atif-handler.websocket.test.ts --reporter=default— 1 test passed, including the real HAR-to-ATIF fixture conversion.pnpm --filter telemetry build,pnpm --filter shared build, andpnpm --filter post-processor build— all passed.- A scratch test of the actual worker replaced the attempt during the awaited completion log, after the completion update matched. The worker still posted
{requestId}while the current attempt was the newer pending run. The API trigger route looks up only requestId and does not check the completed attempt; analwaystemplate remains eligible.
Please carry the originating run identity across that API boundary and add the retry-after-completion-write case, rather than relying on another client-side read before the POST. Keep the docs' report-isolation guarantee aligned with the actual endpoint contract. This is internal worker/API behavior, so Portal screenshots, wording, and Storybook tests are not applicable. Live Azure queue/database race testing was not performed.
Thanks again for addressing the data-corruption risk and for documenting the intended invariant.
|
|
||
| // Trigger report generation now that enrichment is done (best-effort). | ||
| try { | ||
| await this.triggerReportGeneration(doc._id, log); |
There was a problem hiding this comment.
[P2] Carry the originating attempt through report triggering too. There is still a retry window after completed.matchedCount succeeds: this branch awaits logging before calling triggerReportGeneration, and that request sends only { requestId }. /api/v1/reports/trigger then fetches the latest run by request ID without checking the originating run ID or completed status.
I reproduced a retry replacing run A with pending run B during the awaited "Post-processing complete" log. Both guarded updates for A matched, but A's worker still posted the trigger while B was current. An always template can create a report for that incomplete/new attempt, and HTTP retries can hit the same gap.
Please include the message's run ID in the trigger contract and enforce attempt identity/readiness where the API resolves the run and creates the report; a fresh worker-side check alone would leave another TOCTOU window. Add a regression for retry after the successful completion stamp and before the trigger is handled. The database filters here are good; this is the remaining part of the attempt-isolation fix, not a claim that those new filters caused the race.
Fixes #1051.
Post-processing messages already carry a run ID, but the worker and ATIF handler previously updated the request by document ID alone. A stale message could therefore mark a retried attempt as post-processed or write its ATIF URL onto the new attempt.
The worker now claims and updates only the completed
run._idnamed in the message. The ATIF handler reads turns and writesatifUrlunder the same attempt filter. Stale messages are discarded before their handler runs; if a retry replaces the run during processing, the worker skips the completion stamp and report trigger.Validation: 9 focused post-processor/ATIF tests passed;
pnpm --filter telemetry buildandpnpm --filter post-processor buildpassed;git diff --checkpassed. The repository's post-processor lint script could not run becauseeslintis not installed in this checkout.