Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 31 additions & 1 deletion apps/scheduler/src/post-processor-dispatcher.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -116,10 +116,40 @@ describe("PostProcessorDispatcher", () => {

// Should have rolled back status for req-1
expect(collection.updateOne).toHaveBeenCalledWith(
{ _id: "req-1" },
{ _id: "req-1", "run._id": "run-1", "run.postProcessorStatus": "queued" },
{ $unset: { "run.postProcessorStatus": "" } },
);
// Should have stopped the batch (req-2 never sent)
expect(queueClient.sendMessage).toHaveBeenCalledTimes(1);
});

it.each([
{ runId: "run-2", status: "queued", reason: "a newer attempt was queued" },
{ runId: "run-1", status: "processing", reason: "the worker already claimed the message" },
{ runId: "run-1", status: "done", reason: "the worker already completed the message" },
])("preserves status after a send failure when $reason", async ({ runId, status }) => {
db._findOne.mockResolvedValue({ _id: "post-processor", version: 1 });
collection.findOneAndUpdate.mockResolvedValueOnce({
_id: "req-1", run: { _id: "run-1", postProcessorStatus: "queued" },
});

// Model a retry or delivery racing the failed send. Apply the rollback
// only if its Mongo predicates match the document that exists now.
const current: Record<string, string> = {
_id: "req-1", "run._id": runId, "run.postProcessorStatus": status,
};
collection.updateOne = vi.fn(async (filter: Record<string, unknown>) => {
const matches = Object.entries(filter).every(([key, value]) => current[key] === value);
if (matches) delete current["run.postProcessorStatus"];
return { matchedCount: matches ? 1 : 0 };
});
queueClient.sendMessage.mockRejectedValueOnce(new Error("send acknowledgement lost"));

const dispatcher = new PostProcessorDispatcher(collection, db, queueClient, 60_000);
await (dispatcher as any).dispatch();

expect(collection.updateOne).toHaveBeenCalledOnce();
expect(current["run.postProcessorStatus"]).toBe(status);
expect(queueClient.sendMessage).toHaveBeenCalledTimes(1);
});
});
12 changes: 9 additions & 3 deletions apps/scheduler/src/post-processor-dispatcher.ts
Original file line number Diff line number Diff line change
Expand Up @@ -89,12 +89,18 @@ export class PostProcessorDispatcher {
await this.queueClient.sendMessage(message);
dispatched++;
} catch (err) {
// Roll back status so the doc is picked up on next poll cycle
// Release only our still-queued attempt. A retry may have replaced
// the run, or a worker may have received the message despite a lost
// acknowledgement; neither state belongs to this rollback.
await this.collection.updateOne(
{ _id: claimed._id } as any,
{
_id: claimed._id,
"run._id": claimed.run?._id,
"run.postProcessorStatus": "queued",
},
{ $unset: { "run.postProcessorStatus": "" } } as any,
);
console.error(`[PostProcessorDispatcher] Queue send failed for ${claimed._id}, rolled back status:`, err);
console.error(`[PostProcessorDispatcher] Queue send failed for ${claimed._id}, attempted queued-status rollback:`, err);
break; // Stop batch — queue may be unavailable
}
console.log(`[PostProcessorDispatcher] Dispatched ${claimed._id}`);
Expand Down
7 changes: 6 additions & 1 deletion docs/architecture/post-processing.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,11 @@ Two paths trigger post-processing:
1. **Event-driven (primary):** Coder workers enqueue a post-processing message immediately when a run completes, setting `run.postProcessorStatus: "queued"` atomically in the same DB write.
2. **Scheduler backfill:** The `PostProcessorDispatcher` polls every 30 seconds for runs that have `status: "done"` but no `postProcessorStatus` set (or a `postProcessorVersion` below the current target). This catches runs that completed before the event-driven dispatch was deployed, or that need re-processing after a version bump.

If the scheduler's queue send fails, it releases only the claimed run attempt
while its post-processing status is still `queued`. A concurrent retry's newer
attempt is left untouched. The rollback also preserves `processing` or terminal
status when a worker received the message despite a lost send acknowledgement.

## ATIF Generation

[ATIF](https://github.com/AISafety-ATIF/specification) (AI Tool Interaction Format) is an open specification for recording AI agent tool interactions. The post-processor converts HAR (HTTP Archive) recordings captured during benchmark runs into ATIF trajectory files using the [`atifact`](https://github.com/waldekmastykarz/atifact) package (pinned to `^0.10.1`, which extracts trajectories from both HTTP/SSE exchanges and WebSocket exchanges in a HAR).
Expand Down Expand Up @@ -146,4 +151,4 @@ stateDiagram-v2
- **Blob Storage:** `snapshots` container (shared with HAR, tool-calls, and other artifacts)
- **Database:** MongoDB `requests` collection (`run.postProcessorStatus`, `run.postProcessorVersion` fields)
- **KEDA:** ScaledObject scales the worker to zero when queue is empty
- **Version registration:** Kubernetes Job runs on each deploy; Docker Compose uses an init service
- **Version registration:** Kubernetes Job runs on each deploy; Docker Compose uses an init service
Loading