Repository navigation
test: accept and order workflow run cancellation - #120
Conversation
92e6545 to
73aab1f
Compare
zhangshilong123
left a comment
There was a problem hiding this comment.
Approve(head 73aab1f)。两个新文件与运行计划逐条对应;a9c5f20..73aab1f 只有这两个测试文件(680 行新增),没有生产代码或过程记录。
HTTP 边界(integration/workflow_run_cancel_test.go)
- 请求只写
cancel_requested_at:用to_jsonb(r)去掉请求时刻、版本与updated_at做前后比对,其余列逐一不动;响应体只有cancelRequestedAt,且与库里读回的时刻相等;版本恰好 +1;workspaces / execution_work / thread_commands /delete_workspace均为 0。 - 幂等:同键与换键重放都返回首次记录的时刻,版本不再变;
idempotency_records里存的 202 响应就是调用者看到的时刻。 - 授权、范围与窗口:有冻结绑定时发起者/管理员 202、普通成员 403
workflow_run_cancel_forbidden、非成员 403membership_required,无绑定沿用成员规则;跨租户/跨工作流/未知/非 UUID/软删除 404;终态与awaiting_input409 且不落库;未知字段 400。 - 并发两个请求只记录一个时刻、只发生一次版本变化。
- HTTP 写入的请求被
AdvanceWorkflowRunsOnce消费:pending 直接结算cancelled,无started_at、无 Workspace、无未注册执行工作、无删除操作;结算后重放仍返回原时刻,后续推进不再改版本。
并发顺序(internal/core/workflow_run_cancel_order_db_test.go)
- 与调度推进并发:无论谁先,运行保持
running,只释放一条end_session{cancelled}且 run/execution/node 指向正确,会话自己上报后才结算cancelled;Workspace 删除声明恰好 1 条、请求时刻保留、终态后推进不改版本。 - 与会话终态回调并发:接受时运行
cancelled且节点保留实际结果;拒绝时不落库、运行保持succeeded。 - 与结束确认窗口到期 + 迟到回调并发:两种允许终态都断言,迟到结果留在
node_executions.result,Workspace 释放、请求保留。 - 断言的是全部允许的最终行而非常见结果,符合 G-026 的口径;goroutine 内用
t.Errorf而非Fatalf,-race前提成立。
核对
- 与实现逐条对照(
internal/core/workflow_run_cancel.go):202;404/409 的判定顺序(授权先于窗口);end_session的 reason;确认窗口按命令created_at的数据库时钟(测试老化 6 分钟 > 默认 5 分钟);runWorkspaceNodeUnknown先于窗口,均一致。 - 根 ADR 状态写在描述里(保持
approved),本 PR 无 specs 改动,符合计划第 3 条。
非阻塞
- 分支基于
a9c5f20,CI(03:08 启动)的合并树不含 03:11 合并的 #118。#118 只改了runWorkspaceNodeUnknown的谓词与断连心跳;这里 fixture 的节点始终connected且心跳新鲜,两版都判定为可达,结果不受影响。我已把两个文件应用到当前 main4128d86上编译通过(go test -run '^$' ./internal/core ./integration)。合并前 rebase 后重跑一次 CI 更稳,但不阻塞。 - 两个 loop 用例各跑 4 次以覆盖两种顺序,但没有记录实际命中哪一种。作为 G-026 的顺序证据,建议把每次的分支
t.Logf出来(或累计计数),让证据可审计;不影响正确性。
73aab1f to
332e534
Compare
zhangshilong123
left a comment
There was a problem hiding this comment.
Approve(新 head 332e534)。相对上次审的 73aab1f,只改了这两个测试文件;rebase 与增量都核对过。
rebase:df9799c 是 head 的祖先;df9799c..332e534 只有 4 个提交、两个测试文件(710 行新增),没有生产代码或过程记录。check 在 332e534 上 SUCCESS(真实 PostgreSQL 全量 + -race)。改动全部是交付物。
新增的 S5 断言与实现一致(cloud#119 的 internal/core/workflow_run_release.go)
- 取消前未创建 Workspace:终态结算的同一事务写
not_required、error 为空、workspace_released_at为 NULL——对应declareWorkflowRunWorkspaceRelease的「无 Workspace 行」分支,不为未发生的删除伪造释放时刻。 - 有活跃 Workspace 的三个并发用例:终态后为
releasing、无 reason、workspace_released_at为 NULL,且删除完成前不出现released——与「只有 delete 操作成功才写released」的状态机一致;releasing → abandoned需要 Node 失联超过放弃窗口,这里 Node 始终 connected 且心跳新鲜,不会误命中。 - 新断言接在
assertRunWorkspaceReleased之后,保留「恰好 1 条delete_workspace」的原断言;readWorkflowRelease是 S5 已有辅助函数。
t.Logf:两个 loop 用例在每条允许分支记录 attempt 与实际命中顺序,CI 输出可审计。本机回调竞态 4/4 是「请求先提交」、确认窗口竞态 4/4 是「pass 先结算」,互补分支由各自的一次性用例(已提交结果优先、窗口到期结算)确定性覆盖。
非阻塞说明:head 的 CI 合并树是 df9799c;其后 main 又合入 cloud#122(48b77fc,06:24Z)。#122 把 settleWorkflowRunStates 改成返回 CAS 结果、失败时跳过 cleanup,但这里每次结算都在同一 advisory lock 内用刚读到的版本做 CAS,必中;新增的 projectWorkflowRunToIssueRun 对没有 workflow_invocation_ref 的直接运行是 no-op。我已把两个文件应用到 48b77fc 上编译通过。合并前顺手 rebase 再跑一次更稳,不阻塞。
332e534 to
527af0a
Compare
Drive POST /workflows/:wfid/runs/:rid/cancel through the real router and PostgreSQL: the request records one database instant and touches nothing else, replays under either key return that instant, the initiator-or-admin rule and the run scope are enforced, terminal runs answer 409, concurrent requests record once, and the request written over HTTP is what the scheduler pass consumes to settle a pending run cancelled without starting. Refs ora-space/circus#403 Co-Authored-By: Claude Code <noreply@anthropic.com>
Race the cancellation writer against the scheduler pass, against a session terminal callback, and against the end-confirmation window expiring with a late callback. Each case asserts the complete set of permitted final rows: the run is never pulled back into execution, a committed result is kept, a late result survives on the execution row, the Workspace deletion is declared, and the recorded request is retained. Refs ora-space/circus#403 Co-Authored-By: Claude Code <noreply@anthropic.com>
Terminal settlement now records the Workspace's release state in the same transaction. The cancellation evidence checks that half of the row too: a cancelled-before-start run (no Workspace) reports not_required with no release instant, and every live-session ordering case reports releasing with no reason or released_at until the delete completes. Co-Authored-By: Claude Code <noreply@anthropic.com>
The two ordering loops permit either interleaving and asserted the complete set of allowed final rows, but did not say which one occurred. Logging the branch makes the evidence auditable after a CI run instead of inferred from timing. Co-Authored-By: Claude Code <noreply@anthropic.com>
527af0a to
bbfe02f
Compare
zhangshilong123
left a comment
There was a problem hiding this comment.
Approve(head bbfe02f)。
增量:相对我上次 Approve 的 332e534,两个测试文件逐字相同(git diff 332e534 bbfe02f -- integration/workflow_run_cancel_test.go internal/core/workflow_run_cancel_order_db_test.go 为 0 行);相对 be5fcf5 也只有这两个文件,无生产代码、无迁移、无契约变更,符合 #403「PR 只含测试交付物」。
真实 HTTP 边界(integration,6 例):全部经真实路由 POST /api/v1/tenants/:tid/workflows/:wfid/runs/:rid/cancel 与 PostgreSQL。核对通过的点:
- 响应体只有
cancelRequestedAt,且与落库时刻相等;请求只改cancel_requested_at、version与updated_at,其余列(含 status、时间戳、node_states)逐字不变;不产生 Workspace、execution work、thread command 或派生操作。 - 幂等:同键与换键重放都返回同一时刻,且
idempotency_records里存的是调用方看到的 202 与同一个 body。 - 授权与范围:发起者/管理员/普通成员、跨租户、跨工作流、未知 id、软删除、终态 409
run_not_cancellable,拒绝时不留下请求。 - pending 运行的请求被
AdvanceWorkflowRunsOnce的真实收尾路径消费:cancelled、不写started_at、不建 Workspace、不派生删除;S5 记为not_required且不虚构released_at。 - 并发两请求只记录一次、只 bump 一次版本。
顺序证据(core,3 例):断言的是完整允许终态集合而非常见路径——运行不被拉回执行中、已提交结果保留、迟到结果留在 node_executions、释放意图为 releasing 且 released_at 为空、结算后版本不再被后续 pass 改动。cloud#114 的「CAS 命中才清理」已核对:三例的结算都是该运行的首次结算,CAS 必命中,断言不需要放宽,实际也未放宽。
G-026 顺序日志:run 39570203394 里可审计——迟到回调竞态 attempt 0/1「pass 读到已提交结果、节点保留 idle_timeout」、attempt 2/3「pass 先结算、晚到结果留在执行行」;回调竞态 4/4 命中「请求先提交、回调结果结算节点、运行 cancelled」。补充一点供审计:回调竞态的另一个允许分支(回调先结算、请求 409)在 CI 里没有出现,它由 TestWorkflowRunCancelAfterSettlementIsRefused 的 succeeded 子例确定性地覆盖,不是缺口。
CI:run 39570203394 check pass(10m45s;integration 446.665s、internal/core 203.852s,均在 cloud#126 的 20 分钟上限内),本次新增用例全部 PASS。该 check 的合并树是 abbd544(be5fcf5 + bbfe02f);之后 main 前进到 3e4c5d8(cloud#117,只改 agent_run_residuals 迁移与 agent 启动预算,与工作流取消路径无交集),不影响本次增量的结论,合并时并入当前 main 即可。
ADR:decisions/cloud/workflow/0-cloud-side-scheduler-owns-workflow-execution.md 保持 approved;本 PR 只补测试证据,specs 里的证据登记按 #403 计划由后续运行承接。
Approve。
Adds the missing real-HTTP acceptance for workflow-run cancellation, and the G-026 ordering evidence for cancellation against the scheduler, the session callback and cleanup.
What changed
integration/workflow_run_cancel_test.godrivesPOST /api/v1/tenants/:tid/workflows/:wfid/runs/:rid/cancelthrough the real router and PostgreSQL: the request records one database instant and touches nothing else, replays under the same and a fresh idempotency key return that instant, the initiator-or-admin rule and the run scope are enforced, terminal runs answer409 run_not_cancellable, two concurrent requests record once, and the request written over HTTP is what the scheduler pass consumes to settle a pending runcancelledwithout starting it. The settlement also has to record S5's release fact:not_requiredwhen the run never provisioned a Workspace, with no release instant invented.internal/core/workflow_run_cancel_order_db_test.goraces the cancellation writer against the scheduler pass, against a session terminal callback, and against the end-confirmation window expiring with a late callback. Each case asserts the complete set of permitted final rows: the run is never pulled back into execution, a committed result is kept, a late result survives on the execution row, the Workspace deletion is declared, the S5 release fact is recorded (releasingwith no reason and noreleased_atwhile the delete is outstanding, neverreleasedbefore Cloud observes it), and the recorded request is retained. Each ordering attempt logs which branch it hit, so the G-026 evidence can be audited from the test output.No production code changes.
Evidence
REQUIRE_POSTGRES=1 TEST_DATABASE_URL=… go test ./integration -run TestWorkflowRunCancel -count=1REQUIRE_POSTGRES=1 TEST_DATABASE_URL=… go test ./internal/core -run TestWorkflowRunCancel -count=1-race.ADR status
specs/decisions/cloud/workflow/0-cloud-side-scheduler-owns-workflow-execution.mdremainsapproved: this PR adds tests only and implements no new decision.Refs ora-space/circus#403
🤖 Generated with Claude Code