Skip to content

test: accept and order workflow run cancellation - #120

Merged
EricWvi merged 4 commits into
ora-space:mainfrom
Kyber5323:feat/403-workflow-cancel-evidence
Oct 21, 2026
Merged

EricWvi merged 4 commits into
ora-space:mainfrom
Kyber5323:feat/403-workflow-cancel-evidence

Conversation

@Kyber5323

@Kyber5323 Kyber5323 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Adds the missing real-HTTP acceptance for workflow-run cancellation, and the G-026 ordering evidence for cancellation against the scheduler, the session callback and cleanup.

What changed

  • integration/workflow_run_cancel_test.go drives POST /api/v1/tenants/:tid/workflows/:wfid/runs/:rid/cancel through the real router and PostgreSQL: the request records one database instant and touches nothing else, replays under the same and a fresh idempotency key return that instant, the initiator-or-admin rule and the run scope are enforced, terminal runs answer 409 run_not_cancellable, two concurrent requests record once, and the request written over HTTP is what the scheduler pass consumes to settle a pending run cancelled without starting it. The settlement also has to record S5's release fact: not_required when the run never provisioned a Workspace, with no release instant invented.
  • internal/core/workflow_run_cancel_order_db_test.go races the cancellation writer against the scheduler pass, against a session terminal callback, and against the end-confirmation window expiring with a late callback. Each case asserts the complete set of permitted final rows: the run is never pulled back into execution, a committed result is kept, a late result survives on the execution row, the Workspace deletion is declared, the S5 release fact is recorded (releasing with no reason and no released_at while the delete is outstanding, never released before Cloud observes it), and the recorded request is retained. Each ordering attempt logs which branch it hit, so the G-026 evidence can be audited from the test output.

No production code changes.

Evidence

  • REQUIRE_POSTGRES=1 TEST_DATABASE_URL=… go test ./integration -run TestWorkflowRunCancel -count=1
  • REQUIRE_POSTGRES=1 TEST_DATABASE_URL=… go test ./internal/core -run TestWorkflowRunCancel -count=1
  • Both also pass with -race.

ADR status

specs/decisions/cloud/workflow/0-cloud-side-scheduler-owns-workflow-execution.md remains approved: this PR adds tests only and implements no new decision.

Refs ora-space/circus#403

🤖 Generated with Claude Code

@zhangshilong123 zhangshilong123 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve(head 73aab1f)。两个新文件与运行计划逐条对应;a9c5f20..73aab1f 只有这两个测试文件(680 行新增),没有生产代码或过程记录。

HTTP 边界(integration/workflow_run_cancel_test.go)

  • 请求只写 cancel_requested_at:用 to_jsonb(r) 去掉请求时刻、版本与 updated_at 做前后比对,其余列逐一不动;响应体只有 cancelRequestedAt,且与库里读回的时刻相等;版本恰好 +1;workspaces / execution_work / thread_commands / delete_workspace 均为 0。
  • 幂等:同键与换键重放都返回首次记录的时刻,版本不再变;idempotency_records 里存的 202 响应就是调用者看到的时刻。
  • 授权、范围与窗口:有冻结绑定时发起者/管理员 202、普通成员 403 workflow_run_cancel_forbidden、非成员 403 membership_required,无绑定沿用成员规则;跨租户/跨工作流/未知/非 UUID/软删除 404;终态与 awaiting_input 409 且不落库;未知字段 400。
  • 并发两个请求只记录一个时刻、只发生一次版本变化。
  • HTTP 写入的请求被 AdvanceWorkflowRunsOnce 消费:pending 直接结算 cancelled,无 started_at、无 Workspace、无未注册执行工作、无删除操作;结算后重放仍返回原时刻,后续推进不再改版本。

并发顺序(internal/core/workflow_run_cancel_order_db_test.go)

  • 与调度推进并发:无论谁先,运行保持 running,只释放一条 end_session{cancelled} 且 run/execution/node 指向正确,会话自己上报后才结算 cancelled;Workspace 删除声明恰好 1 条、请求时刻保留、终态后推进不改版本。
  • 与会话终态回调并发:接受时运行 cancelled 且节点保留实际结果;拒绝时不落库、运行保持 succeeded。
  • 与结束确认窗口到期 + 迟到回调并发:两种允许终态都断言,迟到结果留在 node_executions.result,Workspace 释放、请求保留。
  • 断言的是全部允许的最终行而非常见结果,符合 G-026 的口径;goroutine 内用 t.Errorf 而非 Fatalf,-race 前提成立。

核对

  • 与实现逐条对照(internal/core/workflow_run_cancel.go):202;404/409 的判定顺序(授权先于窗口);end_session 的 reason;确认窗口按命令 created_at 的数据库时钟(测试老化 6 分钟 > 默认 5 分钟);runWorkspaceNodeUnknown 先于窗口,均一致。
  • 根 ADR 状态写在描述里(保持 approved),本 PR 无 specs 改动,符合计划第 3 条。

非阻塞

  1. 分支基于 a9c5f20,CI(03:08 启动)的合并树不含 03:11 合并的 #118。#118 只改了 runWorkspaceNodeUnknown 的谓词与断连心跳;这里 fixture 的节点始终 connected 且心跳新鲜,两版都判定为可达,结果不受影响。我已把两个文件应用到当前 main 4128d86 上编译通过(go test -run '^$' ./internal/core ./integration)。合并前 rebase 后重跑一次 CI 更稳,但不阻塞。
  2. 两个 loop 用例各跑 4 次以覆盖两种顺序,但没有记录实际命中哪一种。作为 G-026 的顺序证据,建议把每次的分支 t.Logf 出来(或累计计数),让证据可审计;不影响正确性。

@Kyber5323
Kyber5323 force-pushed the feat/403-workflow-cancel-evidence branch from 73aab1f to 332e534 Compare October 21, 2026 06:21

@zhangshilong123 zhangshilong123 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve(新 head 332e534)。相对上次审的 73aab1f,只改了这两个测试文件;rebase 与增量都核对过。

rebase:df9799c 是 head 的祖先;df9799c..332e534 只有 4 个提交、两个测试文件(710 行新增),没有生产代码或过程记录。check 在 332e534 上 SUCCESS(真实 PostgreSQL 全量 + -race)。改动全部是交付物。

新增的 S5 断言与实现一致(cloud#119 的 internal/core/workflow_run_release.go)

  • 取消前未创建 Workspace:终态结算的同一事务写 not_required、error 为空、workspace_released_at 为 NULL——对应 declareWorkflowRunWorkspaceRelease 的「无 Workspace 行」分支,不为未发生的删除伪造释放时刻。
  • 有活跃 Workspace 的三个并发用例:终态后为 releasing、无 reason、workspace_released_at 为 NULL,且删除完成前不出现 released——与「只有 delete 操作成功才写 released」的状态机一致;releasing → abandoned 需要 Node 失联超过放弃窗口,这里 Node 始终 connected 且心跳新鲜,不会误命中。
  • 新断言接在 assertRunWorkspaceReleased 之后,保留「恰好 1 条 delete_workspace」的原断言;readWorkflowRelease 是 S5 已有辅助函数。

t.Logf:两个 loop 用例在每条允许分支记录 attempt 与实际命中顺序,CI 输出可审计。本机回调竞态 4/4 是「请求先提交」、确认窗口竞态 4/4 是「pass 先结算」,互补分支由各自的一次性用例(已提交结果优先、窗口到期结算)确定性覆盖。

非阻塞说明:head 的 CI 合并树是 df9799c;其后 main 又合入 cloud#122(48b77fc,06:24Z)。#122 把 settleWorkflowRunStates 改成返回 CAS 结果、失败时跳过 cleanup,但这里每次结算都在同一 advisory lock 内用刚读到的版本做 CAS,必中;新增的 projectWorkflowRunToIssueRun 对没有 workflow_invocation_ref 的直接运行是 no-op。我已把两个文件应用到 48b77fc 上编译通过。合并前顺手 rebase 再跑一次更稳,不阻塞。

@Kyber5323
Kyber5323 force-pushed the feat/403-workflow-cancel-evidence branch from 332e534 to 527af0a Compare October 21, 2026 07:13
@Kyber5323 Kyber5323 closed this Oct 21, 2026
@Kyber5323 Kyber5323 reopened this Oct 21, 2026
Kyber5323 and others added 4 commits October 21, 2026 16:29
Drive POST /workflows/:wfid/runs/:rid/cancel through the real router and
PostgreSQL: the request records one database instant and touches nothing
else, replays under either key return that instant, the initiator-or-admin
rule and the run scope are enforced, terminal runs answer 409, concurrent
requests record once, and the request written over HTTP is what the
scheduler pass consumes to settle a pending run cancelled without starting.

Refs ora-space/circus#403

Co-Authored-By: Claude Code <noreply@anthropic.com>
Race the cancellation writer against the scheduler pass, against a session
terminal callback, and against the end-confirmation window expiring with a
late callback. Each case asserts the complete set of permitted final rows:
the run is never pulled back into execution, a committed result is kept, a
late result survives on the execution row, the Workspace deletion is
declared, and the recorded request is retained.

Refs ora-space/circus#403

Co-Authored-By: Claude Code <noreply@anthropic.com>
Terminal settlement now records the Workspace's release state in the same
transaction. The cancellation evidence checks that half of the row too: a
cancelled-before-start run (no Workspace) reports not_required with no release
instant, and every live-session ordering case reports releasing with no reason
or released_at until the delete completes.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The two ordering loops permit either interleaving and asserted the complete set
of allowed final rows, but did not say which one occurred. Logging the branch
makes the evidence auditable after a CI run instead of inferred from timing.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@Kyber5323
Kyber5323 force-pushed the feat/403-workflow-cancel-evidence branch from 527af0a to bbfe02f Compare October 21, 2026 08:33

@zhangshilong123 zhangshilong123 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve(head bbfe02f)。

增量:相对我上次 Approve 的 332e534,两个测试文件逐字相同(git diff 332e534 bbfe02f -- integration/workflow_run_cancel_test.go internal/core/workflow_run_cancel_order_db_test.go 为 0 行);相对 be5fcf5 也只有这两个文件,无生产代码、无迁移、无契约变更,符合 #403「PR 只含测试交付物」。

真实 HTTP 边界(integration,6 例):全部经真实路由 POST /api/v1/tenants/:tid/workflows/:wfid/runs/:rid/cancel 与 PostgreSQL。核对通过的点:

  • 响应体只有 cancelRequestedAt,且与落库时刻相等;请求只改 cancel_requested_at、version 与 updated_at,其余列(含 status、时间戳、node_states)逐字不变;不产生 Workspace、execution work、thread command 或派生操作。
  • 幂等:同键与换键重放都返回同一时刻,且 idempotency_records 里存的是调用方看到的 202 与同一个 body。
  • 授权与范围:发起者/管理员/普通成员、跨租户、跨工作流、未知 id、软删除、终态 409 run_not_cancellable,拒绝时不留下请求。
  • pending 运行的请求被 AdvanceWorkflowRunsOnce 的真实收尾路径消费:cancelled、不写 started_at、不建 Workspace、不派生删除;S5 记为 not_required 且不虚构 released_at。
  • 并发两请求只记录一次、只 bump 一次版本。

顺序证据(core,3 例):断言的是完整允许终态集合而非常见路径——运行不被拉回执行中、已提交结果保留、迟到结果留在 node_executions、释放意图为 releasing 且 released_at 为空、结算后版本不再被后续 pass 改动。cloud#114 的「CAS 命中才清理」已核对:三例的结算都是该运行的首次结算,CAS 必命中,断言不需要放宽,实际也未放宽。

G-026 顺序日志:run 39570203394 里可审计——迟到回调竞态 attempt 0/1「pass 读到已提交结果、节点保留 idle_timeout」、attempt 2/3「pass 先结算、晚到结果留在执行行」;回调竞态 4/4 命中「请求先提交、回调结果结算节点、运行 cancelled」。补充一点供审计:回调竞态的另一个允许分支(回调先结算、请求 409)在 CI 里没有出现,它由 TestWorkflowRunCancelAfterSettlementIsRefused 的 succeeded 子例确定性地覆盖,不是缺口。

CI:run 39570203394 check pass(10m45s;integration 446.665s、internal/core 203.852s,均在 cloud#126 的 20 分钟上限内),本次新增用例全部 PASS。该 check 的合并树是 abbd544(be5fcf5 + bbfe02f);之后 main 前进到 3e4c5d8(cloud#117,只改 agent_run_residuals 迁移与 agent 启动预算,与工作流取消路径无交集),不影响本次增量的结论,合并时并入当前 main 即可。

ADR:decisions/cloud/workflow/0-cloud-side-scheduler-owns-workflow-execution.md 保持 approved;本 PR 只补测试证据,specs 里的证据登记按 #403 计划由后续运行承接。

Approve。

@EricWvi
EricWvi merged commit eb5ae5b into ora-space:main Oct 21, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants