Skip to content

feat(core): settle an in-flight workflow session whose Node is gone - #91

Merged
EricWvi merged 3 commits into
ora-space:mainfrom
Kyber5323:feat/242-workflow-node-unreachable-settlement
Oct 20, 2026
Merged

EricWvi merged 3 commits into
ora-space:mainfrom
Kyber5323:feat/242-workflow-node-unreachable-settlement

Conversation

@Kyber5323

Copy link
Copy Markdown
Contributor

本 PR 做什么

工作流运行的 agent 节点在会话执行期间以 running 提交,调度器一直等 Node 上报才结算。Node 再也报不出消息时——心跳超过部署的窗口,或者沙箱连同其上的 Node 一起消失(强制停止留下的正是这个状态)——这个等待永远不会结束:运行停在 running,它的 Workspace 也永远不会被释放。

本 PR 把这种"已派发、无结果"收敛成有界结算:

  • 一份判断:runWorkspaceNodeUnknown 从 Store 方法改为自由函数,绑定条件按运行类型泛化(issue_run_id 或 workflow_run_id,0039 的 XOR 保证二者互斥),IssueRun 的 D5/D8 与工作流调度器问的是同一份 SQL。判断本身(极性、数据库时钟、未配置窗口按失败关闭)一字未改。
  • 窗口复用:窗口从推进轮次传入(Store.DeliveryUnreachableAfter),不新增第二个上限——两个 pass 不会对"同一个 Node 还在吗"给出不同答案。未配置(≤0)时会话保持等待而不是被误判。
  • 结算:超窗的节点以 Cloud 自己的原因码 node_unavailable 结算失败,运行走既有的失败传播进入终态并声明 Workspace 删除。该节点没有已提交结果,因此永不被记为成功(不变量 5)。刻意不用 Node 上报的会话结束原因(interrupted、agent_failed):这里没有任何东西上报过,Cloud 的判断必须和 Node 的报告可区分。
  • 迟到的真实结果仍然落在 node_executions.result 上(那是"实际跑过什么"的唯一记录),但终态运行不会被它改写。

测试

用例 断言
TestWorkflowSessionNodeGoneSettlesTheNodeFailed 心跳超窗、强制停止两种形态:节点与运行都以 node_unavailable 失败,Workspace 删除已声明
TestWorkflowSessionNodeSilenceWithinWindowHolds 窗口内沉默保持 running,越过窗口才结算——判据是可达性不是时长
TestWorkflowSessionUnconfiguredWindowHoldsTheNode 未配置窗口按失败关闭:沙箱已消失也保持等待
TestWorkflowSessionResultAfterTheNodeSettledIsKept 结算后到达的真实结果不移动运行状态、版本与节点状态,但保留在执行行上

把结算逻辑临时关掉重跑,前三个用例会以"运行停在 running"失败——它们确实覆盖这个缺口。

ADR 状态

实现 decisions/cloud/workflow/0-cloud-side-scheduler-owns-workflow-execution.md(approved)D1 条件② 与 D4 的一部分。该 ADR 的取消、恢复与成果资源收尾由 ora-space/circus#75 的后续切片承接,因此本 PR 不改 ADR 状态(保持 approved)。specs 侧 test-cases/cloud/workflow/scheduler-serialism-and-recovery.md 证据表的更新("无已提交结果的节点不进入成功状态"一节)在合并后单独提 PR,引用本 PR 的合并结果。

门禁

本地(Windows):task format:check、task lint(0 issues)、task build 全绿;go test ./internal/core/ 全绿(57s)。go test ./... 另有 cmd/devsetup、internal/gateway、internal/gitpreset、internal/gitserver、integration 的失败,逐条与基线 commit 1b0d408 一致(缺 git http-backend、Windows 文件权限位等环境原因),与本改动无关。

净改动文件

cmd/server/main.go
docs/operations/configuration.md         docs/operations/configuration.en.md
docs/workflows/workflows.md              docs/workflows/workflows.en.md
internal/config/config.go
internal/core/agent_run_release.go
internal/core/store.go
internal/core/workflow_run_session.go    internal/core/workflow_run_session_db_test.go
internal/core/workflow_scheduler.go

Refs ora-space/circus#242

🤖 Generated with Claude Code

@Kyber5323

Copy link
Copy Markdown
Contributor Author

@Zpc001 请 review 这个 PR:WorkflowRun 已派发无结果时的有界结算(S1 / circus#242)。核心是复用 runWorkspaceNodeUnknown 的可达性判断、按 node_unavailable 结算,以及结算后迟到结果不改写终态。通过后请 Approve;合并要等 #70 关闭、运行 #242 转成 run 后按结论核对。

@Zpc001 Zpc001 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. The settlement is correct and well-scoped: runWorkspaceNodeUnknown is generalized to both run kinds through 0039's XOR binding without changing its judgement polarity, the window is threaded from the scheduler rather than introducing a second limit, node_unavailable reuses the published operation failure code, and the four tests cover the claimed gaps (heartbeat-past-window, force-stop sandbox-gone, within-window hold, unconfigured-window hold, late-result-kept). Docs mirror in both languages and the ADR status / net changed files are stated correctly. Approving.

@Kyber5323

Copy link
Copy Markdown
Contributor Author

冲突已解决,head 更新为 4ee609d(把 upstream/main 合入分支,其中包含刚合并的 cloud#96)。

两处冲突及处理:

  • internal/core/workflow_run_session.go:保留两个行为并明确顺序。已派发但未终态的会话先问共享的 runWorkspaceNodeUnknown:Node 不可达时按 node_unavailable 结算失败(本 PR 的 S1 行为);Node 仍可达时交给上游的单回合 EndSession 路径——成功 turnEnded 后放出命令,确认窗口有界,超窗以 session_end_deadline_exceeded 失败。
  • docs/workflows/workflows.en.md:中英文文档同步保留两段行为,删掉重复的 Workspace 释放句。

本地门禁:task format:check、task lint(0 issues)、task build、真实 PostgreSQL 的 go test ./internal/core/ 全绿;PR CI 正在跑。

因为合并提交改变了与 cloud#96 的交互,@Zpc001 请再 review 一次这个合并结果。

Kyber5323 and others added 3 commits October 20, 2026 14:36
…both run kinds

runWorkspaceNodeUnknown only ever read its window argument, so it does not need
the Store receiver, and the binding it tests can cover both kinds of run: a
Workspace is tied to its run by issue_run_id or by workflow_run_id, which 0039
makes mutually exclusive, so one comparison answers "is this Workspace bound to
this run?" without the caller having to say which kind it holds.

Nothing about the judgement changes — its polarity, its database clock and its
fail-closed window stay exactly as documented — but it is now the one place the
heartbeat evidence behind "the Node is unreachable" is read, so the workflow
scheduler's session settlement can ask it instead of defining a second one.

Co-Authored-By: Claude Code <noreply@anthropic.com>
A workflow run's agent node is committed `running` while its session executes,
and the pass holds it until the Node reports. When the Node can no longer report
— its heartbeat has been stale past the deployment's window, or its sandbox is
gone, which is the state a force stop leaves behind — holding the run waits
forever for evidence that cannot arrive: the run stays `running` and its
Workspace is never released.

The settlement now asks the judgement the IssueRun release paths already make
(runWorkspaceNodeUnknown) with the same window, so "the Node is unreachable" has
one definition across run kinds, and fails the node under Cloud's own code
`node_unavailable` rather than borrowing a Node-reported session end reason:
nothing reported anything here, and the two must stay distinguishable. No
committed result exists for the node, so it is never recorded succeeded
(invariant 5), and the run follows the existing failure propagation into its
terminal state and Workspace deletion.

The window is threaded from the pass (Store.DeliveryUnreachableAfter) instead of
introducing a second limit; an unconfigured window fails closed and holds the
session, exactly as it does for D5 and D8.

Tests: both shapes of a gone Node, silence inside the window holding, an
unconfigured window holding, and a late real result staying on the execution row
without rewriting the terminal run.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…ht sessions

The scheduling section named the reasons a session fails to start; add the one
it fails with when its Node is gone, including the window it reads and the fact
that `node_unavailable` is Cloud's own diagnosis rather than a session end
reason a Node reported. The configuration table now says
`delivery_unreachable_after` is also the "is the Node still there" window, since
a deployment that sets it sets both.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@Kyber5323
Kyber5323 force-pushed the feat/242-workflow-node-unreachable-settlement branch from 4ee609d to db0cddd Compare October 20, 2026 06:40
@Kyber5323

Copy link
Copy Markdown
Contributor Author

按 owner 要求改为 rebase:分支现在基于最新 upstream/main(4718aba,含 cloud#96 与 cloud#99),head 更新为 db0cddd,共 3 个 commit。

workflowAgentSessionOutcome 的同一分支里组合了两个行为:

  1. 已派发但未终态时,先问共享的 runWorkspaceNodeUnknown:Node 不可达(心跳超窗或沙箱连同 Node 消失)→ 按 node_unavailable 结算失败并结束运行(本 PR 的 S1)。
  2. Node 仍可达 → 走上游的单回合 EndSession 路径:成功 turnEnded 后放出命令,确认窗口有界,超窗以 session_end_deadline_exceeded 失败。

两处冲突(internal/core/workflow_run_session.go、docs/workflows/workflows.en.md)按上述顺序解决,中英文文档同步。PR 无迁移。

门禁(rebase 后的新 head,真实 PostgreSQL):

  • task format:check、task lint(0 issues)、task build 全绿。
  • 四个 S1 用例:TestWorkflowSessionNodeGoneSettlesTheNodeFailed、...NodeSilenceWithinWindowHolds、...UnconfiguredWindowHoldsTheNode、...ResultAfterTheNodeSettledIsKept 全过。
  • fix(core): 工作流会话在初始回合结束后由调度器结束 #96 的结束路径用例:...EndsAfterSuccessfulTurn、...CancelledTurnDoesNotRequestEnd、...EndRequestsAreScopedPerNode、...EndDeadlineFailsNode 全过。
  • 完整 go test ./internal/core/ 通过。

@Zpc001 请按新 head db0cddd 重新 review;旧 review:lgtm(7a6dbfc)按 owner 的说明已失效。

@Zpc001 Zpc001 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the rebased head (db0cddd, on upstream/main 4718aba with #96 and #99). The composition is correct: the dispatched-but-not-terminal branch first asks runWorkspaceNodeUnknown and settles node_unavailable when the Node is gone (S1), otherwise delegates to workflowSessionEndRequest (the #96 single-turn path). unreachableAfter threads through AdvanceWorkflowRunsOnce -> advanceWorkflowRun -> workflowAgentSessionOutcome with all call sites updated. The four S1 tests are intact and the #96 end-path tests still pass. CI is green. Approving.

@zhangshilong123 zhangshilong123 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

复审通过(head db0cddd,基于 upstream/main 4718aba,MERGEABLE/CLEAN,CI check 通过)。

核对:

  • runWorkspaceNodeUnknown 转自由函数、绑定条件泛化为 issue_run_id OR workflow_run_id,0039 的 XOR 约束保证互斥,D5/D8 与调度器问同一份 SQL,判断本体未改。
  • workflowAgentSessionOutcome 组合顺序正确:先问共享的 runWorkspaceNodeUnknown,Node 不可达 → node_unavailable 结算失败并结束运行;可达 → 委托 #96 的 workflowSessionEndRequest 单回合结束路径。
  • DeliveryUnreachableAfter 从 AdvanceWorkflowRunsOnce → advanceWorkflowRun → workflowAgentSessionOutcome 穿透一致,调用点全部同步。
  • 文档中英文同步,配置说明改为「判定窗口同时用于会话结算」。
  • 4 个 S1 用例覆盖缺口(超窗/沙箱消失、窗口内保持、未配置保持、结算后迟到结果保留)。

无迁移。合并仍由 owner 决定。

@EricWvi
EricWvi merged commit 068d671 into ora-space:main Oct 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants