Repository navigation
fix(core): honor node unknown window after disconnect - #118
Merged
Merged
Conversation
Preserve the last connected heartbeat when receiving disconnected reports, and use that instant for the shared node-unknown window without filtering on connection state. Reconnecting resets the window; repeated disconnect reports cannot extend it. Keep stale connected heartbeat handling, no-live-sandbox delivery settlement, release cleanup, and explicit quiesce refusal behavior unchanged. The same window also applies to workflow session and cancellation settlement. Add real ReportNodeStatus and delivery/release regression coverage for both sides of the window, reconnects, repeated disconnect reports, and cleanup. Validation: task format, go run ./cmd/checkformat and task lint (0 issues). Race-selected integration cases: 11 pass, 0 skip, 0 fail. Full internal/core: 663 pass, 0 skip, 0 fail. Release|Delivery|GiveUp|NodeUnknown|Unreachable integration selection: 52 pass, 0 skip, 1 environment failure in the Git server test (requires git http-backend on Windows), retained per scope. Temporarily reverting both production files makes the within-window delivery and release regressions fail as expected. Refs ora-space/circus#397
Exercise workflow session settlement and cancellation through nodeStatus and the real workflow scheduler. Verify that a recent disconnect keeps the session running until the unreachable window, and that cancellation waits for the existing end-confirmation window. Document the workflow cancellation caller of the shared unknown-node judgement. Production timeout settings remain unchanged. Validation: format, checkformat, lint, focused race tests, the core suite and the requested integration selection. The Git server integration environment still lacks git http-backend on this Windows machine. Reverting the original fix exposes the premature workflow settlements. Refs ora-space/circus#397 Refs ora-space/circus#391
Kyber5323
approved these changes
Oct 21, 2026
Kyber5323
left a comment
Contributor
There was a problem hiding this comment.
review:Approve(head a25c0b65f01229854bd10b212019f8cfa2e5f583)
审阅版本:head a25c0b6(base b7e1801),对照当前 upstream/main a9c5f20,mergeable。依据 specs 根 ADR decisions/cloud/issue-run/0-agent-run-in-disposable-isolated-workspace.md(upstream/main 911c976)的 D5/D8。
核对通过:
- 计时起点改为「最后一次确认可达」:
nodeStatus只在connected报告时刷新last_seen_at,disconnected保留原值;runWorkspaceNodeUnknown不再要求当前connection_state='connected',断连后必须等满窗口。起点已在 PR 描述和运行 issue 中写明;根 ADR 没有更细的规定,这个选择合理,且比「按断连报告起算」略早、最多相差一个心跳间隔。 - 重复断连不延长窗口、重连重新计时;
connected但心跳过期、无存活 sandbox、明确拒绝 quiesce 三个既有分支行为不变。 - 其他 9 处生产代码读取
last_seen_at时都同时要求connection_state='connected',保留旧值不影响它们;node_idle仍刷新该时间(这份报告本身就是可达证据)。 - D5/D8、workflow 会话结算与取消共用同一判断;新增集成用例走真实
ReportNodeStatus与 delivery/release 路径,workflow 用例覆盖窗口内保持 running 与取消仍等 EndSession 确认。 - ADR 状态:根 ADR 已是
implemented,描述写明本 PR 只修既有实现、不改状态;无迁移;4 个改动文件均为交付物;两个 commit 按 fix / test 拆分。
非阻塞:与并行中的 cloud#119(workflow 释放放弃)无重复实现——#119 复用同一个 runWorkspaceNodeUnknown,会自动继承本修复;两个 PR 都改 agent_run_release.go 的相邻位置,后合并者按当时 main rebase 即可,无语义冲突。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
修改
修复 Node 刚被标记为
disconnected时,D5 立即放弃 Revision 交付、D8 随即放弃删除的行为。根 ADR IssueRun:一次运行一个 disposable isolated Workspace 已为implemented:D5 要求 Node 未知超过配置窗口才按不可达放弃交付;D8 共用该窗口(默认 30 分钟)。本 PR 修复既有实现,不修改 ADR 状态。nodeStatus只在 connected 报告时刷新已有last_seen_at,disconnected 报告保留原值。共享runWorkspaceNodeUnknown比较该时间时不再过滤connection_state。起点选择最后一次确认 Node 可达的数据库时间,因为断连通知只说明 Cloud 收到了失联消息,不能证明 Node 已重新可达;重复断连不能延长等待,重连则自然重新计时。无需新增迁移。connected 但心跳已超过窗口时仍沿用原有不可达判断;无存活 sandbox 时仍立即满足 D5 的未知条件;D8 仍保留无 sandbox 的 cleanup 和明确拒绝 quiesce 的行为。workflow 会话收尾与取消路径也共用该 helper,因此 disconnected 但时间仍在窗口内时不再立即满足不可达条件,connected/stale 行为及其他结束确认超时保持不变。其他 9 处生产时间判断均同时要求 connected,不受断连时间保留的影响。
Node 资源(OpenAPI 及 Controller snapshot)的
lastSeenAt在 Node 断连时现在表示最后一次 connected 报告的时间;仓库内没有业务代码根据这个资源字段作决策。测试
新增集成测试通过真实
ReportNodeStatus和 delivery/release 路径执行,数据库时间调整代替 sleep:TestNodeUnknownDisconnectedWithinWindowKeepsDeliveryAndRelease:delivering/releasing 两阶段在窗口内均继续等待,不提前删除或记录失败。TestNodeUnknownDisconnectedPastWindowSettlesDeliveryAndRelease:超过窗口后,D5 放弃交付并声明删除,D8 以node_unavailable结束删除并进入 done。TestNodeUnknownReconnectRestartsWindowButRepeatedDisconnectDoesNot:重连刷新起点,首次及重复断连均保留它,重新到期后放弃。TestNodeUnknownNoLiveSandboxReleasesDeliveryButPreservesCleanup:无存活 sandbox 时 D5 立即释放,删除进入 cleanup 后 D8 不放弃。TestG032_4_RefusedQuiesceNeverTriggersGiveUpEvenPastTheWindow保持不变并纳入 race 运行;原有 D5/D8 和 workflow connected/stale 测试保持不变。TestWorkflowSessionDisconnectedWithinWindowHoldsUntilExpired:通过 nodeStatus 断连后,workflow 会话在窗口内保持 running,超过不可达窗口后变为 failed。TestWorkflowRunCancelDisconnectedWithinWindowWaitsForEndConfirmation:新近断连后取消运行仍先等待 EndSession 确认,仅在原确认窗口到期后变为 cancelled。撤回修复后,窗口内的 D5/D8 会提前进入 releasing/done,workflow 会话会提前失败,取消也会跳过 EndSession 确认等待。
检查结果
测试使用
REQUIRE_POSTGRES=1:task formatgo run ./cmd/checkformattask lintgo test -race -p 1 ./integration -run 'TestNodeUnknown|TestG032_4_RefusedQuiesceNeverTriggersGiveUpEvenPastTheWindow' -count=1 -timeout 60mgo test -race -p 1 ./internal/core -run 'TestWorkflowSessionDisconnectedWithinWindowHoldsUntilExpired|TestWorkflowRunCancelDisconnectedWithinWindowWaitsForEndConfirmation' -count=1 -timeout 60mgo test -p 1 ./internal/core/... -count=1 -timeout 60mgo test -p 1 ./integration -run 'Release|Delivery|GiveUp|NodeUnknown|Unreachable' -count=1 -timeout 60m本机缺少
git http-backend,导致TestGitServerAuthorizesDeliveryPushFromCloudState失败,与本次修改无关。Refs ora-space/circus#397
Refs ora-space/circus#391
🤖 Generated with Claude Code