Skip to content

fix(core): honor node unknown window after disconnect - #118

Merged
EricWvi merged 2 commits into
ora-space:mainfrom
creegon2:fix/397-node-unknown-window
Oct 21, 2026
Merged

EricWvi merged 2 commits into
ora-space:mainfrom
creegon2:fix/397-node-unknown-window

Conversation

@creegon2

@creegon2 creegon2 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

修改

修复 Node 刚被标记为 disconnected 时,D5 立即放弃 Revision 交付、D8 随即放弃删除的行为。根 ADR IssueRun:一次运行一个 disposable isolated Workspace 已为 implemented:D5 要求 Node 未知超过配置窗口才按不可达放弃交付;D8 共用该窗口(默认 30 分钟)。本 PR 修复既有实现,不修改 ADR 状态。

nodeStatus 只在 connected 报告时刷新已有 last_seen_at,disconnected 报告保留原值。共享 runWorkspaceNodeUnknown 比较该时间时不再过滤 connection_state。起点选择最后一次确认 Node 可达的数据库时间,因为断连通知只说明 Cloud 收到了失联消息,不能证明 Node 已重新可达;重复断连不能延长等待,重连则自然重新计时。无需新增迁移。

connected 但心跳已超过窗口时仍沿用原有不可达判断;无存活 sandbox 时仍立即满足 D5 的未知条件;D8 仍保留无 sandbox 的 cleanup 和明确拒绝 quiesce 的行为。workflow 会话收尾与取消路径也共用该 helper,因此 disconnected 但时间仍在窗口内时不再立即满足不可达条件,connected/stale 行为及其他结束确认超时保持不变。其他 9 处生产时间判断均同时要求 connected,不受断连时间保留的影响。

Node 资源(OpenAPI 及 Controller snapshot)的 lastSeenAt 在 Node 断连时现在表示最后一次 connected 报告的时间;仓库内没有业务代码根据这个资源字段作决策。

测试

新增集成测试通过真实 ReportNodeStatus 和 delivery/release 路径执行,数据库时间调整代替 sleep:

  • TestNodeUnknownDisconnectedWithinWindowKeepsDeliveryAndRelease:delivering/releasing 两阶段在窗口内均继续等待,不提前删除或记录失败。
  • TestNodeUnknownDisconnectedPastWindowSettlesDeliveryAndRelease:超过窗口后,D5 放弃交付并声明删除,D8 以 node_unavailable 结束删除并进入 done。
  • TestNodeUnknownReconnectRestartsWindowButRepeatedDisconnectDoesNot:重连刷新起点,首次及重复断连均保留它,重新到期后放弃。
  • TestNodeUnknownNoLiveSandboxReleasesDeliveryButPreservesCleanup:无存活 sandbox 时 D5 立即释放,删除进入 cleanup 后 D8 不放弃。
  • 原有 TestG032_4_RefusedQuiesceNeverTriggersGiveUpEvenPastTheWindow 保持不变并纳入 race 运行;原有 D5/D8 和 workflow connected/stale 测试保持不变。
  • TestWorkflowSessionDisconnectedWithinWindowHoldsUntilExpired:通过 nodeStatus 断连后,workflow 会话在窗口内保持 running,超过不可达窗口后变为 failed。
  • TestWorkflowRunCancelDisconnectedWithinWindowWaitsForEndConfirmation:新近断连后取消运行仍先等待 EndSession 确认,仅在原确认窗口到期后变为 cancelled。

撤回修复后,窗口内的 D5/D8 会提前进入 releasing/done,workflow 会话会提前失败,取消也会跳过 EndSession 确认等待。

检查结果

测试使用 REQUIRE_POSTGRES=1:

命令 结果
task format PASS
go run ./cmd/checkformat PASS
task lint PASS,0 issues
D5/D8 测试及 TestG032_4:go test -race -p 1 ./integration -run 'TestNodeUnknown|TestG032_4_RefusedQuiesceNeverTriggersGiveUpEvenPastTheWindow' -count=1 -timeout 60m PASS 11 / SKIP 0 / FAIL 0
Workflow 新增测试:go test -race -p 1 ./internal/core -run 'TestWorkflowSessionDisconnectedWithinWindowHoldsUntilExpired|TestWorkflowRunCancelDisconnectedWithinWindowWaitsForEndConfirmation' -count=1 -timeout 60m PASS 2 / SKIP 0 / FAIL 0
go test -p 1 ./internal/core/... -count=1 -timeout 60m PASS 665 / SKIP 0 / FAIL 0
go test -p 1 ./integration -run 'Release|Delivery|GiveUp|NodeUnknown|Unreachable' -count=1 -timeout 60m PASS 52 / SKIP 0 / FAIL 1

本机缺少 git http-backend,导致 TestGitServerAuthorizesDeliveryPushFromCloudState 失败,与本次修改无关。

Refs ora-space/circus#397
Refs ora-space/circus#391

🤖 Generated with Claude Code

Preserve the last connected heartbeat when receiving disconnected reports,
and use that instant for the shared node-unknown window without filtering
on connection state. Reconnecting resets the window; repeated disconnect
reports cannot extend it.

Keep stale connected heartbeat handling, no-live-sandbox delivery settlement,
release cleanup, and explicit quiesce refusal behavior unchanged. The same
window also applies to workflow session and cancellation settlement.

Add real ReportNodeStatus and delivery/release regression coverage for both
sides of the window, reconnects, repeated disconnect reports, and cleanup.

Validation: task format, go run ./cmd/checkformat and task lint (0 issues).
Race-selected integration cases: 11 pass, 0 skip, 0 fail. Full internal/core:
663 pass, 0 skip, 0 fail. Release|Delivery|GiveUp|NodeUnknown|Unreachable
integration selection: 52 pass, 0 skip, 1 environment failure in the Git
server test (requires git http-backend on Windows), retained per scope.
Temporarily reverting both production files makes the within-window
delivery and release regressions fail as expected.

Refs ora-space/circus#397
Exercise workflow session settlement and cancellation through nodeStatus
and the real workflow scheduler. Verify that a recent disconnect keeps
the session running until the unreachable window, and that cancellation
waits for the existing end-confirmation window.

Document the workflow cancellation caller of the shared unknown-node
judgement. Production timeout settings remain unchanged.

Validation: format, checkformat, lint, focused race tests, the core suite
and the requested integration selection. The Git server integration
environment still lacks git http-backend on this Windows machine.
Reverting the original fix exposes the premature workflow settlements.

Refs ora-space/circus#397
Refs ora-space/circus#391

@Kyber5323 Kyber5323 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

review:Approve(head a25c0b65f01229854bd10b212019f8cfa2e5f583)

审阅版本:head a25c0b6(base b7e1801),对照当前 upstream/main a9c5f20,mergeable。依据 specs 根 ADR decisions/cloud/issue-run/0-agent-run-in-disposable-isolated-workspace.md(upstream/main 911c976)的 D5/D8。

核对通过:

  • 计时起点改为「最后一次确认可达」:nodeStatus 只在 connected 报告时刷新 last_seen_at,disconnected 保留原值;runWorkspaceNodeUnknown 不再要求当前 connection_state='connected',断连后必须等满窗口。起点已在 PR 描述和运行 issue 中写明;根 ADR 没有更细的规定,这个选择合理,且比「按断连报告起算」略早、最多相差一个心跳间隔。
  • 重复断连不延长窗口、重连重新计时;connected 但心跳过期、无存活 sandbox、明确拒绝 quiesce 三个既有分支行为不变。
  • 其他 9 处生产代码读取 last_seen_at 时都同时要求 connection_state='connected',保留旧值不影响它们;node_idle 仍刷新该时间(这份报告本身就是可达证据)。
  • D5/D8、workflow 会话结算与取消共用同一判断;新增集成用例走真实 ReportNodeStatus 与 delivery/release 路径,workflow 用例覆盖窗口内保持 running 与取消仍等 EndSession 确认。
  • ADR 状态:根 ADR 已是 implemented,描述写明本 PR 只修既有实现、不改状态;无迁移;4 个改动文件均为交付物;两个 commit 按 fix / test 拆分。

非阻塞:与并行中的 cloud#119(workflow 释放放弃)无重复实现——#119 复用同一个 runWorkspaceNodeUnknown,会自动继承本修复;两个 PR 都改 agent_run_release.go 的相邻位置,后合并者按当时 main rebase 即可,无语义冲突。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants