Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/agent-runs/startup-settlement.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,14 @@ From there, two things can happen:
into the **residual ledger** `agent_run_residuals`. Cloud **never fabricates** a Node result, receipt or
Revision.

When the end-confirmation window expires without a real terminal result, the same residual row
records `stop_evidence.stop_unconfirmed`: kind, source `cloud`, observation time and original execution
identity. Repeated scans neither duplicate nor renew it. It changes neither `reason`, `responsibility`
nor resource protection, and grants no termination, deletion or clearance authority. Node exposes its
own stop-confirmation bound and original Scope/Run timestamps through the existing execution query;
both sides correspond to the same execution/Workspace. Cloud's 2/6/5 windows retain their original
anchors and durations.

## The residual ledger

`agent_run_residuals` records a run that has ended as business but whose external execution and resources
Expand Down
2 changes: 2 additions & 0 deletions docs/agent-runs/startup-settlement.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@
- **结束确认窗口也到期,结果仍然未知**:在同一个事务里把运行写为 `failed`、`done`,Thread 为 `ended`,交付状态为 `failed`,并写入一行
**残留账本** `agent_run_residuals`。Cloud **不伪造** Node 结果、收据或 Revision。

结束确认窗口耗尽且原执行仍无真实终态时,同一残留行的 `stop_evidence.stop_unconfirmed` 记录命名证据:类型、来源 `cloud`、观察时间和原执行身份。重复扫描不追加副本、不续期;`reason`、`responsibility` 和资源保护不因这条证据改变。它不表示 Node 已停止,不授权强停、数据删除或清账。Node 自己的停止确认上限和原 Scope/Run 时间证据通过原执行查询可见,两侧按同一执行 / Workspace 归属对应,Cloud 的 2/6/5 窗口保持原起点和时长。

## 残留账本

`agent_run_residuals` 记录一个业务上已经结束、但外部执行和资源仍可能存在的运行:
Expand Down
29 changes: 29 additions & 0 deletions integration/agent_restore_stop_migration_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
package integration

import "testing"

// A named observation is additive evidence; old responsibility and business facts are immutable.
func TestMigration0059RestoreStopEvidenceFreshUpgradeAndRepeat(t *testing.T) {
fresh, _ := testSchema(t, "test_stop58f_")
migratedTwice(t, newStoreOnSchema(t, fresh))
if columnNullable(t, fresh, "agent_run_residuals", "stop_evidence") {
t.Fatal("stop evidence must be an object, including when empty")
}
pool, _ := testSchema(t, "test_stop58u_")
applyMigrationsUpTo(t, pool, migrationsBefore(t, "0059_agent_restore_stop_evidence.sql"))
seed := seedOwnershipChain(t, pool)
_, err := pool.Exec(`INSERT INTO agent_run_residuals(run_id,tenant_id,project_id,workspace_id,runtime_generation,responsibility,reason)
VALUES($1,$2,$3,$4,0,'{"original":"retained"}','agent_start_failed')`, seed.runID, seed.tenantID, seed.projectID, seed.runWorkspaceID)
must(t, err)
var before, after, evidence string
must(t, pool.QueryRow("SELECT row_to_json(r)::text FROM agent_run_residuals r WHERE run_id=$1", seed.runID).Scan(&before))
migratedTwice(t, newStoreOnSchema(t, pool))
must(t, pool.QueryRow("SELECT (to_jsonb(r)-'stop_evidence')::text,stop_evidence::text FROM agent_run_residuals r WHERE run_id=$1", seed.runID).Scan(&after, &evidence))
var equal bool
must(t, pool.QueryRow("SELECT $1::jsonb=$2::jsonb", before, after).Scan(&equal))
if !equal || evidence != "{}" {
t.Fatalf("upgrade rewrote responsibility or invented stop evidence: %s", evidence)
}
_, err = pool.Exec("UPDATE agent_run_residuals SET stop_evidence='[]'::jsonb WHERE run_id=$1", seed.runID)
wantPGError(t, err, "23514")
}
3 changes: 2 additions & 1 deletion integration/agent_run_controlled_reclaim_migration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -137,10 +137,11 @@ func TestControlledResidualMigrationUpgradePreserves0057HistoryAndDoesNotInventD
uuid.NewString(), d8.tenantID, d8.userID, d8.projectID, d8.runWorkspaceID)
must(t, err)
const snapshot = `SELECT jsonb_build_object(
'residuals',(SELECT jsonb_agg(to_jsonb(r) ORDER BY run_id) FROM agent_run_residuals r),
'residuals',(SELECT jsonb_agg(to_jsonb(r)-'stop_evidence' ORDER BY run_id) FROM agent_run_residuals r),
'runs',(SELECT jsonb_agg(to_jsonb(r) ORDER BY id) FROM issue_runs r),
'workspaces',(SELECT jsonb_agg(to_jsonb(w) ORDER BY id) FROM workspaces w),
'operations',(SELECT jsonb_agg(to_jsonb(o) ORDER BY id) FROM operations o))::text`
// Migrate applies every later migration too; stop_evidence is 0059's additive column, not a rewrite of history.
var before, after string
must(t, pool.QueryRow(snapshot).Scan(&before))
store := newStoreOnSchema(t, pool)
Expand Down
3 changes: 2 additions & 1 deletion integration/agent_run_residual_reclaim_migration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,14 @@ func seedResidualMigrationRow(t *testing.T, pool *sql.DB, seed skeletonSeed, rea
}

// Compare the complete old ledger and business rows, rather than only the newly added defaults.
// Columns added by later additive migrations are excluded, as in the other upgrade-path tests.
func residualMigrationHistory(t *testing.T, pool *sql.DB) string {
t.Helper()
var snapshot string
must(t, pool.QueryRow(`SELECT jsonb_build_object(
'residuals',(SELECT jsonb_agg(to_jsonb(r)-ARRAY['reclaim_state','disposition','reclaim_id','reclaim_mode',
'reclaim_operation_id','reclaim_authorization','reclaim_targets','reclaim_evidence','reclaim_version',
'reclaim_checked_at','reclaim_conflicted_at'] ORDER BY run_id) FROM agent_run_residuals r),
'reclaim_checked_at','reclaim_conflicted_at','stop_evidence'] ORDER BY run_id) FROM agent_run_residuals r),
'runs',(SELECT jsonb_agg(to_jsonb(r) ORDER BY id) FROM issue_runs r),
'workspaces',(SELECT jsonb_agg(to_jsonb(w) ORDER BY id) FROM workspaces w),
'operations',(SELECT jsonb_agg(to_jsonb(o) ORDER BY id) FROM operations o))::text`).Scan(&snapshot))
Expand Down
52 changes: 52 additions & 0 deletions internal/core/agent_run_restore_stop_db_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
package core

import (
"encoding/json"
"testing"
)

// Repeated reconciliation must not renew the observation or change the original responsibility.
func TestRestoreStopUnknownAddsOneResidualObservationWithoutReclaimAuthority(t *testing.T) {
store := commandStore(t)
scene := seedTakeoverScene(t, store)
requestStartupEnd(t, store, scene.dispatchableScene)
expireConfirmation(t, store, scene.run)
startupPass(t, store)
var before, after, responsibility, reason string
if err := store.Pool.QueryRow(`SELECT stop_evidence::text,responsibility::text,reason
FROM agent_run_residuals WHERE run_id=$1`, scene.run).Scan(&before, &responsibility, &reason); err != nil {
t.Fatal(err)
}
var evidence map[string]struct {
Kind string `json:"kind"`
Source string `json:"source"`
ExecutionID string `json:"executionId"`
ObservedAt string `json:"observedAt"`
}
if err := json.Unmarshal([]byte(before), &evidence); err != nil {
t.Fatal(err)
}
fact, ok := evidence["stop_unconfirmed"]
if !ok || len(evidence) != 1 || fact.Kind != "stop_unconfirmed" || fact.Source != "cloud" || fact.ExecutionID != scene.execution || fact.ObservedAt == "" || reason != "agent_start_failed" {
t.Fatalf("invalid named stop evidence: %s / %s", before, reason)
}
startupPass(t, store)
startupPass(t, store)
var currentResponsibility, currentReason string
if err := store.Pool.QueryRow(`SELECT stop_evidence::text,responsibility::text,reason
FROM agent_run_residuals WHERE run_id=$1`, scene.run).Scan(&after, &currentResponsibility, &currentReason); err != nil {
t.Fatal(err)
}
if before != after || responsibility != currentResponsibility || reason != currentReason {
t.Fatal("stop evidence or original responsibility changed across reconciliation")
}
var pending, protected, retained bool
if err := store.Pool.QueryRow(`SELECT e.result IS NULL,NOT w.admission_open,r.cleared_at IS NULL
FROM agent_run_residuals r JOIN node_executions e ON e.execution_id=$2
JOIN workspaces w ON w.id=r.workspace_id WHERE r.run_id=$1`, scene.run, scene.execution).Scan(&pending, &protected, &retained); err != nil || !pending || !protected || !retained {
t.Fatalf("observation fabricated stop or lost residual protection: %v %v %v %v", pending, protected, retained, err)
}
if countDeleteOperations(t, store, scene.run) != 0 {
t.Fatal("unconfirmed stop authorized deletion")
}
}
6 changes: 6 additions & 0 deletions internal/core/agent_run_startup_budget.go
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,12 @@ func (s *Store) abandonExpiredAgentStartup(t *transaction, runID string) bool {
}
t.exec(`INSERT INTO agent_run_residuals(run_id,tenant_id,project_id,workspace_id,runtime_generation,responsibility,reason)
VALUES($1,$2,$3,$4,$5,$6,'agent_start_failed') ON CONFLICT(run_id) DO NOTHING`, runID, run.S("tenantId"), w.S("projectId"), wid, w.N("runtimeGeneration"), jsonText(refs))
// The expired business window says nothing about physical stop. Retain one named observation
// on this same residual, addressed by the original execution, without rewriting responsibility.
t.exec(`UPDATE agent_run_residuals SET stop_evidence=stop_evidence||jsonb_build_object(
'stop_unconfirmed',jsonb_build_object('kind','stop_unconfirmed','source','cloud',
'observedAt',clock_timestamp(),'executionId',$2::text))
WHERE run_id=$1 AND NOT stop_evidence ? 'stop_unconfirmed'`, runID, e.S("executionId"))
t.exec(`UPDATE workspaces SET admission_open=false,admission_epoch=admission_epoch+1,version=version+1 WHERE id=$1`, wid)
t.exec(`UPDATE runtime_controls SET state='reconciling',binding_confirmed=false,input_closed=false,
version=version+1,updated_at=clock_timestamp() WHERE workspace_id=$1`, wid)
Expand Down
5 changes: 5 additions & 0 deletions internal/core/migrations/0059_agent_restore_stop_evidence.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
-- Named stop evidence is neither a new residual state nor permission to reclaim resources.
-- Keep the original reason and immutable responsibility independent of this observation.
ALTER TABLE agent_run_residuals
ADD COLUMN stop_evidence jsonb NOT NULL DEFAULT '{}'::jsonb
CHECK (jsonb_typeof(stop_evidence) = 'object');
4 changes: 3 additions & 1 deletion internal/core/migrations/README.en.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ SQL is embedded in the binary through `embed.FS` in `core/store.go`. `Store.Migr

## File index

There are currently 60 SQL files.
There are currently 61 SQL files.

| SQL file | Schema change |
|---|---|
Expand Down Expand Up @@ -70,6 +70,7 @@ There are currently 60 SQL files.
| [0056_workflow_invocation_reference.sql](0056_workflow_invocation_reference.sql) | Invocation reference from workflow IssueRuns to WorkflowRuns |
| [0057_agent_run_residual_reclamation.sql](0057_agent_run_residual_reclamation.sql) | Residual reclamation responsibility, appended evidence and disposition |
| [0058_agent_run_controlled_reclamation.sql](0058_agent_run_controlled_reclamation.sql) | Deployment-controlled reclamation intents, dependent data deletion and per-target termination |
| [0059_agent_restore_stop_evidence.sql](0059_agent_restore_stop_evidence.sql) | Named unconfirmed-stop evidence without changing residual responsibility or reclaim authority |

0050 copies no personal connection or key and backfills no historical run. Clearing configuration retains its revision; run options are stored separately from shared run input.

Expand All @@ -91,6 +92,7 @@ There are currently 60 SQL files.
- 0056 first settles every unlinked non-terminal historical workflow IssueRun (including running and deferred) as failed with `workflow_execution_unavailable_before_invocation_linkage`, preserving its input and outcome without inventing an execution. It then adds the `(workflow_invocation_ref, tenant_id)` → `workflow_runs(id, tenant_id)` composite foreign key (targeting the `workflow_run_tenant_identity` unique constraint 0044 already added, without duplicating it), a unique reference, and a CHECK that non-terminal workflow rows carry a reference while other executors carry none, all unconditionally validated. Historical terminal workflow rows may keep a NULL reference. Fresh, seeded upgrade, repeated application and constraint validation coverage is in `integration/workflow_invocation_migration_test.go`.
- 0057 only adds residual-ledger columns: defaults initialize existing D4/D5 rows as `reconciling`, without backfilling targets or evidence, rewriting old `reason`, `responsibility` or business settlement, or backfilling D8. Legacy rows with non-null `cleared_at` cause an explicit migration error and rollback pending compatibility handling (A2). Reentrant DDL preserves progressed rows. The schema accepts `released`, `terminated_data_retained` and `externally_confirmed`; this step's automatic path implements only `released`, leaving the other dispositions to later controlled entry points. The authorization transaction checks the operation's tenant and Workspace relationships; a UUID foreign key alone does not establish resource scope. See [agent_run_residual_reclaim_migration_test.go](../../../integration/agent_run_residual_reclaim_migration_test.go) for fresh, upgrade, DDL replay and constraint coverage.
- 0058 preserves 0057 history, responsibility, evidence, disposition and first `cleared_at`, and never backfills D8. Only an explicit root intent owns protection and evidence for ledgerless D8. Root termination and dependent data deletion each have three states; each resource has one root, at most one deletion intent and one fixed deletion operation. Command keys are unique throughout a tenant. Successful targets require both boolean confirmations, so a missing JSON field cannot pass through SQL NULL. Fresh, 0057-upgrade and constraint coverage is in [agent_run_controlled_reclaim_migration_test.go](../../../integration/agent_run_controlled_reclaim_migration_test.go).
- 0059 only adds `stop_evidence`, initially empty for existing rows; no historical stop observation is invented. Covered by `TestMigration0059RestoreStopEvidenceFreshUpgradeAndRepeat`.
- New migrations must verify fresh databases, upgrades from the preceding schema, repeated application and key constraints. See [migration_upgrade_path_test.go](../../../integration/migration_upgrade_path_test.go) and domain upgrade tests in integration.

See [Persistence](../../../docs/core/persistence.en.md) for ownership/transactions, and [Workflows](../../../docs/workflows/workflows.en.md) / [External Git line](../../../docs/repositories/external-git.en.md) for runtime semantics.
4 changes: 3 additions & 1 deletion internal/core/migrations/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ SQL 通过 `core/store.go` 的 `embed.FS` 编入二进制。`Store.Migrate` 在

## 文件索引

当前共 60 个 SQL 文件。
当前共 61 个 SQL 文件。

| SQL 文件 | schema 变化 |
|---|---|
Expand Down Expand Up @@ -70,6 +70,7 @@ SQL 通过 `core/store.go` 的 `embed.FS` 编入二进制。`Store.Migrate` 在
| [0056_workflow_invocation_reference.sql](0056_workflow_invocation_reference.sql) | workflow IssueRun 到 WorkflowRun 的调用引用 |
| [0057_agent_run_residual_reclamation.sql](0057_agent_run_residual_reclamation.sql) | 残留回收责任、追加证据和处置状态 |
| [0058_agent_run_controlled_reclamation.sql](0058_agent_run_controlled_reclamation.sql) | 部署受控回收意图、从属数据删除与逐目标终止 |
| [0059_agent_restore_stop_evidence.sql](0059_agent_restore_stop_evidence.sql) | 停止未确认的命名证据,不改变残留责任和回收授权 |

0050 不复制个人连接或 Key,不回填历史运行;清除配置保留 revision,运行选项独立于共享运行输入保存。

Expand All @@ -91,6 +92,7 @@ SQL 通过 `core/store.go` 的 `embed.FS` 编入二进制。`Store.Migrate` 在
- 0056 先把没有引用的历史非终态 workflow IssueRun(含 running、deferred)结算为 failed(`workflow_execution_unavailable_before_invocation_linkage`),保留其输入和结果、不虚构执行;再加 `(workflow_invocation_ref, tenant_id)` → `workflow_runs(id, tenant_id)` 复合外键(引用 0044 已建的 `workflow_run_tenant_identity` 唯一约束,不重复建)、唯一引用,以及「非终态 workflow 行必须有引用、其他执行器不得有引用」的 CHECK,无条件 VALIDATE。历史终态 workflow 行可保留 NULL 引用。空库、带历史数据升级、重复应用与约束验证状态见 `integration/workflow_invocation_migration_test.go`。
- 0057 只给残留账本增列:存量 D4/D5 行通过默认值初始化为 `reconciling`,不回填目标或证据,不改旧 `reason`、`responsibility` 或业务结算,也不回填 D8。历史行若已有非空 `cleared_at`,迁移明确报错并回滚,等待显式兼容处理(A2)。DDL 可重入,不会重置已推进条目。数据库接受 `released`、`terminated_data_retained`、`externally_confirmed` 三种处置;本步自动路径只实现 `released`,其余处置留给后续受控入口。operation 的租户和 Workspace 关联由授权事务比对,UUID 外键本身不证明资源范围。空库、升级、DDL 重放和约束覆盖见 [agent_run_residual_reclaim_migration_test.go](../../../integration/agent_run_residual_reclaim_migration_test.go)。
- 0058 不改 0057 的历史列、责任、证据、处置或首次 `cleared_at`,也不回填 D8;无账本的 D8 只由显式根意图承载保护与证据。根终止与从属删卷各自三态;每资源一个根、最多一个从属删除意图及固定删除 operation,命令键在 tenant 范围统一唯一。目标的成功约束要求两个布尔确认均为 true,缺字段不能以 SQL NULL 通过。新库、0057 升级与约束覆盖见 [agent_run_controlled_reclaim_migration_test.go](../../../integration/agent_run_controlled_reclaim_migration_test.go)。
- 0059 只增 `stop_evidence`,存量为空对象,不追认历史未知停止;覆盖见 `TestMigration0059RestoreStopEvidenceFreshUpgradeAndRepeat`。
- 新增迁移须验证空库、前一 schema 升级、重复应用与关键约束。覆盖见 [migration_upgrade_path_test.go](../../../integration/migration_upgrade_path_test.go) 和 integration 的领域升级测试。

数据归属与事务规则见 [持久化](../../../docs/core/persistence.md),运行语义见 [工作流](../../../docs/workflows/workflows.md) 和 [外部 Git 线](../../../docs/repositories/external-git.md)。
Loading