Skip to content

feat(agent-org): add irreversible archive and guarded team delete - #888

Draft
ShiboSheng wants to merge 7 commits into
codex/issue-760-pr5-pause-resumefrom
codex/issue-761-archive-team-delete
Draft

feat(agent-org): add irreversible archive and guarded team delete#888
ShiboSheng wants to merge 7 commits into
codex/issue-760-pr5-pause-resumefrom
codex/issue-761-archive-team-delete

Conversation

@ShiboSheng

@ShiboSheng ShiboSheng commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Problem

Fixes #761.

PR5 provides durable Pause/Resume and exact ownership for background shell commands and subagents, but a Team still needs a separate irreversible close boundary. Without PR6, Archived has no durable receipt, late Provider/Task/Inbox writes can cross the boundary, and generic Session deletion cannot safely dismantle a Team hierarchy.

After PR5 was updated, integration testing exposed an additional lifecycle gap in PR6: Archive waited for runtime release and memory work, but not for the real execution owner of a background job. A shell could be marked Killed while its process tree or replay pipeline was still draining, and a subagent could be marked Killed while its Tokio task was still running. Archive could therefore publish a quiesced receipt too early, and Team Delete did not have exact Session-scoped evidence or cleanup for retained job results and tombstones.

A separate final-reply regression was also found in the PR6 stack: after a Member Turn completed its Task, assistant persistence reused execution admission and rejected the exact completing Turn because a terminal Task is no longer runnable. The Task completed durably while the final assistant message could be lost.

This Draft is stacked directly on the latest PR5 branch codex/issue-760-pr5-pause-resume at ecaa1c721cbb0f4ec60c0136b85c53531421f1ec. PR6 HEAD is 4781b832935e95620f5310aef9a38b5568819d35; the comparison contains seven PR6 commits and 87 files with 7,121 additions and 692 deletions.

Solution

Add an irreversible Archive protocol and an Archived-only Team Delete protocol, then integrate PR5's real background-execution finality into both boundaries:

  • Commit the Archived fence, generation bump, timestamp, unique request/receipt, and cancellation of open Task, Turn, Inbox, Plan Approval, intervention, and Pause-continuation work in one immediate SQLite transaction.
  • Capture canonical Root/Member ownership in per-Session teardown receipts. Release every captured runtime with exact lease/generation checks, at most three attempts and at most 60 seconds total. A failed teardown keeps the Team Archived and records bounded diagnostic evidence.
  • Add exact Session indexes to the shared background-job registry. Registration, removal, replacement, expiry, Archive, and Delete maintain the same indexes and lock order; no registry lock is held across an async wait and no process-wide scan is added.
  • Treat visible job status and execution finality as different facts. Shell finality requires the process tree and replay pipeline to end; subagent finality requires a terminal status and a finished JoinHandle. Archive waits for runtime release, memory jobs, and background execution, then repeats the Session barrier after runtime release to close late-registration races.
  • Recheck Archived/generation at Provider install, Wake, Task, Inbox, Turn, finalizer, recovery, and message-materialization commit boundaries. Late work returns team_archived or becomes a stale no-op.
  • Separate execution admission from final-assistant persistence. Only the exact still-running owner Turn with matching Task event, generation, status, and materialization provenance may persist its terminal reply.
  • Make Team Delete fail closed when any Root/Member Session still owns real execution. Database failure leaves registry evidence intact; database success is followed by exact cleanup of live jobs, results, owner/Session provenance, and tombstones. Other Teams and ordinary SDE Sessions are not scanned or cleared.
  • Expose Archive and Team Delete through the existing Tauri/UI flow. Archived Root, Member, and Group views are read-only; physical Delete requires the danger-zone confirmation and acknowledgement checkbox.
  • Add a debug/WebDriver-only, read-only runtime-evidence endpoint for deterministic observation. Product Archive/Delete actions still go through their real UI buttons and Tauri commands.

The final integration commit is 4781b8329 fix(agent-core): drain archived background jobs before delete. It adds no new schema, dependency, public product wire, product UI state, polling loop, Archived watchdog, or migration beyond the already-reviewed PR6 contracts.

Potential risks

  • Persistence compatibility: the full PR6 stack expands the strict Agent Org runtime manifest and Archive columns. There is intentionally no in-place migration. PR5 and PR6 binaries must use a fresh/reset or commit-matched Agent Org database. Rollback requires reverting the seven PR6 commits together and using the database paired with the target commit.
  • Irreversibility: Archive cannot be undone. A stop failure leaves retained_runtime evidence and blocks Delete; it never reopens the Team. Physical Delete is permanent and remains separately confirmed.
  • Concurrency: safety depends on the durable fence, activation/Turn generations, runtime leases, exact Session indexes, fixed lock ordering, and transaction-time ownership rechecks. The included race tests cover late registration, late JoinHandle attachment, stale callbacks, repeated Archive, transaction failure, and scoped Delete cleanup. Future writers must reuse these gates.
  • Registry cleanup after commit: Delete purges in-memory job state only after the database transaction commits. A theoretically unexpected post-commit purge failure is reported as team_deleted_but_background_job_purge_failed; the database deletion cannot be rolled back at that point. Runtime fencing and preflight tests are intended to make this state unreachable.
  • Provider coverage: three repeated Archive scenarios plus Pause/Resume and Task smoke used one configured non-mock Provider. Other Provider adapters rely on the shared runtime contract and deterministic tests; they were not each exercised with live credentials.
  • Generic derived caches: the approved integration scope clears Agent Org-owned rows, canonical Session data, filesystem history, shell replay data, and the background-job registry. Existing rebuildable cross-feature Session/turn projections continue to follow the repository's generic cache lifecycle; byte-for-byte erasure of every derived projection would be a separate delete-completeness change.
  • Stack dependency: this PR targets PR5 and cannot merge independently of its runtime ownership and Pause/Resume contracts. Git history proves PR6 is based directly on current PR5; GitHub may temporarily show an unknown merge state while recalculating after the force-with-lease update.
  • Repository gate: pnpm run check:circular exits before graph analysis because Madge cannot resolve the pre-existing React artifact ?raw imports. The failing module is outside this diff; commit hooks report zero circular dependencies in staged TypeScript files.
  • UI evidence: rendered E2E and packaged-app Computer Use cover the real lifecycle and confirmation flow. A static screenshot is not attached because it cannot prove process-tree termination, durable receipt ordering, restart behavior, or database cleanup.

Verification

Automated checks on the rebased implementation:

  • cargo check -p agent_core — passed.
  • cargo check -p org2 — passed.
  • Focused background-job registry suite — 23 passed, including Session isolation, Killed-but-running subagent blocking, late JoinHandle attachment, shell finality, tombstone expiry, and scoped purge.
  • Focused Archive suite — 8 passed.
  • Focused Team Delete persistence suite — 12 passed.
  • cargo test -p agent_core — 3,230 passed, 0 failed, 2 ignored; doc tests completed with 11 ignored.
  • cargo clippy -p agent_core --all-targets -- -D warnings — passed.
  • Commit hook Clippy for both agent_core and desktop org2 — passed; no hook was bypassed.
  • cargo fmt --all -- --check — passed.
  • pnpm test — 1,116 files and 8,777 tests passed.
  • pnpm run lint — passed with 0 errors and 5 pre-existing warnings outside this diff.
  • pnpm run check:circular — blocked before graph analysis by the existing Madge raw-import resolver failure described above.
  • Node syntax and Prettier checks for both changed E2E specs — passed.
  • git diff --check and staged secret/personal-path/build-artifact scan — passed.

Rendered, packaged-app, and real-machine verification:

  • The real Archive → durable quiesced → Overview refresh → danger-zone Delete rendered spec passed for idle/completed, working, and paused Teams in one packaged-app run.
  • WEBDRIVER=1 pnpm run tauri:build:fast passed. The packaged binary SHA-256 is 706a3d30459a38801ccef113acd05a39acc005b63509d0f108e15070d7cec74e. Product source was frozen before this build; the only later adjustment was E2E test code, not application code.
  • Three consecutive non-mock Provider Archive rounds passed. Read-only UI appeared in 85–88 ms; full quiescence completed in 2,296–2,299 ms on attempt 1. Each TERM-resistant parent/child process group disappeared, active runtime/Turn/job/blocker counts reached zero, exactly one replay remained, and a 5-second quiet window showed no late message, Task, Inbox, replay, or process write.
  • PR5 regression passed in the same packaged app and isolated home: Pause drained the parent/child process group and runtime; restart + Resume produced exactly one continuation and one final replay; Task lifecycle smoke completed one Task.
  • Computer Use operated the same packaged app through Archive, quit, relaunch, persisted read-only confirmation, and permanent danger-zone Delete. After the confirmed isolated test deletion, the target Agent Org runtime rows, Archive receipt, canonical Sessions, messages, events, active Turn intents, file-history directories, shell replay owners, and OS processes were absent. Three unrelated archived Teams and the PR5 smoke Team remained.
  • Performance guard passed for this change. Archive work is on-demand and bounded; idle, hidden, restart, and repeated-open states own no new timer, worker, poll, or full-registry scan. In a clean isolated home, four settled samples measured 0.0%, 0.6%, 0.7%, and 1.4% CPU with approximately 58 MB resident memory.

Architecture and rollback

The architecture audit covered all ten relevant layers: schema, authoritative ownership, domain semantics, control flow, concurrency/lock ordering, persistence transaction boundaries, wire/test surface, initialization/restart, compatibility/isolation, and naming/dead-code/test coverage. The performance lifecycle matrix covered active Archive, idle/hidden, restart, repeated open/close, multi-Team isolation, and ordinary SDE isolation. Verdict: pass for the PR6 background-job integration; no long-lived lifecycle owner was added.

To roll back, revert the seven PR6 commits together and use a fresh/reset Agent Org database or the database paired with PR5. If Archive retains execution, keep the Team Archived, inspect the receipt evidence, and do not run Team Delete until the execution owner is proven absent. There is no migration to reverse for the final background-job integration commit.

@ShiboSheng
ShiboSheng requested a review from Neonforge98 August 23, 2026 07:53
@Harry19081 Harry19081 added enhancement New feature or request agent Agent runtime, behavior, memory, providers, or orchestration cloud-collaboration Cloud sync, organizations, channels, or collaboration labels Aug 23, 2026
@ShiboSheng
ShiboSheng force-pushed the codex/issue-761-archive-team-delete branch from e43c5c7 to 8ac2812 Compare August 24, 2026 10:39
Commit the irreversible Archived state, generation bump, cancellation facts, and bounded teardown receipts atomically. Capture canonical Root and Member ownership so retries are idempotent and later deletion can prove the same Team boundary.\n\nVerification:\n- cargo test -q -p agent_core -- --test-threads=1 — 3,193 passed on the complete PR6 working tree\n- cargo clippy -p agent_core --all-targets -- -D warnings — passed on the complete PR6 working tree\n- cargo fmt --all -- --check — passed

Pre-commit hook ran. Total eslint: 5, total circular: 0
Reject post-Archive writes at Task, Inbox, Turn, Provider, Wake, recovery, and Resume boundaries. Release captured Root and Member runtimes with exact-lease checks, three bounded rounds, and one-shot startup reconciliation without a persistent watchdog.

Verification:

- cargo test -q -p agent_core -- --test-threads=1 — 3,193 passed on the complete PR6 working tree

- cargo clippy -p agent_core --all-targets -- -D warnings — passed on the complete PR6 working tree

- packaged BuildFast Archive races, retained-runtime deadline, restart, and ten-runtime parallel teardown — passed

Pre-commit hook ran. Total eslint: 5, total circular: 0
Route every Agent Org Root or Member away from generic Session Delete. Allow the dedicated Team Delete only after Archived persistence, a quiesced Archive receipt, unchanged ownership, and zero current runtime leases are all proven.

Verification:

- cargo test -q -p agent_core -- --test-threads=1 — 3,193 passed on the complete PR6 working tree

- generic Root and Member Delete fail-closed tests — passed

- isolated packaged-app Team Delete preserved unrelated Team data — passed in four repeated rendered runs

Pre-commit hook ran. Total eslint: 5, total circular: 0
Wire Archive and guarded Team Delete through Tauri, make every Archived composer read-only, keep history visible, surface bounded teardown diagnostics, and require a second danger-zone confirmation before physical deletion.

Verification:

- pnpm test — 1,116 files and 8,771 tests passed on the frozen PR6 tree

- pnpm run lint — passed with five pre-existing ProjectManager warnings and zero errors

- rendered Archive/Delete E2E — passed three consecutive full runs plus one final smoke

- BuildFast packaged-app Computer Use and SQLite evidence — passed for PR5 baseline, fake Provider, and configured real Provider

Pre-commit hook ran. Total eslint: 5, total circular: 0
Packaged-app testing exposed two terminal-boundary gaps: late session callbacks could still reach persistence after Archive, and Team Delete removed sidebar ownership without closing cached Team tabs or unregistering session marker files.

Route late callbacks through the Archived fence, keep the destructive confirmation modal stable, close every deleted Root/Member tab atomically, select one safe fallback view, and unregister each committed deletion from the file registry.

Verification:
- pnpm test — 1,116 files and 8,776 tests passed
- RUST_TEST_THREADS=1 pnpm run cargo:test — 1,113 passed and 1 ignored
- cargo test -p agent_core — 3,197 passed and 2 ignored
- cargo clippy -p agent_core --all-targets -- -D warnings — passed
- rendered Archive/Delete E2E — 3 frozen-code runs and 9 UI paths passed
- BuildFast packaged-app Computer Use — real Provider Archive/Delete and restart passed

Pre-commit hook ran. Total eslint: 5, total circular: 0
TaskExecution assistant persistence reused the execution-admission gate, so a Turn that completed its own Task lost its final reply because terminal Tasks are not runnable.

Split the phase-specific validation while preserving the shared live Turn authority and Archive transaction fence. Allow completed or failed Tasks only when the latest terminal event belongs to the exact owner and Turn; keep cancellation, reassignment, stale generations, and new execution admission closed.

Verification:
- cargo test -p agent_core — 3,215 passed, 2 ignored
- cargo clippy -p agent_core --all-targets -- -D warnings — passed
- pnpm test — 1,116 files and 8,777 tests passed

Pre-commit hook ran. Total eslint: 5, total circular: 0
Archive receipts could become quiesced after a job was marked killed even though its process tree, replay pipeline, or subagent task was still executing. Team Delete also left retained background-job registry state after the durable hierarchy was removed.

Add exact Session indexes and execution-finality barriers, make Archive wait for runtime, memory, and background owners, and make Delete fail closed before commit and purge only after a successful transaction. Add deterministic race coverage, rendered lifecycle coverage, and read-only WebDriver evidence.

Verification:
- cargo test -p agent_core: 3230 passed, 2 ignored
- cargo clippy -p agent_core --all-targets -- -D warnings: passed
- cargo fmt --all -- --check: passed
- pnpm test: 8777 passed
- pnpm run lint: passed with 0 errors and 5 pre-existing warnings
- pnpm run check:circular: blocked by the existing Madge raw-import resolver failure
- WEBDRIVER=1 pnpm run tauri:build:fast: passed
- BuildFast real-provider Archive x3, Pause/Resume, Task smoke, restart, and UI Delete: passed

Pre-commit hook ran. Total eslint: 5, total circular: 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Agent runtime, behavior, memory, providers, or orchestration cloud-collaboration Cloud sync, organizations, channels, or collaboration enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(agent-org): [6/10] separate irreversible Archive from physical Delete

2 participants