Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/electron-windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@ jobs:

- uses: actions/setup-dotnet@v5
with:
dotnet-version: '8.0.x'
# Pin the same SDK resolved by 8.0.x; avoid its broken aka.ms blob redirect.
dotnet-version: '8.0.427'

- name: Install locked dependencies
run: npm ci
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/python-windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,8 @@ jobs:

- uses: actions/setup-dotnet@v5
with:
dotnet-version: '8.0.x'
# Pin the same SDK resolved by 8.0.x; avoid its broken aka.ms blob redirect.
dotnet-version: '8.0.427'

- name: Build actual Windows executable
working-directory: electron
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/source-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,8 @@ jobs:

- uses: actions/setup-dotnet@v5
with:
dotnet-version: "8.0.x"
# Pin the same SDK resolved by 8.0.x; avoid its broken aka.ms blob redirect.
dotnet-version: '8.0.427'

- name: Install pinned uv
run: python -m pip install uv==0.12.8
Expand Down
11 changes: 9 additions & 2 deletions agent_host/adapters/pi.py
Original file line number Diff line number Diff line change
Expand Up @@ -477,5 +477,12 @@ async def cancel(self, run_id):

async def close(self):
runs = tuple(self._runs.items())
await asyncio.gather(*(self.cancel(key) for key, _ in runs), return_exceptions=True)
await asyncio.gather(*(run.client.close() for _, run in runs), return_exceptions=True)
cancelled = await asyncio.gather(*(self.cancel(key) for key, _ in runs), return_exceptions=True)
closed = await asyncio.gather(*(run.client.close() for _, run in runs), return_exceptions=True)
errors = []
if any(isinstance(result, BaseException) or not isinstance(result, dict)
or result.get("confirmed") is not True for result in cancelled):
errors.append("pi_cancel_unconfirmed")
if any(isinstance(result, BaseException) for result in closed):
errors.append("pi_close_failed")
return {"settled": not errors, "errors": errors}
19 changes: 16 additions & 3 deletions agent_host/provider_runtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -628,7 +628,7 @@ def project_reconciled_terminal_result(
metadata=_public_provider_metadata(record.metadata),
)

async def close(self) -> None:
async def close(self) -> dict[str, Any]:
"""Release Provider-owned runtime resources during Host shutdown.

Provider execution lifecycles stay behind their adapters. The Host
Expand All @@ -638,11 +638,14 @@ async def close(self) -> None:
deliberately not fabricated here.
"""

errors: list[str] = []
if self._admission_cleanup_tasks:
await asyncio.gather(
outcomes = await asyncio.gather(
*tuple(self._admission_cleanup_tasks),
return_exceptions=True,
)
if any(isinstance(outcome, BaseException) for outcome in outcomes):
errors.append("admission_cleanup_failed")
retained_control_tasks = {
task
for task in (
Expand All @@ -660,16 +663,26 @@ async def close(self) -> None:
await self._wait_for_retained_task(task)
except BaseException:
logger.exception("provider retained control transaction failed during close")
errors.append("retained_control_failed")
for provider_id, adapter in tuple(self._adapters.items()):
close = getattr(adapter, "close", None)
if not callable(close):
continue
try:
outcome = close()
if inspect.isawaitable(outcome):
await outcome
outcome = await outcome
if isinstance(outcome, dict) and outcome.get("settled") is not True:
errors.append(f"adapter_unsettled:{provider_id}")
except Exception:
logger.exception("provider adapter close failed: %s", provider_id)
errors.append(f"adapter_close_failed:{provider_id}")
pending = [record.run_id for record in self._runs.values()
if record.status not in {"done", "error", "cancelled"}
or (record.task_handle is not None and not record.task_handle.done())
or (record.terminal_publication_task is not None
and not record.terminal_publication_task.done())]
return {"settled": not errors and not pending, "errors": errors, "pending_runs": pending}

def add_orphaned_run(self, *, provider: str, run: dict[str, Any]) -> None:
"""Restore one adapter-owned resumable checkpoint without interpreting it."""
Expand Down
1 change: 1 addition & 0 deletions config/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -346,6 +346,7 @@ These are not pending mechanical migrations:
| wallpaper scenario helpers | wallpaper/scenario overrides | Both wallpaper hosts resolve component-local media overrides without importing the heavyweight application settings facade. |
| `vn_player/runtime.py` | `VN_*` values | These describe one VN session, not the application startup snapshot. |
| `server/runtime_status.py` | build/workspace metadata | The launcher supplies diagnostic facts for the current process. |
| `server/backend_cleanup.py` | `AMADEUS_BACKEND_INSTANCE_NONCE` | The desktop supplies this in-memory process identity; cleanup must bind its launch registration to the authenticated desktop before runtime imports. It is not a user setting or persisted credential. |
| `asr/qwen3_asr_sidecar.py` | `QWEN3_ASR_DEVICE`, `QWEN3_ASR_REQUIRE_CUDA` | The isolated interpreter reads the environment supplied by its parent. |
| `tts/gpt_sovits_sidecar.py` | `TTS_REQUIRE_CUDA` | The isolated interpreter enforces the device requirement supplied by its parent. |
| `server/presentation_runtime.py` | `AMADEUS_PRESENTATION_LOCALE`, `AMADEUS_WALLPAPER_CAPTION_MODE` | Presentation resolves its current, component-owned preferences. |
Expand Down
3 changes: 3 additions & 0 deletions config/catalog_legacy.json
Original file line number Diff line number Diff line change
Expand Up @@ -241,6 +241,9 @@
"AMADEUS_VISUAL_PROFILES_PATH",
"AMADEUS_WORK_LEDGER_PATH"
],
"server/backend_cleanup.py": [
"AMADEUS_BACKEND_INSTANCE_NONCE"
],
"server/browser_branch_planner.py": [
"AMADEUS_BROWSER_BRANCH_LLM_ENABLED",
"BROWSER_BRANCH_DOM_CHARS",
Expand Down
126 changes: 126 additions & 0 deletions docs/evidence/pr261-crash-recovery/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# PR #261: recovery after an unclean backend exit

Baseline: `019c79bfdab664cea3a8555a42877dc362ed8abe`.

The external review's composition probe was reproduced against that revision:
an unconfirmed update plus an owner without a graceful-exit receipt blocked
both backend launch and scheduled rollback on every subsequent launch. The
normal-exit control restored the previous version. Direct installer rollback
tests had missed the production update-boundary predicate.

## Repaired contracts

- Recovery holds the desktop's stopped-root/startup mutex, verifies the retained
wheel and native confirmation, and restores the previous package record and
encrypted configuration. It does not require historical child-exit evidence,
apply frontend changes, or collect either installation. The recovery scope
cannot authorize a new installation. A currently running backend still blocks
recovery.
- Installation and deferred file removal continue to require physical process
exit evidence. The outside Windows Job observer now waits for the root and all
descendants even after a crash or unsettled teardown. A durable containment
marker, written only after successful root assignment, prevents an empty but
never-assigned Job from becoming evidence. Missing marker, invalid identities,
unreadable records and live descendants remain blocked.
- Graceful Provider/Work completion is diagnostic information, independent of
local process lifetime. No Work outcome, recovery decision, lease or replay
authority is changed by cleanup-index retirement. Existing `settled` receipts
retain their original validation; new `exited` receipts certify only physical
exit.
- A later Windows boot can retire otherwise unrecoverable process owners and
unknown state. The index stores the OS-maintained BootId counter documented
by [Microsoft](https://learn.microsoft.com/windows-hardware/design/device-experiences/oem-hvci-enablement).
Neither wall-clock estimates nor a closed listener count as a reboot. A
missing/backward boot identity grants no retirement. Evidence files remain
archived. An older index without BootId first acquires the current binding
without clearing owners; if old exit evidence is missing, Windows must be
restarted after that adoption.
- Windows query/session-end events request best-effort cleanup without vetoing
shutdown. A cancelled query does not leave the app in permanent quit intent.
Crash recovery does not rely on these callbacks completing. Linux/macOS keep
the original 2.5-second graceful-stop timeout.

## Regression coverage

The installer composition test executes the actual boundary declarations from
`index.ts`, with real store, journal, catalog and cleanup-index implementations.
It installs an update, registers an unresolved owner, reopens, schedules recovery,
and verifies repeated later launches can use the old version while maintenance
and removal remain blocked. A later injected boot permits deferred removal.
It also checks live-root rejection and retained credentials. A separate theme
test prevents record restoration from incidentally deleting either version.

Native Windows tests exercise root termination without graceful approval,
unsettled Work diagnostics, and a descendant surviving its parent both with and
without graceful approval. The descendant is left running for the negative
check and released normally for the positive check. Boot-transition tests cover
same-boot refusal, unavailable/backward identity, legacy adoption, unknown-state
retirement and fresh owners after retirement.

The shipping-app failure journey includes both normal rollback and a second
pending update whose owned backend is forcibly terminated before rollback.
The test captures the exact ChildProcess created with its isolated backend
nonce; it never searches for or terminates another process by name. The first
qualification attempt exposed a test-instrument error (the main module's
private variable was unavailable to inspector evaluation). The fixture now
captures spawn and refreshes the ESM binding; production exports no test hook.

## Validation and limits

- `cd electron; npm test`: **685 passed, 0 skipped**.
- `cd electron; npm run build`: passed configuration/SDK/Pi/TypeScript/Vite checks;
the existing large-chunk warning remains.
- Focused Python cleanup, lifecycle and smoke-probe suite: **96 passed**.
- Python lint on changed files, JavaScript syntax checks and `git diff --check`:
passed.
- Real offline author journey: **12 checks passed**, including native Provider
regressions and retained/default versus explicit package-data removal.
- Full shipping-app qualification: **14 normal extension checks + 23
failure/recovery checks passed**, including **13 app restarts**. Both the
forced pending-update backend termination and subsequent rollback/old-runtime
startup assertions passed. Final Electron and backend exit was confirmed.

Commands used (generated fixture paths are local to this qualification):

```powershell
cd electron
npm test
npm run build
cd ..
.\.venv\Scripts\python.exe -m pytest tests/test_backend_cleanup.py tests/test_electron_model_less_smoke.py tests/test_chat_ingress_lifecycle.py tests/test_vts_worker_lifecycle.py -q --basetemp=build/pr261-recovery-python-all
.\.venv\Scripts\python.exe -X utf8 tools/probes/extension_author_journey.py
.\.venv\Scripts\python.exe -X utf8 tools/probes/extension_catalog_journey.py --execute --store-ui --prepare-only --failure-fixture --wheels build/extension-author-journey/86129dff75f6/wheels
.\.venv\Scripts\python.exe -X utf8 tools/smoke_electron_model_less.py --report-dir build/pr261-recovery-app-final --extension-fixture build/extension-catalog-journey/220348df14c1 --extension-failures
```

The final application report is under
`build/pr261-recovery-app-final/electron_model_less_20261025T101246Z_99d7e4/`.
Raw logs are not committed because they contain local paths. The full packaged
executable publication journey was not rerun for this patch.

This is Windows qualification. Boot changes are injected into the OS-read
boundary; the real OS counter is read and checked, but no physical reboot or
Windows logoff was performed. Linux/macOS routing remains fixture-tested, not a
new native OS installation qualification. Catalog transport and native dialogs
use the existing isolated smoke fixture; plugin packages, application startup,
backend processes, process ownership and persisted state are real. No shared
dependency environment was synchronized and the worktree's junctions remain
untouched.

## CI follow-up — 2026-10-26

The Electron Windows job at `15bc3995` failed one of 685 tests: the native
BootId observation returned unavailable after approximately the 10-second query
budget, but the test required an integer unconditionally. The log did not expose
whether the registry value was missing or the query timed out. Production
already treats unavailable evidence conservatively.

BootId now uses a direct, bounded `reg.exe` query, avoiding the PowerShell host.
Only one `REG_DWORD` value in the unsigned 32-bit range is accepted and cached.
The reader tests cover missing/query-timeout, empty, wrong-type, overflow and
duplicate evidence, including unchanged unknown ownership and no unavailable
result caching. The native test verifies either stable evidence or continued
ownership fencing. No workflow, timeout increase, skip or cleanup exemption was
added. Full Electron tests: **686 passed, 0 skipped**; build passed. The earlier
Python and real-app results above belong to the recovery implementation, not a
new rerun for this registry-reader follow-up.
Loading
Loading