feat: write a release manifest, stop apps removed from the desired set

AGENTS.md

@@ -393,7 +393,7 @@ GitHub Actions workflow (`.github/workflows/release.yml`) manages releases via [
 
 Deployment helpers live in this repository, entirely under `deploy/`, run only on the GitHub Actions runner - the target host never runs any of this:
 
-- `deploy/deploy.py` is the deploy entrypoint. It resolves and downloads every ref in `app_refs` (the app bundles, at least one required — flightdeck's own `apps/` catalog is just another entry, not implicit) and merges them into a release tree locally; for each app in the target's `apps` mapping, downloads its `env_refs` (still encrypted), decrypts them with the target's private SOPS age key, writes the plaintext into that app's `.env` in the release tree, and renders that app's `*.tpl` files in place with the decrypted values (see "Config Templates" above). It then opens an SSH connection per host, pushes the finished release as one tarball (real `.env`, already-rendered config, all versioned together), bootstraps networks/directories idempotently, switches a timestamped release, and runs `docker compose pull && docker compose up -d` per app directly (no wrapper script on the host at all).
+- `deploy/deploy.py` is the deploy entrypoint. It resolves and downloads every ref in `app_refs` (the app bundles, at least one required — flightdeck's own `apps/` catalog is just another entry, not implicit) and merges them into a release tree locally; for each app in the target's `apps` mapping, downloads its `env_refs` (still encrypted), decrypts them with the target's private SOPS age key, writes the plaintext into that app's `.env` in the release tree, and renders that app's `*.tpl` files in place with the decrypted values (see "Config Templates" above). It writes a `manifest.json` into the release tree (`schema_version`, `release` timestamp, resolved `app_refs`/`env_refs` - the actual tag `resolve.py` pulled, never `@latest` - and the desired `apps` list; no secrets, no target identifier, since whoever's reading it is already on that specific host). It then opens an SSH connection per host, pushes the finished release as one tarball (real `.env`, already-rendered config, the manifest, all versioned together), bootstraps networks/directories idempotently, reads the *previous* release's `manifest.json` off `current` and `docker compose down`s any app present there but no longer in the desired set (using `current` before it moves, so that app's last-known compose file/`.env` are still intact - safe even though every release tree already contains every app's compose file regardless of whether the target wants it, since `build_release` copies the whole catalog every time), switches `current` to the new timestamped release, and runs `docker compose pull && docker compose up -d` per desired app directly (no wrapper script on the host at all).
 - `deploy/resolve.py`, `deploy/collisions.py`, `deploy/vault.py`, and `deploy/render.py` hold, respectively, the ref-resolution, ciphertext collision-detection, decryption, and template-rendering logic - each with real `unittest` coverage in `deploy/tests/`.
 - `.github/actions/encrypt-env/` is a local composite action for rendering `vaults/` manifests from GitHub Secrets/Variables, encrypting them for age recipients, and publishing `.sops.env` as a GitHub Release asset — vault manifests hold only env/secrets, not app selection
 - `.github/workflows/deploy-shared.yml` is a reusable workflow consumer repos call to run `deploy/deploy.py` from GitHub Actions over an optional Tailscale connection, without holding any deploy secrets in this repository

README.md

@@ -31,7 +31,8 @@ The deploy is push-based and runs entirely on the GitHub Actions runner:
 3. Check each app's env sources for key collisions from the still-encrypted ciphertext (SOPS's dotenv output only encrypts values, so key names are readable without decryption) — scoped to that app's own sources, not across apps.
 4. Decrypt each app's env with the target's private SOPS age key (a GitHub Secret) and write it straight into that app's `.env` in the release tree.
 5. Render that app's `*.tpl` config files in place, next to its `docker-compose.yml`, using the decrypted values — the same substitution `envsubst` does, run here instead of on the host.
-6. Push the finished release (real `.env`, already-rendered config, one tarball) to each host over SSH, switch the `current` symlink, and run `docker compose pull && docker compose up -d` per app.
+6. Write a `manifest.json` into the release tree — resolved `app_refs`/`env_refs` (the actual tag pulled, not `@latest`) and the desired app set, no secrets.
+7. Push the finished release (real `.env`, already-rendered config, the manifest, one tarball) to each host over SSH. Before switching `current`, compare the new desired app set against the previous release's `manifest.json` and `docker compose down` anything no longer desired, then switch the `current` symlink and run `docker compose pull && docker compose up -d` per app.
 
 What gets deployed — which app bundles, which apps actually run, and which encrypted env sources feed each one — is configured declaratively per target; see "Vaults And Targets" below for the manifest format.
 

deploy/deploy.py

@@ -40,9 +40,11 @@ def build_release(config, work_dir):
     apps_dir = release_dir / "apps"
     apps_dir.mkdir(parents=True)
 
+    resolved_app_refs = []
     for index, ref in enumerate(config["app_refs"], start=1):
         package_dir = pull_dir / f"apps-{index}"
-        bundle = download_ref(ref, package_dir / "pull", default_asset=APPS_BUNDLE_ASSET)
+        bundle, resolved_ref = download_ref(ref, package_dir / "pull", default_asset=APPS_BUNDLE_ASSET)
+        resolved_app_refs.append(resolved_ref)
         extract_dir = package_dir / "extract"
         with ZipFile(bundle) as archive:
             archive.extractall(extract_dir)
@@ -60,7 +62,7 @@ def build_release(config, work_dir):
             else:
                 shutil.copy2(entry, target)
 
-    return release_dir
+    return release_dir, resolved_app_refs
 
 
 def render_app_configs(release_dir, app, values):
@@ -73,11 +75,14 @@ def render_app_configs(release_dir, app, values):
 
 def resolve_app_envs(config, work_dir, release_dir, age_key_file):
     pull_dir = work_dir / "envs"
+    resolved_env_refs = {}
     for app, app_config in config["apps"].items():
-        paths = [
+        downloaded = [
             download_ref(ref, pull_dir / app / str(index))
             for index, ref in enumerate(app_config["env_refs"], start=1)
         ]
+        paths = [path for path, _ in downloaded]
+        resolved_env_refs[app] = [resolved_ref for _, resolved_ref in downloaded]
         check_env_collisions(paths)
 
         plaintext = f"APP_NAME={app}\n" + "".join(decrypt_env(path, age_key_file) for path in paths)
@@ -86,6 +91,20 @@ def resolve_app_envs(config, work_dir, release_dir, age_key_file):
         app_env_path.chmod(0o600)
 
         render_app_configs(release_dir, app, parse_dotenv(plaintext))
+    return resolved_env_refs
+
+
+def write_release_manifest(release_dir, release_name, resolved_app_refs, resolved_env_refs):
+    manifest = {
+        "schema_version": 1,
+        "release": release_name,
+        "app_refs": resolved_app_refs,
+        "apps": sorted(resolved_env_refs),
+        "env_refs": resolved_env_refs,
+    }
+    manifest_path = release_dir / "manifest.json"
+    manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n")
+    manifest_path.chmod(0o644)
 
 
 def list_required_networks(release_dir):
@@ -123,6 +142,17 @@ def push_release(connection, archive_path, release_path):
     connection.run(f"chmod 600 {shlex.quote(release_path)}/apps/*/.env", hide=True)
 
 
+def stop_removed_apps(connection, current_path, apps):
+    result = connection.run(f"cat {shlex.quote(current_path)}/manifest.json 2>/dev/null || true", hide=True)
+    if not result.stdout.strip():
+        return
+    previous = json.loads(result.stdout)
+    removed = sorted(set(previous.get("apps", [])) - set(apps))
+    for app in removed:
+        compose_dir = f"{current_path}/apps/{app}"
+        connection.run(f"cd {shlex.quote(compose_dir)} 2>/dev/null && docker compose down || true", hide=True)
+
+
 def prune_releases(connection, releases_path, keep_releases):
     result = connection.run(f"ls -1dt {shlex.quote(releases_path)}/*/ 2>/dev/null || true", hide=True)
     releases = [line.strip().rstrip("/") for line in result.stdout.splitlines() if line.strip()]
@@ -131,7 +161,7 @@ def prune_releases(connection, releases_path, keep_releases):
         connection.run("rm -rf " + " ".join(shlex.quote(release) for release in stale), hide=True)
 
 
-def deploy_to_host(host, archive_path, apps, networks, config):
+def deploy_to_host(host, archive_path, apps, networks, config, release_name):
     connection = Connection(host)
     connection.client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
 
@@ -141,7 +171,6 @@ def deploy_to_host(host, archive_path, apps, networks, config):
 
     releases_path = f"{base_path}/releases"
     current_path = f"{base_path}/current"
-    release_name = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
     release_path = f"{releases_path}/{release_name}"
 
     bootstrap_host(connection, base_path, networks)
@@ -152,6 +181,7 @@ def deploy_to_host(host, archive_path, apps, networks, config):
         env_path = f"{release_path}/apps/{app}/.env"
         connection.run(f"echo {shlex.quote(f'DATA_DIR={data_dir}')} >> {shlex.quote(env_path)}", hide=True)
 
+    stop_removed_apps(connection, current_path, apps)
     connection.run(f"ln -sfn {shlex.quote(release_path)} {shlex.quote(current_path)}", hide=True)
 
     for app in apps:
@@ -183,16 +213,18 @@ def main():
         age_key_file.write_text(config["sops_age_key"])
         age_key_file.chmod(0o600)
 
-        release_dir = build_release(config, work_dir)
-        resolve_app_envs(config, work_dir, release_dir, age_key_file)
+        release_dir, resolved_app_refs = build_release(config, work_dir)
+        resolved_env_refs = resolve_app_envs(config, work_dir, release_dir, age_key_file)
+        release_name = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
+        write_release_manifest(release_dir, release_name, resolved_app_refs, resolved_env_refs)
         archive_path = archive_release(release_dir, work_dir)
         networks = list_required_networks(release_dir)
 
         apps = list(config["apps"])
 
         for host in config["hosts"]:
             print(f"Deploying to {host}")
-            deploy_to_host(host, archive_path, apps, networks, config)
+            deploy_to_host(host, archive_path, apps, networks, config, release_name)
 
 
 if __name__ == "__main__":

deploy/resolve.py

@@ -79,4 +79,4 @@ def download_ref(ref, out_dir, default_asset=None, run=subprocess.run):
     path = out_dir / resolved.asset
     if not path.is_file():
         raise RefError(f"{resolved.asset} was not found in {resolved.repo}@{tag}")
-    return path
+    return path, f"{resolved.repo}@{tag}:{resolved.asset}"