feat(recipes): add the bundled catalog rule and its three common recipes

CHANGELOG.md

@@ -1,5 +1,26 @@
 # Changelog
 
+## Unreleased
+
+### Features
+
+- Apply the `devtools` and `python-dev` recipes on AlmaLinux, Rocky, and
+  openSUSE. All eight bundled guests now carry both recipes.
+- Apply the `build-deps`, `service-tools`, and `pkg-tools` recipes on all eight
+  bundled guests. `build-deps` installs a C compiler, `make`, and
+  `pkg-config`, then reports each as an output. `service-tools` installs
+  `lsof`, `strace`, and the process tools, then reports whether the guest runs
+  systemd or OpenRC, and where `lsof` and `strace` are. `pkg-tools` installs
+  the tool that answers which package owns a file, then reports that tool and
+  the package manager.
+
+### Fixes
+
+- `stoat up` no longer hangs on a guest whose cloud-init keeps its run
+  directory root-only. Stoat polls `cloud-init status` on its own deadline and
+  retries an unreadable probe under the guest's escalation. Fedora 44 ships the
+  cloud-init version that caused the hang.
+
 ## v0.3.0
 
 Three enterprise Linux guests, common developer recipes, and read-only

CONTRIBUTING.md

@@ -89,9 +89,9 @@ reviewer rejects a PR that ignores them.
 
 ## Recipes
 
-Bundled recipes are `devtools`, `docker`, `python-dev`, `tailscale`, and
-`xfce`. The bundled catalog changes only through an approved design. Do not
-add opportunistic recipe IDs. `devtools` and `python-dev` are the two common
+Bundled recipes are `devtools`, `python-dev`, `build-deps`, `service-tools`,
+`pkg-tools`, `docker`, `tailscale`, and `xfce`. The bundled catalog changes
+only through an approved design. Do not add opportunistic recipe IDs. `devtools` and `python-dev` are the two common
 developer recipes. Write a new recipe in `~/.stoat/recipes/<name>/` from
 `stoat recipe new`; see `docs/recipes/writing-your-own.md` and
 `docs/recipes/sharing.md` for installing and pinning remote recipes. Every

docs/recipes/catalog.md

@@ -0,0 +1,115 @@
+# The bundled recipe catalog
+
+This page says what stoat bundles, why each entry is there, and what keeps the
+set from growing without limit.
+
+## Rules
+
+- Two common developer recipes stay as they are: `devtools` and `python-dev`.
+- A task that exists on every guest gets one recipe with a script per OS
+  family. It does not get one recipe per OS.
+- A recipe earns its place by needing real OS-specific behaviour. A package
+  name that differs is not enough on its own; a package group, a pattern, or a
+  different init system is.
+- At most three OS-specific recipes per guest, for tasks with no counterpart
+  elsewhere.
+- Every recipe declares schema 3 outputs and a health check, applies safely a
+  second time, and names no user account of its own.
+
+## What is bundled
+
+| Recipe | Guests | Purpose |
+|---|---|---|
+| `devtools` | all eight | Git, a compiler, an editor, basic fetch tools |
+| `python-dev` | all eight | Python 3, pip, an isolated environment |
+| `build-deps` | all eight | What building someone else's source tree needs |
+| `service-tools` | all eight | Inspecting a running service and the processes behind it |
+| `pkg-tools` | all eight | Querying the package manager beyond install and remove |
+| `docker` | five | Docker engine and the compose plugin |
+| `tailscale` | five | `tailscaled`, started and joined |
+| `xfce` | four | XFCE with autologin on tty1 |
+
+`devtools` and `build-deps` overlap on a compiler and `make`. The boundary is
+the task: `devtools` equips the VM for writing code in it, and `build-deps`
+equips it for compiling a project that expects autotools, `pkg-config`, and the
+distribution's own packaging headers. On Arch the boundary does not hold:
+`devtools` installs `base-devel`, and `build-deps` installs only `base-devel`,
+so `build-deps` is a subset of `devtools` on that guest.
+
+## The three new common recipes
+
+Each installs the tooling for one task. None writes a report, because a report
+generated at provision time is stale by the first time anyone reads it.
+
+### build-deps
+
+| Family | Packages |
+|---|---|
+| Debian, Ubuntu | `build-essential`, `pkg-config`, `autoconf`, `automake`, `libtool`, `dpkg-dev` |
+| Fedora, AlmaLinux, Rocky | the development tools group, `pkgconf-pkg-config`, `rpm-build` |
+| openSUSE | pattern `devel_basis` |
+| Arch | `base-devel` |
+| Alpine | `alpine-sdk`, `build-base` |
+
+Outputs: `compiler`, `make`, `pkg_config`.
+Health: each of the three responds to `--version`.
+
+The RPM and openSUSE entries are a group and a pattern, which the package
+manager expands. The group carries two ids: dnf5 on Fedora calls it
+`development-tools`, and dnf4 on AlmaLinux 9 and Rocky 9 calls it
+`development`. The script asks for the first id and falls back to the second. Arch's `base-devel` is a metapackage that pulls the same
+tools in one name. That is the OS-specific behaviour this recipe exists for.
+
+### service-tools
+
+| Family | Packages |
+|---|---|
+| Debian, Ubuntu | `lsof`, `strace`, `procps` |
+| Fedora, AlmaLinux, Rocky | `lsof`, `strace`, `procps-ng` |
+| openSUSE | `lsof`, `strace`, `procps` |
+| Arch | `lsof`, `strace`, `procps-ng` |
+| Alpine | `lsof`, `strace`, `procps`, `openrc` |
+
+Outputs: `service_manager` (`systemd` or `openrc`), `lsof`, `strace`.
+Health: the service manager answers a status query, and `lsof -v` runs.
+
+Alpine runs OpenRC and every other bundled guest runs systemd, so the health
+check and the reported manager differ by guest rather than by package name.
+
+### pkg-tools
+
+| Family | Packages |
+|---|---|
+| Debian, Ubuntu | `apt-file`, `dpkg-dev` |
+| Fedora, AlmaLinux, Rocky | `dnf-utils` |
+| openSUSE | `zypper`, `libzypp` |
+| Arch | `pacman-contrib` |
+| Alpine | `apk-tools` |
+
+Outputs: `query_tool` (the binary that answers "which package owns this file"),
+`manager`.
+Health: the query tool runs.
+
+## OS-specific candidates, not yet selected
+
+These have no counterpart on the other guests. None is implemented. Each needs
+its own contract tests and its own live qualification before it is claimed.
+
+| Guest | Candidate | Why it is OS-specific | Open question |
+|---|---|---|---|
+| AlmaLinux, Rocky, Fedora | SELinux tooling: `setroubleshoot-server`, `policycoreutils-python-utils` | SELinux exists only on the RPM family | Does a VM with SELinux in permissive mode need it? |
+| openSUSE | `osc`, the Open Build Service client | OBS is openSUSE's own build service | Is a build-service client in scope for a local VM tool? |
+| Arch | `namcap`, PKGBUILD linting | PKGBUILD is Arch's format | Overlaps `base-devel` from `build-deps` |
+| Alpine | `abuild`, APKBUILD tooling | APKBUILD is Alpine's format | Overlaps `alpine-sdk` from `build-deps` |
+| Debian, Ubuntu | `devscripts`, `lintian` | Debian packaging tooling | Overlaps `dpkg-dev` from `build-deps` |
+
+Four of the five overlap a common recipe, which is the rule in this page doing
+its job. Only the SELinux entry is clearly separate, and it needs a decision
+about whether a permissive-mode VM benefits from it.
+
+## Qualification
+
+A recipe is claimed for a guest after it has been applied on that guest's
+advertised release, from a binary built at a known commit, with the recipe
+reporting healthy and its declared outputs resolving to real executables in the
+guest. The retained evidence names the run, the binary hash, and the commit.

docs/recipes/overview.md

@@ -67,29 +67,32 @@ succeeded. It does not reboot live VMs because a live root is temporary.
 ## Bundled recipes
 
 The bundled catalog changes only through an approved design. Do not add
-opportunistic recipe IDs. It currently contains these five recipes:
+opportunistic recipe IDs. It currently contains these eight recipes:
 
 | Recipe | Supported guests | Purpose |
 |---|---|---|
 | `devtools` | Alpine, Ubuntu, Debian, Fedora, Arch, AlmaLinux, Rocky, openSUSE | Git, compiler tools, editor and basic fetch tools |
-| `docker` | Alpine, Ubuntu, Debian, Fedora, Arch | Docker engine and compose plugin; schema 3 parameter `user`, output `socket`, health check `docker info` |
 | `python-dev` | Alpine, Ubuntu, Debian, Fedora, Arch, AlmaLinux, Rocky, openSUSE | Python 3, pip, and an isolated development environment; schema 3 parameters `user` and optional `venv_dir`, with smoke-only mode when `venv_dir` is empty |
+| `build-deps` | Alpine, Ubuntu, Debian, Fedora, Arch, AlmaLinux, Rocky, openSUSE | What building someone else's source tree needs; schema 3 outputs `compiler`, `make`, `pkg_config`, health check each responds to `--version` |
+| `service-tools` | Alpine, Ubuntu, Debian, Fedora, Arch, AlmaLinux, Rocky, openSUSE | Inspecting a running service and the processes behind it; schema 3 outputs `service_manager` (`systemd` or `openrc`), `lsof`, `strace`, health check service manager answers status query and `lsof -v` runs |
+| `pkg-tools` | Alpine, Ubuntu, Debian, Fedora, Arch, AlmaLinux, Rocky, openSUSE | Querying the package manager beyond install and remove; schema 3 outputs `query_tool`, `manager`, health check query tool runs |
+| `docker` | Alpine, Ubuntu, Debian, Fedora, Arch | Docker engine and compose plugin; schema 3 parameter `user`, output `socket`, health check `docker info` |
 | `tailscale` | Alpine, Ubuntu, Debian, Fedora, Arch | Install and start `tailscaled`; schema 3 required secret `authkey`, health check `tailscale version` |
 | `xfce` | Alpine, Ubuntu, Debian, Arch | XFCE desktop with autologin startx on tty1; requests a disk-VM reboot |
 
-`devtools` and `python-dev` are the two common developer recipes. The
-remaining bundled recipes are existing capabilities and are not part of that
-developer pair.
+`devtools`, `python-dev`, `build-deps`, `service-tools`, and `pkg-tools` run
+on all guests. `docker`, `tailscale`, and `xfce` require capabilities present
+only on some guests.
 
 `python-dev` requires the configured guest account. For example, a cloud guest
 whose SSH account is `stoat` must use `python-dev.user=stoat`; an Alpine guest
 using `root` must use `python-dev.user=root`.
 
 The scripts are in `internal/recipes/bundled/` in the source tree. Each
-recipe has a manifest. Docker, devtools, python-dev, and Tailscale use
-OS-specific script overrides because package names and repository setup differ.
-XFCE uses one script with guest prelude verbs and therefore has no `[scripts]`
-table.
+recipe has a manifest. Docker, devtools, python-dev, Tailscale, build-deps,
+service-tools, and pkg-tools use OS-specific script overrides because package
+names and repository setup differ. XFCE uses one script with guest prelude
+verbs and therefore has no `[scripts]` table.
 
 Cloud images use their own package manager and a cloud-init seed. Cloud-init
 currently wraps recipe bodies in shell commands; it does not perform the SSH

docs/reference/cli.md

@@ -518,8 +518,12 @@ not an additional manifest-match condition.
 ```
 $ stoat recipes --os alpine --backend apkovl
 NAME                           DESCRIPTION
+build-deps                     toolchain for building software from source
 devtools                       git, a compiler, an editor and basic fetch tools
 docker                         Docker engine and the compose plugin
+pkg-tools                      tools for querying the package manager
+python-dev                     Python 3 with pip and an isolated development environment
+service-tools                  tools for inspecting services and the processes behind them
 tailscale                      Tailscale daemon, installed and started (join manually)
 xfce                           XFCE desktop with autologin startx on tty1
 ```
@@ -555,8 +559,12 @@ search order.
 ```
 $ stoat recipe list
 NAME                 SCOPE     COMMIT   DESCRIPTION
+build-deps           bundled            toolchain for building software from source
 devtools             bundled            git, a compiler, an editor and basic fetch tools
 docker               bundled            Docker engine and the compose plugin
+pkg-tools            bundled            tools for querying the package manager
+python-dev           bundled            Python 3 with pip and an isolated development environment
+service-tools        bundled            tools for inspecting services and the processes behind them
 tailscale            bundled            Tailscale daemon, installed and started (join manually)
 xfce                 bundled            XFCE desktop with autologin startx on tty1
 ```

internal/cli/subcommands_test.go

@@ -237,13 +237,13 @@ func TestRecipesListsAndFilters(t *testing.T) {
 		t.Fatal("recipes returned nothing with no filter")
 	}
 
-	// debian satisfies xfce, devtools, docker, python-dev, and tailscale OS lists.
+	// debian satisfies xfce, devtools, docker, python-dev, build-deps, service-tools, pkg-tools, and tailscale OS lists.
 	_, only := runJSON(t, "recipes", "--os", "debian", "--backend", "cloudinit")
 	debian, _ := dataOf(t, only)["recipes"].([]any)
 	if len(debian) == 0 {
 		t.Fatal("recipes --os debian --backend cloudinit returned nothing")
 	}
-	want := map[string]bool{"xfce": true, "devtools": true, "docker": true, "python-dev": true, "tailscale": true}
+	want := map[string]bool{"xfce": true, "devtools": true, "docker": true, "python-dev": true, "tailscale": true, "build-deps": true, "service-tools": true, "pkg-tools": true}
 	for _, r := range debian {
 		m, _ := r.(map[string]any)
 		name, _ := m["name"].(string)

internal/recipes/bundled/build-deps/install-alpine.sh

@@ -0,0 +1,33 @@
+#!/bin/sh
+# toolchain for building software from source. Runs as root over ssh on a
+# booted Alpine VM.
+set -e
+
+# -c enables community (alpine-sdk lives there, outside the base set); -1
+# picks a mirror and refreshes indexes, so no separate `apk update`.
+# setup-apkrepos runs apk update with no lock-wait, so another apk that holds
+# the database lock fails it with exit 99. Retry until the lock frees, up to
+# ~60s.
+n=0
+until setup-apkrepos -c -1; do
+    n=$((n + 1))
+    [ "$n" -ge 30 ] && { echo "apk database stayed locked; giving up" >&2; exit 1; }
+    sleep 2
+done
+
+stoat_pkg_install alpine-sdk build-base
+
+compiler=$(command -v cc 2>/dev/null || command -v gcc 2>/dev/null || true)
+make_bin=$(command -v make 2>/dev/null || true)
+pkg_config=$(command -v pkg-config 2>/dev/null || command -v pkgconf 2>/dev/null || true)
+[ -n "$compiler" ] || { echo "build-deps: C compiler was not installed" >&2; exit 1; }
+[ -n "$make_bin" ] || { echo "build-deps: make was not installed" >&2; exit 1; }
+[ -n "$pkg_config" ] || { echo "build-deps: pkg-config was not installed" >&2; exit 1; }
+if [ -n "${STOAT_OUTPUT:-}" ]; then
+    {
+        printf 'compiler=%s\n' "$compiler"
+        printf 'make=%s\n' "$make_bin"
+        printf 'pkg_config=%s\n' "$pkg_config"
+    } >> "$STOAT_OUTPUT"
+fi
+echo "build-deps installed: alpine-sdk, build-base"

internal/recipes/bundled/build-deps/install-arch.sh

@@ -0,0 +1,22 @@
+#!/bin/sh
+# toolchain for building software from source. Runs as root over ssh on a
+# booted Arch VM.
+set -e
+
+stoat_pkg_setup
+stoat_pkg_install base-devel
+
+compiler=$(command -v cc 2>/dev/null || command -v gcc 2>/dev/null || true)
+make_bin=$(command -v make 2>/dev/null || true)
+pkg_config=$(command -v pkg-config 2>/dev/null || command -v pkgconf 2>/dev/null || true)
+[ -n "$compiler" ] || { echo "build-deps: C compiler was not installed" >&2; exit 1; }
+[ -n "$make_bin" ] || { echo "build-deps: make was not installed" >&2; exit 1; }
+[ -n "$pkg_config" ] || { echo "build-deps: pkg-config was not installed" >&2; exit 1; }
+if [ -n "${STOAT_OUTPUT:-}" ]; then
+    {
+        printf 'compiler=%s\n' "$compiler"
+        printf 'make=%s\n' "$make_bin"
+        printf 'pkg_config=%s\n' "$pkg_config"
+    } >> "$STOAT_OUTPUT"
+fi
+echo "build-deps installed: base-devel (gcc/make/pkg-config)"

internal/recipes/bundled/build-deps/install-debian.sh

@@ -0,0 +1,23 @@
+#!/bin/sh
+# toolchain for building software from source. Runs as root over ssh on a
+# booted Ubuntu or Debian VM.
+set -e
+
+export DEBIAN_FRONTEND=noninteractive
+stoat_pkg_setup
+stoat_pkg_install build-essential pkg-config autoconf automake libtool dpkg-dev
+
+compiler=$(command -v cc 2>/dev/null || command -v gcc 2>/dev/null || true)
+make_bin=$(command -v make 2>/dev/null || true)
+pkg_config=$(command -v pkg-config 2>/dev/null || command -v pkgconf 2>/dev/null || true)
+[ -n "$compiler" ] || { echo "build-deps: C compiler was not installed" >&2; exit 1; }
+[ -n "$make_bin" ] || { echo "build-deps: make was not installed" >&2; exit 1; }
+[ -n "$pkg_config" ] || { echo "build-deps: pkg-config was not installed" >&2; exit 1; }
+if [ -n "${STOAT_OUTPUT:-}" ]; then
+    {
+        printf 'compiler=%s\n' "$compiler"
+        printf 'make=%s\n' "$make_bin"
+        printf 'pkg_config=%s\n' "$pkg_config"
+    } >> "$STOAT_OUTPUT"
+fi
+echo "build-deps installed: build-essential, pkg-config, autoconf, automake, libtool, dpkg-dev"

internal/recipes/bundled/build-deps/install-rpm.sh

@@ -0,0 +1,28 @@
+#!/bin/sh
+# toolchain for building software from source. Runs as root on a booted dnf
+# guest: Fedora, AlmaLinux or Rocky.
+set -e
+
+stoat_pkg_setup
+# The group carries the same packages under two ids: dnf5 on Fedora calls it
+# development-tools, and dnf4 on AlmaLinux 9 and Rocky 9 calls it development.
+# dnf fails with "Module or Group is not available" on the wrong id.
+if ! stoat_pkg_install @development-tools; then
+    stoat_pkg_install @development
+fi
+stoat_pkg_install pkgconf-pkg-config rpm-build
+
+compiler=$(command -v cc 2>/dev/null || command -v gcc 2>/dev/null || true)
+make_bin=$(command -v make 2>/dev/null || true)
+pkg_config=$(command -v pkg-config 2>/dev/null || command -v pkgconf 2>/dev/null || true)
+[ -n "$compiler" ] || { echo "build-deps: C compiler was not installed" >&2; exit 1; }
+[ -n "$make_bin" ] || { echo "build-deps: make was not installed" >&2; exit 1; }
+[ -n "$pkg_config" ] || { echo "build-deps: pkg-config was not installed" >&2; exit 1; }
+if [ -n "${STOAT_OUTPUT:-}" ]; then
+    {
+        printf 'compiler=%s\n' "$compiler"
+        printf 'make=%s\n' "$make_bin"
+        printf 'pkg_config=%s\n' "$pkg_config"
+    } >> "$STOAT_OUTPUT"
+fi
+echo "build-deps installed: the development tools group, pkgconf-pkg-config, rpm-build"

internal/recipes/bundled/build-deps/install-zypper.sh

@@ -0,0 +1,22 @@
+#!/bin/sh
+# toolchain for building software from source. Runs as root on a booted
+# openSUSE VM.
+set -e
+
+stoat_pkg_setup
+stoat_pkg_install -t pattern devel_basis
+
+compiler=$(command -v cc 2>/dev/null || command -v gcc 2>/dev/null || true)
+make_bin=$(command -v make 2>/dev/null || true)
+pkg_config=$(command -v pkg-config 2>/dev/null || command -v pkgconf 2>/dev/null || true)
+[ -n "$compiler" ] || { echo "build-deps: C compiler was not installed" >&2; exit 1; }
+[ -n "$make_bin" ] || { echo "build-deps: make was not installed" >&2; exit 1; }
+[ -n "$pkg_config" ] || { echo "build-deps: pkg-config was not installed" >&2; exit 1; }
+if [ -n "${STOAT_OUTPUT:-}" ]; then
+    {
+        printf 'compiler=%s\n' "$compiler"
+        printf 'make=%s\n' "$make_bin"
+        printf 'pkg_config=%s\n' "$pkg_config"
+    } >> "$STOAT_OUTPUT"
+fi
+echo "build-deps installed: devel_basis pattern"

internal/recipes/bundled/build-deps/recipe.toml

@@ -0,0 +1,24 @@
+name = "build-deps"
+description = "toolchain for building software from source"
+os = ["alpine", "ubuntu", "debian", "fedora", "arch", "almalinux", "rocky", "opensuse"]
+stage = "provision"
+script = "install-debian.sh"
+schema = 3
+
+[scripts]
+alpine = "install-alpine.sh"
+ubuntu = "install-debian.sh"
+debian = "install-debian.sh"
+fedora = "install-rpm.sh"
+almalinux = "install-rpm.sh"
+rocky = "install-rpm.sh"
+opensuse = "install-zypper.sh"
+arch = "install-arch.sh"
+
+[outputs]
+compiler = "path to the C compiler"
+make = "path to make"
+pkg_config = "path to pkg-config"
+
+[health]
+check = "cc --version >/dev/null 2>&1 && make --version >/dev/null 2>&1 && { pkg-config --version >/dev/null 2>&1 || pkgconf --version >/dev/null 2>&1; }"