diff --git a/README.md b/README.md index 6f677b4..753c61a 100644 --- a/README.md +++ b/README.md @@ -19,12 +19,26 @@ session to type `vault.yaml` into. Every secret read is logged with the requesting cert's serial number — *"who pulled what, when"* is a one-liner in Cloud Audit Logs. -**Status:** Zero-touch and live for the **m4 pools** (2026-07). One flow, driven by the +**Status:** Zero-touch and live for the **m4 pools**. One flow, driven by the `orchestrator/` CLI (`reprovision`): quarantine → drain → EACS wipe → DEP re-enroll → SimpleMDM profiles + signed **bootstrap PKG** → SCEP cert in keychain → auto-mint SecureToken + escrow Bootstrap Token → bootstrap PKG fetches `vault.yaml` → puppet → worker re-registered -in Taskcluster → auto-unquarantine. Proven end-to-end across the m4 staging pool -(`macmini-m4-111…115`) and verified reprovision-ready on the m4 production pool. +in Taskcluster → validate → unquarantine. + +🏁 **Proven at wave scale (2026-08-14, RELOPS-2515):** 37 hosts taken from DEP arrival to +production in a day — **SIP enabled throughout**, no Recovery trip, no per-host console work. +The full operator sequence and every failure mode we hit is in the +**[🧑‍🚀 Runbook](docs/RUNBOOK.md)**. + +```bash +mint → os-update → preflight → add-to-group → validate → unquarantine +``` + +Two of those steps earned their place immediately. **`add-to-group`** automates what used to be +a hand-click in the SimpleMDM UI — and since group membership delivers the bootstrap PKG, +`puppet_role`, the CLT, the admin key and passwordless sudo *all at once*, it is the single "go" +action. **`validate`** is the fitness gate: it caught **31 of 33** hosts unfit on a display fault +that puppet, the sentinel, the worker and the disk all reported as perfectly healthy. **Why the SecureToken mint needs an on-network runner.** DEP skips Setup Assistant, so the managed admin holds no SecureToken until a PAM (password) login — and Apple only grants the @@ -172,7 +186,9 @@ silently drops URI SANs with URL-encoded chars (e.g. `Mac%20mini`). │ ├── INSTALL-on-worker.md ─ install guide + threat-model table │ └── fetch-vault-mtls.swift ─ historical URLSession variant (dead end) │ -├── docs/ 📖 rendered walkthrough (index.html + deep-dive.html) +├── docs/ 📖 RUNBOOK.md ⭐ operator field guide (provision + reprovision, +│ per-action -j table, symptom-first troubleshooting) +│ + rendered walkthrough (index.html + deep-dive.html) │ + reprovision-architecture.html (one-page flow diagram) │ + eacs.html (what EACS actually does, button-press → re-enroll) ├── .github/workflows/test.yml ✅ CI: pytest + terraform fmt/validate @@ -316,6 +332,26 @@ strict checks all green. ## 🛠️ Open work +### 🐞 Known bugs, both found at wave scale on 2026-08-14 + +- 🎚️ **`add-to-group --quarantine-on-register` makes one `-j` serve two bottlenecks.** It couples + a SimpleMDM-bound action (3 API calls/host) to a 30-minute Taskcluster-bound wait. Raise `-j` + for wall-clock and SimpleMDM 429s; lower it for SimpleMDM and 33 hosts serialize into ~5½ hours. + At `-j12` it *looked* like 5 hosts failed — but **12 had already been added to the group** (the + add succeeded, the follow-up `push_apps` 429'd), so killing the batch orphaned 12 live + bootstraps with no watcher, which would have put them into production unvalidated. **Fix:** + rate-limit the SimpleMDM calls independently of `-j`, or split the watch into its own batch + action. Workaround in the [Runbook](docs/RUNBOOK.md#j-trap). +- 🪟 **The Safari automation AppleScript assumes a window exists.** It does `window 1 of process + "Safari"` without launching Safari or waiting for a window, so once a session is disturbed every + retry fails forever with `Invalid index` — and `open -a Safari` doesn't help, because Safari + restores a windowless session. A reboot is the reliable repair. Needs a ronin_puppet fix to + launch-and-wait. Also unsuppressed: **`Setup Assistant -MiniBuddy`**, the *per-user* first-login + assistant, which `Skip Setup Assistant - All Screens` does **not** cover and which steals focus + from the UI scripting on any host where `cltbld` is freshly created. + +### 📋 Everything else + - 🪪 **Per-role expansion** — verified live state (2026-07): - **Live** (SCEP provisioner + populated `vault-*` secret): `gecko_t_osx_1500_m4` (`scep-no-sip`), `gecko_t_osx_1500_m4_staging` (`scep-osx-1500-m4-staging`), and diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md new file mode 100644 index 0000000..5a5cd98 --- /dev/null +++ b/docs/RUNBOOK.md @@ -0,0 +1,361 @@ +# 🧑‍🚀 Provisioning Runbook + +**The operator's field guide. Copy-paste sequences up top, then the failure modes that +actually happen — each one with the symptom you'll see first, not the cause.** + +Every gotcha in here cost someone a real debugging session. They're written symptom-first +because that's the order you meet them in. + +--- + +## ⚡ TL;DR — the two sequences + +### 🆕 Fresh hardware → production + +```bash +cd ~/git/relops-bootstrap/orchestrator +HOSTS=~/Desktop/wave.txt # one short hostname per line, '#' comments ok + +uv run reprovision batch $HOSTS --action mint -j3 # SecureToken (idempotent, often a no-op) +uv run reprovision batch $HOSTS --action os-update -j3 # in-place upgrade to target OS +# ⏳ wait ~35 min: ~14GB download, then startosinstall, then a reboot +uv run reprovision batch $HOSTS --action preflight --allow-sip-enabled -j3 +uv run reprovision batch $HOSTS --action add-to-group --quarantine-on-register -j2 +# ⏳ blocks ~30 min per host: this is the bootstrap, and the watch that holds each host +uv run reprovision batch $HOSTS --action validate -j3 # ← the gate. Do not skip. +uv run reprovision unquarantine macmini-m4-XXX # only what validate passed +``` + +### 🔄 Existing host → wiped → back to production + +```bash +uv run reprovision add-to-group macmini-m4-211 # ⚠️ REQUIRED FIRST — see below +uv run reprovision run macmini-m4-211 # quarantine→drain→wipe→re-enroll→mint→BST→bootstrap +uv run reprovision validate macmini-m4-211 +uv run reprovision unquarantine macmini-m4-211 +``` + +> ⚠️ **`run` has no group step.** It assumes the bootstrap PKG will arrive "during DEP +> convergence" — which only happens if the host is already in a group carrying that PKG. +> Production groups **do not** carry it. Skip the `add-to-group` and the wipe succeeds, then +> the host waits a full hour for a sentinel that can never be written. Always `add-to-group` +> first; it's additive and idempotent. + +--- + + + +## 🎚️ Concurrency: `-j` means something different per action + +**This is the single easiest way to break a wave.** `-j` is not one knob — each action is +bound by a different resource. + +| action | safe `-j` | bound by | notes | +|---|---|---|---| +| `mint` | 3–4 | SSH | fast; often a no-op when the token already exists | +| `os-update` | **irrelevant** ⚠️ | the mirror | **launch-and-return** — `-j` paces only the *launches*. The **hosts-file length is the real concurrency.** 33 hosts = 33 simultaneous 14GB pulls | +| `preflight` | 3–4 | SSH | read-only | +| `add-to-group` | **2** 🚨 | **SimpleMDM API** | 3 API calls per host. At `-j12` the 429 retry budget blew and hosts failed | +| `validate` | 3 | SSH | read-only | +| `provision` | 3 | MDC1 imaging throughput | matches the runner's `RUNNER_MAX_CONCURRENT` | + + + +### 🪤 The `-j` trap that bit us hardest + +`add-to-group --quarantine-on-register` couples a **SimpleMDM-bound action** to a +**30-minute Taskcluster-bound wait**, so one `-j` has to serve both. Raise it for wall-clock +and you hammer SimpleMDM; lower it for SimpleMDM and 33 hosts serialize into ~5½ hours. + +At `-j12` on 2026-08-14 it looked like 5 hosts failed. What actually happened: **12 hosts had +already been added to the group** (the add succeeded, the follow-up `push_apps` 429'd), so +killing the batch orphaned 12 live bootstraps with no watcher. They'd have gone into +production unvalidated. + +**Workaround until this is fixed** — split it along the resource boundary: + +```bash +# 1. the SimpleMDM half, gently +uv run reprovision batch $HOSTS --action add-to-group -j2 + +# 2. the Taskcluster half, all at once (pre-resolve creds so N processes don't hammer 1Password) +export REPROVISION_TC_CLIENT_ID=... REPROVISION_TC_ACCESS_TOKEN=... +export REPROVISION_QUARANTINE_ON_REGISTER_MAX_WAIT_SECONDS=5400 # 900s default is too short! +grep '^macmini' $HOSTS | xargs -P 33 -I{} \ + sh -c 'nohup uv run reprovision quarantine-on-register {} > /tmp/qor/{}.log 2>&1' +``` + +> 🧠 **Why 5400.** The default budget is sized for a watch started *after* bootstrap, when +> registration is a minute away. Started at group-add it must span the **whole** bootstrap +> (~30 min). Too short and the watch expires before there's anything to quarantine — and the +> host goes live unheld, i.e. exactly the failure the flag exists to prevent. + +--- + +## 🔬 Never trust `ok`. Verify the thing itself. + +`ok` means *the command succeeded*, which is often **not** the same as *the work happened*. + +| action | what `ok` really means | how to actually verify | +|---|---|---| +| `os-update` | the upgrade **launched** | `pgrep -x curl` + watch `/private/tmp/InstallAssistant-*.zip` grow. 10 hosts at exactly `0:10` is the launch check, not progress — and it's the same signature the old self-bootout bug produced | +| `add-to-group` | this host's 3 API calls returned | **query group membership** — an add can succeed while `push_apps` fails, so the ✗ list understates what changed | +| `mint` | the step ran | `sysadminctl -secureTokenStatus admin` says `ENABLED` | +| `validate` | 🟢 this one is trustworthy | it *is* the verification | + +```bash +# group membership — the authoritative answer +uv run python -c " +from orchestrator.clients import simplemdm as s +print(len(s.assignment_group_device_ids(2417981)), 'devices')" +``` + +--- + +## 🖥️ The KVM / refresh-rate trap + +**Symptom:** a host takes tasks and fails **100%** of them in ~43s each, then reboots. Looks +like a boot loop or a dead unit. + +**Cause:** the KVM negotiates **1280x1024@75Hz**. mozharness runs a pre-test refresh-rate +check and *fatally halts* at anything but 60Hz — before a single test runs. + +``` +Running pre test command verify refresh rate with '.../macosx_resolution_refreshrate.py --check' + Refresh Rate: 75.00 Hz + ERROR: expected refresh rate = 60.00, instead got 75.00. + FATAL - Halting on failure +Exit Code: 3 run-tests - Wall time: 0s +``` + +**This is the rack default, not a one-off.** On 2026-08-14, **31 of 33** hosts came up at +75Hz. The two exceptions were the only two whose first task didn't fail. + +### 😈 Why it's so easy to misdiagnose + +- The worker exits **code 0** with `Host is not dynamically provisioned; exiting`. Nothing + looks crashed. +- Disk, semaphores, puppet and the bootstrap are **all healthy** and byte-identical to a + working host. +- `system_profiler SPDisplaysDataType` over SSH shows **only the GPU, no display section**, so + it will neither confirm nor deny the refresh rate. +- A `TROUBLE`-style counter built on `Aborted:true` / `task exception` / `malformed-payload` + stays at **0** — these are ordinary task *failures*. + +**Fix:** set the KVM correctly, then `validate`. ❌ **Reprovisioning cannot fix it** — a wipe +does not change what the KVM negotiates. + +**Triage across a pool** — grep failed task logs for `instead got`; display faults separate +from real test failures instantly. A *missing* `live_backing.log` means something else +entirely (the task never ran). + +```bash +# read it straight off the host (must be inside the GUI session — see below) +ssh admin@$H 'uid=$(id -u cltbld); sudo launchctl asuser "$uid" /usr/local/bin/python3 -c " +import Quartz +d=Quartz.CGMainDisplayID(); m=Quartz.CGDisplayCopyDisplayMode(d) +print(\"%.2fHz %dx%d\" % (Quartz.CGDisplayModeGetRefreshRate(m), + Quartz.CGDisplayModeGetPixelWidth(m), Quartz.CGDisplayModeGetPixelHeight(m)))"' +``` + +> 🪟 **Over a plain SSH login this returns `0.00Hz 0x0`** — not an error, just zeros, which a +> naive check sails straight past. CoreGraphics needs a window server, so you must hop into the +> console user's session with `launchctl asuser`. And it only works **after** bootstrap: +> `cltbld` doesn't exist until puppet creates it, which is why `validate` can't live in +> `preflight`. + +--- + + + +## 🎭 Safari remote automation: the two ways it wedges + +The AppleScript UI-scripting step is the most fragile part of the bootstrap. Both failures +present identically — `safari-*-has-run` semaphores never appear and puppet retries forever. + +### 1. 🧙 MiniBuddy steals focus + +**Symptom:** semaphores never written; `frontmost: Setup Assistant`. + +`Setup Assistant -MiniBuddy` is the **per-user first-login assistant** — and +`Skip Setup Assistant - All Screens` does **not** suppress it. That profile governs *device* +setup at first boot; MiniBuddy runs when a brand-new **user account** logs in for the first +time. On a reprovisioned host `cltbld` is recreated by puppet after the wipe, so it gets a +fresh one. It sits in front of Safari and `System Events` clicks go to the frontmost app. + +```bash +sudo pkill -f "Setup Assistant" +``` + +### 2. 🪟 Safari running with zero windows + +**Symptom:** `Can't get window 1 of process "Safari". Invalid index.` + +The script assumes an already-running Safari **with a window**. It neither launches the app +nor waits for a window. If a previous attempt closed them, every retry fails forever — and +`open -a Safari` won't help, because Safari restores a windowless session. + +**Fix: reboot.** A clean login gives Safari real windows (verified: 0 → 2 windows), and that's +how the ~50 successful provisions worked. + +```bash +ssh admin@$H 'sudo shutdown -r now' +``` + +> 🐛 **Known bug, needs a ronin_puppet fix:** the AppleScript should *launch* Safari and *wait* +> for a window rather than assuming one. Until then, a disturbed session wedges permanently and +> the repair is a reboot. + +**Don't "just log in again"** to fix a stalled host — `cltbld` is already logged in, and a +fresh login re-triggers MiniBuddy. + +--- + +## 🆔 Hostname is not a device key + +**Symptom:** `add-to-group` reports `no SimpleMDM device with that name — has it finished DEP +enrolling?` on a host that is enrolled, upgraded, and answering SSH. + +A fresh DEP arrival is named **`Mac mini`** in SimpleMDM (`device_name` like `Mac mini (39)`). +The hostname comes from **DHCP**, and nothing in ronin_puppet runs `scutil --set`, so it is +never written back. The API answers **HTTP 200 with zero hits** — silent and confident. + +Only *already-provisioned* hosts carry their hostname (every older r8 shows +`name='macmini-r8-118'`), which is why this looked correct when tested against the existing +fleet. Resolution is now by **serial**, read from the host over SSH. 🔧 Fixed in #65. + +--- + +## 🔐 1Password will drop out mid-wave + +**Symptom:** `couldn't read op://...` or `timed out reading op://... after 30s`, on many hosts +at once. **Not a host problem.** + +`batch` resolves secrets **once in the parent** and passes them to children via `REPROVISION_*` +(#68) — before that, a 10-host batch fired 10+ `op read` calls in seconds and lost **9 of 10 +hosts**. But the pre-warm is best-effort: if the *parent's* read fails, children fall back and +fail too. + +```bash +op read "op://RelOps/RelOps Worker Admin Key/notesPlain" >/dev/null && echo primed +``` + +If that hangs, `op` is waiting on a biometric/desktop-app approval — approve it and retry. ⚠️ +Standalone commands (not `batch`) each resolve their own secrets, so pre-resolve into env when +looping over many hosts by hand. + +--- + +## 🧨 SimpleMDM group rules + +- ✅ **ADD, never MOVE.** Assignment groups are additive. *Moving* a host out of a group strips + every profile that group carried — on m4-214 a move silently removed **Skip Setup Assistant** + and the **FDA SSH Keygen Wrapper**, and the host then hung on the Wi-Fi pane at first boot, + presenting as "Safari automation is broken". Cost most of a day. There is deliberately **no + remove/move verb** in the client, and a test asserts it stays that way. +- 🚫 **Never add the bootstrap PKG to a production group.** Every member would run the bootstrap + mid-task. `PROTECTED_GROUP_IDS` refuses this before any HTTP call, hard-coded — a guard you + can override with an env var is not a guard. +- ⚠️ **Membership does not prove the PKG was pushed.** The already-a-member path skips + `push_apps`, so a host added by hand in the UI can sit in the group with nothing installed. + `add-to-group` warns when the payload is missing. + +--- + +## 🍏 Apple platform facts that shape everything + +| fact | consequence | +|---|---| +| 🔥 **EACS re-enables SIP** | Never infer post-wipe SIP state from what was set before. Proven on m4-214. Run `csrutil status`. | +| 🎫 DEP skips Setup Assistant → **no SecureToken** | Only an interactive PAM login grants the *first* token. Key-based SSH cannot. This is why the runner must be on-network. | +| 🔑 SecureToken arrival is **inconsistent** | 240–244 needed a manual login; 245–254 arrived with tokens; of 255–288 only 2 needed minting. Don't assume either way — `mint` is idempotent, run it. | +| 🗝️ **BST escrowed ≠ BST generated** | `wipe` uses `DoNotObliterate` and **fails closed** without an escrowed Bootstrap Token — deliberately, so a BST-less box never falls back to a full obliterate. | +| 🖥️ Headless minis don't composite | Screenshots succeed but come out blank. No script fix; needs display infra. | +| 🔁 Hosts reboot **between tasks** | A `worker=down` reading often just means you caught one mid-cycle. `generic-worker`'s `Resolved N tasks in total` is **per session** and re-reads `1` forever — count distinct `Resolving task ` lines instead. | + +--- + +## 🌐 Infrastructure notes + +- 📦 **The `releng-pxe1` mirror is nginx now**, not the old single-threaded `python3 -m + http.server` with a listen backlog of 5. Measured while upgrading: **107 MB/s at 4 concurrent, + 205 at 10, 314 at 12, 681 at 23**, no stalls. It is not the bottleneck. +- 🌍 **DNS: use the system resolver.** `host` and `nslookup` query DNS directly and bypass the + VPN's split-DNS, so they `NXDOMAIN` hosts you can SSH into *right now*. Use + `dscacheutil -q host -a name `. A bad DNS probe once "found" 14 nonexistent hosts. +- ⏱️ **A DNS race can break the vault fetch.** `curl: (6) Could not resolve host: + forge.relops.mozilla.com` early in bootstrap means the network wasn't up yet. It retries; + check whether `/var/root/vault.yaml` exists before treating it as fatal. +- 🐚 **zsh doesn't word-split.** `for n in $HOSTS` iterates **once** with the whole string, + silently producing empty results. Use `for n in $(echo $HOSTS)` or an array. This has burned + us repeatedly in throughput one-liners. + +--- + +## 🚦 Quarantine discipline + +**Always `validate` before releasing.** It is the only step that catches a host which is +perfect on every other signal and still fails 100% of tasks. + +``` +exit 0 → fit → safe to unquarantine +exit 2 → not ready → hasn't bootstrapped; nothing to judge yet +exit 1 → UNFIT → do not release; the output names why +``` + +`--quarantine-on-register` **narrows the race, it does not close it.** The worker registers +about a minute after the sentinel and can `claimWork` immediately; the watch polls every 5s, so +exposure is seconds. Expect **each host to claim exactly one task** before the hold lands. + +That one task is a **free early-warning signal** — on 2026-08-14 it's what revealed the +fleet-wide 75Hz fault. The quarantine kept the damage to one task per host (31 total) instead of +~15 each (450+), which is the rate an unheld bad host burns work at. + +```bash +# classify a pool's failures fast +# "instead got" → display/KVM no log → the task never ran otherwise → real failure +``` + +--- + +## 🗺️ Where things live + +| what | where | +|---|---| +| Per-host batch logs | `~/.local/state/reprovision/batch-/.log` | +| Bootstrap driver log (on host) | `/var/log/m4-bootstrap-driver.log` | +| Bootstrap log (on host) | `/var/log/m4-bootstrap.log` | +| Completion sentinel | `/var/log/m4-bootstrap-complete` | +| PKG postinstall log | `/var/log/m4-bootstrap-pkg-postinstall.log` | +| Worker log | `/opt/worker/logs/stderr.log` | +| Semaphores | `/var/tmp/semaphore/`, `/Users/cltbld/Library/Preferences/semaphore/` | +| Bootstrap group | SimpleMDM assignment group **2417981** (`gecko-t-osx-1500-m4-bootstrap`) | +| Production group | **2017918** — 🚫 never add the bootstrap PKG here | + +`startosinstall` writes thousands of `Preparing: N.N%` lines, so `tail` is useless on the +upgrade log — filter with `grep -v 'Preparing:'`. + +--- + +## 🧯 Quick triage table + +| symptom | likely cause | first move | +|---|---|---| +| 100% task failure, ~43s each, worker exits 0 | KVM at 75Hz | read the display mode; fix the KVM | +| Safari semaphores never appear | MiniBuddy focus **or** windowless Safari | `pkill -f "Setup Assistant"`, else reboot | +| `no SimpleMDM device with that name` | hostname isn't a device key | ensure SSH works so the serial can be read | +| Many hosts fail on `op://` at once | 1Password session dropped | `op read ` to prime, retry | +| Host waits forever for the sentinel | not in a group carrying the PKG | `add-to-group` | +| `NO-DNS` on a host you can SSH to | `host`/`nslookup` bypassing split-DNS | use `dscacheutil` | +| Task fails with **no** `live_backing.log` | the task never ran | check for a reboot mid-task | +| `add-to-group` 429s | `-j` too high | drop to `-j2`; re-run (idempotent) | +| Worker down on one probe | reboots between tasks | probe again before believing it | + +--- + +## 🔗 See also + +- [`orchestrator/README.md`](../orchestrator/README.md) — the pipeline, every command, secret handling +- [`../README.md`](../README.md) — architecture, mTLS/SCEP design, bringing it up +- [eacs.html](https://mozilla-platform-ops.github.io/relops-bootstrap/eacs.html) — what EACS + actually does, button-press → re-enroll diff --git a/orchestrator/README.md b/orchestrator/README.md index 8d9be76..67269c9 100644 --- a/orchestrator/README.md +++ b/orchestrator/README.md @@ -12,6 +12,10 @@ can't Erase-All-Content-and-Settings it again, can't escrow trust, and can't boo headless. `reprovision` automates the whole dance, including the one step everyone assumes needs a human at the keyboard: **minting the first SecureToken**. +> 🧑‍🚀 **Provisioning a wave right now? Start with the [Runbook](../docs/RUNBOOK.md).** +> Copy-paste sequences, the per-action `-j` table, and the failure modes written symptom-first — +> the 75Hz KVM trap, the two ways Safari wedges, and why `ok` doesn't mean the work happened. + --- ## The pipeline @@ -143,16 +147,32 @@ sudo bash -c 'set -a; . /var/root/reprovision-runner/runner.env; set +a; Both share the same core — **mint → escrow BST → signed-PKG bootstrap → sentinel**. They differ only at the front. -### Reprovision an existing host → back to prod *(what `run` does; proven on m4-80/m4-81)* +### 🔄 Reprovision an existing host → back to prod *(what `run` does)* Preconditions: host already in the SimpleMDM group (SCEP / CLT / `relops_key_admin` / bootstrap PKG / DEP fixed-pw), **BST escrowed**, quarantined. ```bash -reprovision run macmini-m4-80 +reprovision add-to-group macmini-m4-211 # ⚠️ REQUIRED FIRST for a prod-group host +reprovision run macmini-m4-211 # quarantine → drain → wipe (EACS) → wait-reenroll → mint → escrow-bst → wait-sentinel +reprovision validate macmini-m4-211 # ← the gate +reprovision unquarantine macmini-m4-211 ``` +> ⚠️ **`run` has no group-membership step.** It assumes the bootstrap PKG arrives "during DEP +> convergence", which only happens if the host already belongs to a group carrying that PKG — +> and **production groups do not.** Skip the `add-to-group` and the wipe succeeds, then +> `wait-sentinel` polls for a full hour for something that can never be written. `add-to-group` +> is additive and idempotent, so running it first is always safe. +> +> 🔥 **The host comes back SIP-ON.** EACS re-enables SIP regardless of its prior state, so don't +> infer post-wipe SIP from what you saw before — that's how m4-214 surprised us. +> +> 🎭 Expect the Safari automation to need a nudge on a reprovisioned host: `cltbld` is recreated +> by puppet, so it gets a fresh **MiniBuddy** first-login assistant that steals focus from the UI +> scripting. See the [Runbook](../docs/RUNBOOK.md#safari-wedge). + Stays quarantined by default. If TC creds are unavailable, skip the TC-dependent front and run from `wipe` (the box is already quarantined): @@ -174,28 +194,46 @@ reprovision provision macmini-m4-201 # preflight → mint → escrow-bst → bootstrap-pkg → wait-sentinel ``` -For a hardware refresh where a tech racks the minis and disables SIP in the same visit, -everything after that visit is remote: - -1. **Assign** the machines (by serial) to DEP/ADE and to the **intake** group — the group that - carries DEP account-setup, `relops-ssh` and the OS-install job, but *not* the bootstrap PKG. - Make sure DHCP reservations exist first: nothing in puppet sets the macOS hostname, so a - worker registers under whatever name DHCP hands it. -2. **At the rack:** power on → DEP enrolls (Setup Assistant skipped) → `csrutil disable`. -3. **MDM:** in-place update to the target OS. Stagger it — the installer download is what - saturates the link. -4. `reprovision batch hosts.txt --action preflight` — read-only; tells you which hosts actually - came out at the target OS with SIP off, *before* you commit to anything. -5. **Move the passing hosts into the bootstrap group** — membership installs the signed PKG, so - the move *is* the trigger. Keeping the PKG out of the intake group is what stops an OS - install from landing on top of a converging puppet run. -6. `reprovision batch hosts.txt --action provision -j 3 --quarantine-on-register`. - -> **If the tech can't disable SIP at the rack**, it's because Recovery authenticates against a -> **volume owner** and a DEP host with Setup Assistant skipped has none until an interactive -> login. Fix without a second trip: `reprovision batch hosts.txt --action mint` (remote, creates -> the volume owner), then Recovery. `mint` is idempotent, so `provision` re-running it later is -> a no-op — which is why it's safe to run early. +### 🌊 A whole wave, start to finish + +This is the sequence that put 37 hosts into production on 2026-08-14. Nothing here needs a +second trip to the rack, and **SIP stays enabled** — no Recovery visit required. + +```bash +HOSTS=~/Desktop/wave.txt + +reprovision batch $HOSTS --action mint -j3 # ① SecureToken (idempotent) +reprovision batch $HOSTS --action os-update -j3 # ② in-place upgrade, ~35 min +reprovision batch $HOSTS --action preflight --allow-sip-enabled -j3 # ③ read-only gate +reprovision batch $HOSTS --action add-to-group --quarantine-on-register -j2 # ④ the GO +reprovision batch $HOSTS --action validate -j3 # ⑤ fitness — do not skip +reprovision unquarantine macmini-m4-XXX # ⑥ release what passed +``` + +**Prerequisite that isn't ours:** DHCP reservations for every MAC before racking. Nothing in +puppet sets the macOS hostname — no `scutil --set` anywhere — so **DHCP decides `worker_id`**. + +**④ is the single "go" action.** Group membership delivers the bootstrap PKG, `/etc/puppet_role`, +the CLT, the admin key and passwordless sudo *all at once*, then the host provisions itself +unattended. It is not one gate among several — before it, inspecting a host tells you almost +nothing. + +**⑤ is not optional.** `validate` is the only step that catches a host which is green on every +other signal and still fails 100% of its tasks. On 2026-08-14 it found **31 of 33** hosts unfit +on a display fault that puppet, the sentinel, the worker and the disk all reported as healthy. + +> 🎚️ **Mind `-j` per action — it means something different each time.** `os-update` is +> launch-and-return, so the **hosts-file length** is the real concurrency, not `-j`. +> `add-to-group` is SimpleMDM-bound and wants **`-j2`**. Full table and the `-j12` incident: +> [Runbook → Concurrency](../docs/RUNBOOK.md#concurrency). + +> 🔑 **SecureToken arrival is inconsistent**, so always run ① rather than assuming: hosts +> 240–244 needed a manual interactive login, 245–254 arrived with tokens already granted, and of +> 255–288 only two needed minting. `mint` is idempotent, so it's free when unnecessary. +> +> If a tech *does* need Recovery (e.g. to disable SIP), note Recovery authenticates against a +> **volume owner**, and a DEP host with Setup Assistant skipped has none until an interactive +> login. `mint` creates one remotely — run it first and skip the second trip. A fresh host was never quarantined, so by default it **starts claiming work as soon as generic-worker registers.** To hold it out of the pool: @@ -218,13 +256,16 @@ Fails closed: without TC credentials it refuses **up front**, rather than spendi bootstrap window to discover it can't quarantine anything. Validate, then `reprovision unquarantine `. -### Batches +### 📦 Batches ```bash -reprovision batch hosts.txt --action preflight # read-only readiness sweep -reprovision batch hosts.txt --action mint # before the Recovery trip +reprovision batch hosts.txt --action preflight # read-only readiness sweep +reprovision batch hosts.txt --action mint # SecureToken, idempotent +reprovision batch hosts.txt --action os-update # in-place OS upgrade (launch-and-return) +reprovision batch hosts.txt --action add-to-group -j2 # the GO — SimpleMDM-bound, keep it low +reprovision batch hosts.txt --action validate # fitness check before release reprovision batch hosts.txt --action provision -j 3 \ - --quarantine-on-register # the real run + --quarantine-on-register # fresh-host all-in-one ``` `hosts.txt` is one short hostname per line, `#` comments allowed. Each host runs as its own @@ -246,6 +287,30 @@ Concurrency defaults to **3**, matching the runner's `RUNNER_MAX_CONCURRENT`: th MDC1 network and imaging throughput, not local CPU. Pushing past it is what took ~12 of 25 hosts offline simultaneously on the 2026-05-12 batch. +> 🎚️ **But `-j` is not one knob.** Each action is bound by a different resource, so the safe +> value differs — and two of them will mislead you: +> +> - **`os-update` ignores `-j` entirely.** It's launch-and-return (~10s/host), so `-j` paces only +> the launches while the ~14GB download runs detached. **The hosts-file length is the real +> concurrency.** 33 hosts means 33 simultaneous pulls no matter what you pass. +> - **`add-to-group` wants `-j2`.** It makes 3 SimpleMDM calls per host, and the API's rate +> limiter is unforgiving. At `-j12` the 429 retry budget was exhausted in ~64s. +> +> The full table, plus why `add-to-group --quarantine-on-register` couples two different +> bottlenecks to one `-j` and how to split them: +> [Runbook → Concurrency](../docs/RUNBOOK.md#concurrency). + +> 🔬 **`ok` means the command succeeded, not that the work happened.** For `os-update`, `ok` +> means *the upgrade launched* — verify with `pgrep -x curl` plus growth of +> `/private/tmp/InstallAssistant-*.zip`. For `add-to-group`, an add can succeed while the +> follow-up `push_apps` fails, so the ✗ list **understates** what changed — query group +> membership to get the truth. `validate` is the one action whose `ok` is self-verifying. + +**Credentials are resolved once in the parent** and handed to children via `REPROVISION_*`. +Before that, each child hit 1Password independently and a 10-host batch lost **9 of 10** to +`op` timeouts. Standalone (non-`batch`) commands still resolve their own, so pre-resolve into +env when looping over many hosts by hand. + --- ## Secrets: nothing touches your shell @@ -293,9 +358,11 @@ direct `REPROVISION_*` value always wins over its `_REF`. | `reprovision run ` | Full pipeline, **including an EACS wipe**. `--unquarantine` returns it to service at the end. | | `reprovision provision ` | Fresh DEP host → prod. **No wipe in this path.** `--no-wait` stops after the BST escrow. | | `reprovision preflight ` | Read-only readiness check (OS version, SIP, SecureToken, BST). Needs no SimpleMDM/TC credential. | +| 🚀 `reprovision add-to-group ` | **ADD** the host to the SimpleMDM bootstrap group — the action that triggers the whole bootstrap. Additive only, never a move. Idempotent. Refuses production groups. `--quarantine-on-register` starts the registration watch here, where it belongs. | +| ✅ `reprovision validate ` | Read-only **fitness** check on a bootstrapped host — display mode (60Hz), last puppet run, worker. **Run this before every unquarantine.** Exit 2 = not bootstrapped yet, exit 1 = bootstrapped but UNFIT. | | `reprovision quarantine-on-register ` | Watch for a fresh worker to register, then quarantine it on sight. | | `reprovision wait-bootstrap-pkg ` | Confirm the signed PKG landed — i.e. the host is in the bootstrap group. | -| `reprovision batch ` | Run `--action preflight\|mint\|provision` across a host list, `-j` at a time, one log per host. | +| `reprovision batch ` | Run `--action preflight\|mint\|os-update\|add-to-group\|validate\|provision` across a host list, `-j` at a time, one log per host. | | `reprovision quarantine ` | Quarantine in Taskcluster. | | `reprovision drain ` | Wait for the current task to finish. | | `reprovision wipe ` | EACS via SimpleMDM (`DoNotObliterate`, BST-guarded). | @@ -320,7 +387,9 @@ skipped separately from failed, which is what makes "38 ok, 15 not ready, 2 brok | `-j` / `REPROVISION_BATCH_MAX_CONCURRENT` | `3` | | `REPROVISION_PREFLIGHT_SSHD_WAIT_SECONDS` | `60` | | `REPROVISION_QUARANTINE_ON_REGISTER_POLL_SECONDS` | `5` (this interval *is* the exposure window) | -| `REPROVISION_QUARANTINE_ON_REGISTER_MAX_WAIT_SECONDS` | `900` | +| `REPROVISION_QUARANTINE_ON_REGISTER_MAX_WAIT_SECONDS` | `900` — ⚠️ sized for a watch started *after* bootstrap. Driving the watch by hand from group-add needs **5400**, or it expires before there is anything to quarantine | +| `REPROVISION_VALIDATE_EXPECTED_REFRESH_HZ` | `60.0` — matches what mozharness itself enforces, so `validate` agrees with CI rather than inventing a second standard | +| `REPROVISION_BOOTSTRAP_GROUP_ID` | `2417981` (`gecko-t-osx-1500-m4-bootstrap`). Production groups are separately blocked in `clients.simplemdm.PROTECTED_GROUP_IDS`, which this **cannot** override | | `REPROVISION_BOOTSTRAP_PKG_MAX_WAIT_SECONDS` | `300` — how long to wait for the PKG before calling it a group problem | ### Wrong-group detection