mozilla-platform-ops · rcurranmoz · Aug 19, 2026 · Aug 19, 2026
diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md
@@ -16,12 +16,13 @@ because that's the order you meet them in.
 cd ~/git/relops-bootstrap/orchestrator
 HOSTS=~/Desktop/wave.txt          # one short hostname per line, '#' comments ok
 
+uv run reprovision group-parity                             # ← do the hosts get prod's profiles?
 uv run reprovision batch $HOSTS --action mint         -j3   # SecureToken (idempotent, often a no-op)
 uv run reprovision batch $HOSTS --action os-update    -j3   # in-place upgrade to target OS
 #   ⏳ wait ~35 min: ~14GB download, then startosinstall, then a reboot
 uv run reprovision batch $HOSTS --action preflight --allow-sip-enabled -j3
-uv run reprovision batch $HOSTS --action add-to-group --quarantine-on-register -j2
-#   ⏳ blocks ~30 min per host: this is the bootstrap, and the watch that holds each host
+uv run reprovision batch $HOSTS --action add-to-group --quarantine-on-register
+#   ⏳ two phases, automatically: a paced SimpleMDM add, then every watcher at once (~30 min)
 uv run reprovision batch $HOSTS --action validate     -j3   # ← the gate. Do not skip.
 uv run reprovision unquarantine macmini-m4-XXX              # only what validate passed
 ```
@@ -55,24 +56,48 @@ bound by a different resource.
 | `mint` | 3–4 | SSH | fast; often a no-op when the token already exists |
 | `os-update` | **irrelevant** ⚠️ | the mirror | **launch-and-return** — `-j` paces only the *launches*. The **hosts-file length is the real concurrency.** 33 hosts = 33 simultaneous 14GB pulls |
 | `preflight` | 3–4 | SSH | read-only |
-| `add-to-group` | **2** 🚨 | **SimpleMDM API** | 3 API calls per host. At `-j12` the 429 retry budget blew and hosts failed |
+| `add-to-group` | **2** 🚨 | **SimpleMDM API** | 3 API calls per host. At `-j12` the 429 retry budget blew and hosts failed. Clamped to `REPROVISION_SIMPLEMDM_MAX_CONCURRENT` (2) regardless of `-j` |
+| `quarantine-on-register` | all of them | local process count | an idle poll loop; `-j` does not apply |
 | `validate` | 3 | SSH | read-only |
 | `provision` | 3 | MDC1 imaging throughput | matches the runner's `RUNNER_MAX_CONCURRENT` |
 
 <a id="j-trap"></a>
 
-### 🪤 The `-j` trap that bit us hardest
+### 🪤 The `-j` trap that bit us hardest — now fixed
 
-`add-to-group --quarantine-on-register` couples a **SimpleMDM-bound action** to a
-**30-minute Taskcluster-bound wait**, so one `-j` has to serve both. Raise it for wall-clock
-and you hammer SimpleMDM; lower it for SimpleMDM and 33 hosts serialize into ~5½ hours.
+`add-to-group --quarantine-on-register` used to couple a **SimpleMDM-bound action** to a
+**30-minute Taskcluster-bound wait**, so one `-j` had to serve both. Raise it for wall-clock and
+you hammered SimpleMDM; lower it for SimpleMDM and 33 hosts serialized into ~5½ hours.
+
+**It now splits itself into two phases**, so the command below is all you run:
+
+```bash
+uv run reprovision batch $HOSTS --action add-to-group --quarantine-on-register
+```
+
+Phase 1 adds hosts at the SimpleMDM cap (2), clamping `-j` with a warning if you asked for more.
+Phase 2 then watches every host at once, with the bootstrap-spanning budget passed explicitly —
+no `REPROVISION_QUARANTINE_ON_REGISTER_MAX_WAIT_SECONDS` export needed. Both phases log under one
+batch directory, and a `Ctrl-C` writes `added.txt` naming every host that is bootstrapping
+unwatched, plus the command to re-attach:
+
+```bash
+uv run reprovision batch ~/.local/state/reprovision/batch-<stamp>/added.txt \
+  --action quarantine-on-register
+```
+
+A host whose add *failed* is still watched — the add can succeed while the follow-up `push_apps`
+429s, which is exactly what happened on 2026-08-14.
+
+<details>
+<summary>The old manual workaround, for reference</summary>
 
 At `-j12` on 2026-08-14 it looked like 5 hosts failed. What actually happened: **12 hosts had
 already been added to the group** (the add succeeded, the follow-up `push_apps` 429'd), so
 killing the batch orphaned 12 live bootstraps with no watcher. They'd have gone into
 production unvalidated.
 
-**Workaround until this is fixed** — split it along the resource boundary:
+Split it along the resource boundary by hand:
 
 ```bash
 # 1. the SimpleMDM half, gently
@@ -90,6 +115,8 @@ grep '^macmini' $HOSTS | xargs -P 33 -I{} \
 > (~30 min). Too short and the watch expires before there's anything to quarantine — and the
 > host goes live unheld, i.e. exactly the failure the flag exists to prevent.
 
+</details>
+
 ---
 
 ## 🔬 Never trust `ok`. Verify the thing itself.
@@ -259,6 +286,30 @@ looping over many hosts by hand.
 - ⚠️ **Membership does not prove the PKG was pushed.** The already-a-member path skips
   `push_apps`, so a host added by hand in the UI can sit in the group with nothing installed.
   `add-to-group` warns when the payload is missing.
+- 🧬 **Check profile parity before a wave**, not after the first host hangs:
+
+  ```bash
+  uv run reprovision group-parity                          # the whole bootstrap group
+  uv run reprovision group-parity --host macmini-m4-241    # one host
+  ```
+
+  Read-only, API-only, no SSH unless you pass `--host`. It builds a baseline from the profiles
+  *every* sampled production device shares, then names anything a target device lacks — with the
+  m4-214 story attached for the profiles that caused it.
+
+  It also runs a second, **peer-wise membership pass**: a group that two thirds of these devices
+  are in but some are not means those devices were *moved* rather than added. This catches strictly
+  more than the profile diff, because the groups a mis-clicked host loses are mostly app-bearing —
+  on 2026-08-19 three hosts were missing `Relops Public SSH Key`, `Sudoers`, `Enable SSH` and
+  `DEP Enrollment`, i.e. no admin key, no passwordless sudo and no sshd after their next wipe. Two
+  of them were also in the prod group, so they still received the profiles and a profile-only diff
+  said nothing.
+
+  It compares **effective per-device** profile sets, deliberately. Diffing the assignment groups
+  themselves reports Skip Setup Assistant and the FDA SSH Keygen Wrapper as missing from the
+  bootstrap group — true, and irrelevant: its devices get both from the additive DEP Enrollment
+  group. Don't "simplify" it back to a group-level diff; it would cry wolf on the exact pair from
+  the postmortem.
 
 ---
 

@@ -202,10 +202,11 @@ second trip to the rack, and **SIP stays enabled** — no Recovery visit require
 ```bash
 HOSTS=~/Desktop/wave.txt
 
+reprovision group-parity                                 # ⓪ profile parity, before anything
 reprovision batch $HOSTS --action mint      -j3          # ① SecureToken (idempotent)
 reprovision batch $HOSTS --action os-update  -j3          # ② in-place upgrade, ~35 min
 reprovision batch $HOSTS --action preflight --allow-sip-enabled -j3   # ③ read-only gate
-reprovision batch $HOSTS --action add-to-group --quarantine-on-register -j2   # ④ the GO
+reprovision batch $HOSTS --action add-to-group --quarantine-on-register   # ④ the GO (2 phases)
 reprovision batch $HOSTS --action validate  -j3          # ⑤ fitness — do not skip
 reprovision unquarantine macmini-m4-XXX                  # ⑥ release what passed
 ```
@@ -293,12 +294,14 @@ hosts offline simultaneously on the 2026-05-12 batch.
 > - **`os-update` ignores `-j` entirely.** It's launch-and-return (~10s/host), so `-j` paces only
 >   the launches while the ~14GB download runs detached. **The hosts-file length is the real
 >   concurrency.** 33 hosts means 33 simultaneous pulls no matter what you pass.
-> - **`add-to-group` wants `-j2`.** It makes 3 SimpleMDM calls per host, and the API's rate
->   limiter is unforgiving. At `-j12` the 429 retry budget was exhausted in ~64s.
+> - **`add-to-group` is capped at 2 for you.** It makes 3 SimpleMDM calls per host and the API's
+>   rate limiter is unforgiving — at `-j12` the 429 retry budget was exhausted in ~64s. A larger
+>   `-j` is clamped, with a warning.
+> - **`add-to-group --quarantine-on-register` runs as two phases**, because a SimpleMDM-bound add
+>   and a 30-minute Taskcluster-bound watch cannot share one `-j`. The add is paced; the watchers
+>   all start at once.
 >
-> The full table, plus why `add-to-group --quarantine-on-register` couples two different
-> bottlenecks to one `-j` and how to split them:
-> [Runbook → Concurrency](../docs/RUNBOOK.md#concurrency).
+> The full table: [Runbook → Concurrency](../docs/RUNBOOK.md#concurrency).
 
 > 🔬 **`ok` means the command succeeded, not that the work happened.** For `os-update`, `ok`
 > means *the upgrade launched* — verify with `pgrep -x curl` plus growth of
@@ -358,6 +361,7 @@ direct `REPROVISION_*` value always wins over its `_REF`.
 | `reprovision run <host>` | Full pipeline, **including an EACS wipe**. `--unquarantine` returns it to service at the end. |
 | `reprovision provision <host>` | Fresh DEP host → prod. **No wipe in this path.** `--no-wait` stops after the BST escrow. |
 | `reprovision preflight <host>` | Read-only readiness check (OS version, SIP, SecureToken, BST). Needs no SimpleMDM/TC credential. |
+| 🧬 `reprovision group-parity` | Read-only: do a group's hosts get the profiles a working production host gets? Builds a baseline from the profiles every sampled prod device shares and names what a target lacks. Compares **effective per-device** sets, not assignment groups — the bootstrap group carries neither m4-214 profile yet its devices hold both, via the additive DEP Enrollment group. Also flags **membership outliers** — a device missing a group two thirds of its peers are in was moved, not added, and the groups it lost are mostly app-bearing (admin key, sudo, sshd), which a profile diff cannot see. `--host` checks one box; needs only the SimpleMDM API key otherwise. |
 | 🚀 `reprovision add-to-group <host>` | **ADD** the host to the SimpleMDM bootstrap group — the action that triggers the whole bootstrap. Additive only, never a move. Idempotent. Refuses production groups. `--quarantine-on-register` starts the registration watch here, where it belongs. |
 | ✅ `reprovision validate <host>` | Read-only **fitness** check on a bootstrapped host — display mode (60Hz), last puppet run, worker. **Run this before every unquarantine.** Exit 2 = not bootstrapped yet, exit 1 = bootstrapped but UNFIT. |
 | `reprovision quarantine-on-register <host>` | Watch for a fresh worker to register, then quarantine it on sight. |
@@ -387,7 +391,9 @@ skipped separately from failed, which is what makes "38 ok, 15 not ready, 2 brok
 | `-j` / `REPROVISION_BATCH_MAX_CONCURRENT` | `3` |
 | `REPROVISION_PREFLIGHT_SSHD_WAIT_SECONDS` | `60` |
 | `REPROVISION_QUARANTINE_ON_REGISTER_POLL_SECONDS` | `5` (this interval *is* the exposure window) |
-| `REPROVISION_QUARANTINE_ON_REGISTER_MAX_WAIT_SECONDS` | `900` — ⚠️ sized for a watch started *after* bootstrap. Driving the watch by hand from group-add needs **5400**, or it expires before there is anything to quarantine |
+| `REPROVISION_QUARANTINE_ON_REGISTER_MAX_WAIT_SECONDS` | `900` — sized for a watch started *after* bootstrap. Every path that starts the watch earlier now passes a bootstrap-spanning budget explicitly (`--max-wait-seconds`), so you should not need to export this |
+| `REPROVISION_SIMPLEMDM_MAX_CONCURRENT` | `2` — fan-out cap for SimpleMDM-bound batch work, independent of `-j` |
+| `REPROVISION_REFERENCE_GROUP_ID` | `2017918` — the production group `group-parity` measures against (read-only) |
 | `REPROVISION_VALIDATE_EXPECTED_REFRESH_HZ` | `60.0` — matches what mozharness itself enforces, so `validate` agrees with CI rather than inventing a second standard |
 | `REPROVISION_BOOTSTRAP_GROUP_ID` | `2417981` (`gecko-t-osx-1500-m4-bootstrap`). Production groups are separately blocked in `clients.simplemdm.PROTECTED_GROUP_IDS`, which this **cannot** override |
 | `REPROVISION_BOOTSTRAP_PKG_MAX_WAIT_SECONDS` | `300` — how long to wait for the PKG before calling it a group problem |