group-parity, and split the two resources add-to-group was straddling

[rcurranmoz]

Two changes from reviewing the m4 refresh path before the next wave. Both are about the same failure shape: a wave that looks fine right up until a host is already in production.

1. reprovision group-parity

Answers a question nothing in the toolchain could: do the hosts in a group get the profiles a working production host gets? Read-only, API-only, no SSH unless you pass --host.

The m4-214 incident turned on this — a host missing Skip Setup Assistant and the FDA SSH Keygen Wrapper hung at first boot and presented as "Safari automation is broken", costing most of a day. The postmortem's advice was to diff profiles show -type configuration by hand against a known-good host, which needs SSH to a box that by definition may not be reachable.

Two things verified against the live account shaped the implementation:

• Assignment-group records carry no profiles relationship at all (the keys are apps / device_groups / devices / media). The link lives on the profile side as relationships.groups.
• The comparison has to be per-device for correctness, not convenience. Diffing assignment groups reports Skip Setup Assistant and the FDA wrapper as missing from the bootstrap group — true, and irrelevant: its devices get both from the additive DEP Enrollment group. Crying wolf on exactly the pair from the postmortem would train operators to ignore the check. There is a test pinning this against a well-meaning simplification.

The baseline is the intersection of several sampled reference devices, not one sampled host, so an atypical prod box can't drag a profile in and the result doesn't depend on API listing order.

A profile diff alone under-reports, so there's a second peer-wise membership pass: a group that two thirds of the devices are in but some are not means those were moved rather than added.

What it found on first run

• TCC - CI Worker Support Binaries (PPPC) missing on 38/40 bootstrap-group devices — attached to prod 2017918 but not to 2417981, on a SIP-ON wave where a profile is the only way system-level TCC grants land. Since fixed.
• Three hosts missing Relops Public SSH Key, Sudoers, Enable SSH and DEP Enrollment — no admin key, no passwordless sudo, no sshd after their next wipe. Two of them (m4-211, m4-215) are also in the prod group, so they still receive the profiles and the profile diff said nothing about them. One of those two is staged for a wipe.

Devices are labelled with serial and name, because a bare device id is not identifying — a matching enrollment date was enough for me to mistake one for m4-214.

2. Split the two resources add-to-group --quarantine-on-register was straddling

It coupled an action bound by the SimpleMDM API (three calls per host, no SSH to pace it) to a ~30-minute Taskcluster-bound wait, with one --concurrency serving both. Raise it for wall-clock and you hammer SimpleMDM; lower it for SimpleMDM and 33 hosts serialise into 5½ hours.

At -j12 on 2026-08-14 this looked like five hosts failing when twelve had already been added — the POST succeeded and the follow-up push_apps 429'd — so killing the batch orphaned twelve live, autonomous bootstraps with nothing watching them. They would have reached production unvalidated. The runbook has carried a two-command workaround ever since; this is that workaround, built in.

• The add phase is clamped to simplemdm_max_concurrent whatever -j says, and warns rather than silently obeying.
• The watch phase runs every host at once — a watcher is an idle poll loop.
• A host whose add FAILED is still watched, since that is precisely the case where it's in the group anyway. Only skipped (exit 2) means it was definitely never added.
• The bootstrap-spanning budget is passed as --max-wait-seconds, so nobody needs to know to export a 5400s override. A budget that expires early puts the host live unheld — the failure the flag exists to prevent.
• Both phases log under one batch directory, and Ctrl-C writes added.txt naming every host bootstrapping unwatched, with the command to re-attach.
• quarantine-on-register is a first-class batch action now, replacing the xargs -P 33 nohup workaround.

Testing

• 248 tests pass, ruff clean.
• 20 new tests for parity, 9 for the phase split.
• group-parity exercised live against the real account (read-only), both group-wide and --host macmini-m4-111.
• The phase split verified via --dry-run at -j12, confirming the clamp, the two phases and the explicit budget.

Runbook and orchestrator README updated; the old manual -j workaround is preserved in a <details> block rather than deleted.

🤖 Generated with Claude Code