Emit a real CycloneDX SBOM on macOS workers, on every role - #1327
Open
rcurranmoz wants to merge 1 commit into
Open
Emit a real CycloneDX SBOM on macOS workers, on every role#1327rcurranmoz wants to merge 1 commit into
rcurranmoz wants to merge 1 commit into
Conversation
The macos_sbom module existed but could not do its job:
* The exec was refreshonly and subscribed to its own script file, so the
SBOM only regenerated when the generator was edited. It was a snapshot
of the last time someone touched the script, not of the host.
* It was wired into one role out of 27 (gecko_t_osx_1015_r8_staging), so
nothing on r8 prod, 1400, M4 or the VM images produced one at all.
* It wrote Markdown, which no scanner or SBOM tool can consume. No purls,
no hashes, no licenses.
* It spent 30-60s per run on `system_profiler SPApplicationsDataType` to
collect app bundles, the least interesting part of a CI worker.
* `enabled` was declared and never used, and the log was appended to
forever.
Rewritten to emit CycloneDX 1.6 JSON, validated against the upstream
schema, and included from all 27 macOS roles.
Inventory now comes from /var/db/receipts/*.plist read directly with
plistlib (the real installer receipt database, versions and install dates,
no per-package pkgutil fork), plus pip for each interpreter present, Xcode,
Homebrew if any survived uninstall_homebrew, and /usr/local/bin hashed with
its code-signing authority so the Developer ID worker binaries are visible.
Two outputs in /var/sbom, both world-readable so a collector can scrape
them over SSH without root:
sbom.cdx.json full document, rewritten only when the inventory
actually changes, so its mtime is the drift timestamp
fingerprint.json small summary with a stable sbom_sha256
The fingerprint is the point. Every host is Puppet-managed from a role and
every VM image is immutable, so the fleet holds a handful of distinct
software states rather than one document per host. Hosts collapse by hash,
and a host whose hash differs from its role's usual hash has drifted.
Safety, since this now runs fleet-wide including on signing hosts:
* The generator never exits non-zero and each collector is individually
guarded. An SBOM is not worth failing a Puppet run over.
* A run_sbom.sh wrapper probes for a working python3 and falls back to the
CLT interpreter, because build and signing roles don't install
packages::python3. Adding it to them would have been a bigger change
than this one.
* Only generic-worker and taskcluster are asked for a version, with the
argv each actually accepts, and the answer is sanity-checked before it
is believed. livelog and taskcluster-proxy ignore --version and start
serving; probing livelog hung for the full timeout in testing.
* Regeneration is gated to once per macos_sbom::refresh_minutes (default
daily) so Puppet doesn't report a change on every run. Steady-state cost
measured at 0.6s.
Verified on macOS 26.6.1/arm64: 663 components, valid against the official
CycloneDX 1.6 JSON schema, byte-identical across consecutive runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Member
|
Nice. Yeah, I think we should move the cloud ones to CycloneDX 1.6 JSON format (or also generate that format). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
macos_sbomhas been in the tree since #705 but could not actually do its job:refreshonly => truesubscribed to its own script file, so the SBOM regenerated only when someone edited the generator — not when the host's software changed.gecko_t_osx_1015_r8_staging). Nothing on r8 prod, 1400, M4, or the VM image roles produced one.grype.system_profiler SPApplicationsDataType, which reports app bundles — the least interesting part of a CI worker's surface.enabledwas declared and never used; the log grew without bound.What changed
CycloneDX 1.6 JSON, validated against the upstream schema, included from all 27 macOS roles.
Inventory sources:
/var/db/receipts/*.plistviaplistlibpkgutilper packagepip list, per interpreter present/usr/local/bin, SHA-256 + codesign authorityuninstall_homebrewrather than assuming itTwo outputs in
/var/sbom, both0644so a collector can scrape them over SSH without root:sbom.cdx.json— full document, rewritten only when the inventory changes, so its mtime is the drift timestampfingerprint.json— small summary carrying a stablesbom_sha256The fingerprint is the point
Every host is Puppet-managed from a role and every VM image is immutable, so the fleet holds a handful of distinct software states, not one document per host. Hosts collapse by hash — and a host whose hash differs from its role's usual hash has drifted, which is a signal we don't currently have anywhere.
The hash covers component identity only (type, name, version, purl, SHA-256). Timestamps, install dates and the serial number are excluded, so two hosts built from the same role produce the same hash and a same-version reinstall doesn't churn it. The
serialNumberis derived from the content hash, so it's stable across runs too.Safety, since this now runs fleet-wide including on signing hosts
run_sbom.shprobes for a working python3 and falls back to the CLT interpreter. 11 of the 27 macOS roles (build + signing) don't installpackages::python3, and adding it to them would have been a bigger and riskier change than this one./usr/bin/python3is probed rather than assumed, since on a host without the CLT it's a stub.generic-workerandtaskclusterare asked for a version, using the argv each actually accepts, and the answer is sanity-checked before it's believed.livelogandtaskcluster-proxyignore--versionand start serving — probingliveloghung for the full timeout during testing, which is exactly the failure this avoids on a live worker.macos_sbom::refresh_minutes(default 1440) so Puppet doesn't report a resource change on every run. A generator update invalidates the fingerprint and forces a regeneration that run.Verification
Ran on macOS 26.6.1 / arm64:
bom-1.6.schema.jsonupstream)inventory unchangedand left the document alonepre-commit runclean (puppet-validate, puppet-lint, shellcheck)Not yet exercised on a Catalina or Mojave host, or inside a packer VM build — those get the profile here and should be watched on first apply. On the VM roles this runs during phase-2 puppet, so the image ships with an SBOM of itself.
Follow-up (not in this PR)
Hangar ingest: a sync that reads
fingerprint.jsonover SSH and stores documents deduped by hash, following thesync/windows_inventory.py+SyncLogpattern. Deliberately left out so the on-disk output can be eyeballed on real hosts before a schema is committed to.Worth noting for whoever asked for the SBOM: this describes the provisioned image. Toolchains mozharness fetches per-task are ephemeral and are not covered.
🤖 Generated with Claude Code