ci(bootstrap): prune four macOS-only failures from the ratchet baseline - #2388
Merged
Conversation
The first test-ratchet run on master reported all four spec_first_* tests under 'now PASS' -- they fail on macOS/arm64 where the baseline was taken and pass on ubuntu-latest. A CI-passing test left in the baseline is invisible if it breaks: the ratchet fails only on names new relative to the baseline. The baseline must describe the platform that gates. Closes #2387
Contributor
PR DashboardGenerated at: 2026-08-22 08:24:44 UTC
Summary
Seal Status
|
Contributor
|
📓 NotebookLM Notebook linked to this PR
This notebook contains session context, decisions, and artifacts for this work. |
gHashTag
enabled auto-merge (squash)
August 22, 2026 08:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2387
What
Four tests are in
scripts/ci/test-baseline.txtas failing, and the Linux runner passesthem. From the first
test-ratchetrun on master (e109851b):They fail on macOS/arm64, where the baseline was generated, and pass on
ubuntu-latest.Why it matters
A CI-passing test left in the baseline is invisible if it breaks. The ratchet fails only
on names that are new relative to the baseline, so these four could start failing on Linux
tomorrow and the gate would stay green — the exact blind spot the ratchet exists to remove,
reintroduced in its own input.
The general rule this fixes: the baseline must describe the platform that gates. A
baseline taken somewhere else describes a different suite.
What this does not establish
path separators and filesystem ordering are all plausible and none is confirmed. The four
are
spec_first_*inference-vs-reference comparisons, which suggests a numeric or orderingdifference rather than a build problem, but that is a guess and is labelled as one.
failures the ratchet does not know about. That is the correct trade — CI is what gates —
but it is a trade, not a free win, and it is worth a note in the file, which this adds.
Related: #2382, #2383, #2386.