Skip to content

ci(bootstrap): prune four macOS-only failures from the ratchet baseline - #2388

Merged
gHashTag merged 1 commit into
masterfrom
fix/2387-prune-platform-baseline
Aug 22, 2026
Merged

ci(bootstrap): prune four macOS-only failures from the ratchet baseline#2388
gHashTag merged 1 commit into
masterfrom
fix/2387-prune-platform-baseline

Conversation

@gHashTag

@gHashTag gHashTag commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Closes #2387

What

Four tests are in scripts/ci/test-baseline.txt as failing, and the Linux runner passes
them. From the first test-ratchet run on master (e109851b):

73 targets ran; 379 failing test(s) (baseline 383)
4 baselined test(s) now PASS — prune them from the baseline:
  - spec_first_layer2_packs_two_neuron_trits            (tests/bitnet_layer.rs)
  - spec_first_mlp2_two_layer_inference_matches_reference (tests/bitnet_mlp.rs)
  - spec_first_mlp3_three_layer_inference_matches_reference (tests/bitnet_mlp3.rs)
  - spec_first_neuron_n_accumulates_and_quantizes       (tests/bitnet_neuron_nchunk.rs)

They fail on macOS/arm64, where the baseline was generated, and pass on ubuntu-latest.

Why it matters

A CI-passing test left in the baseline is invisible if it breaks. The ratchet fails only
on names that are new relative to the baseline, so these four could start failing on Linux
tomorrow and the gate would stay green — the exact blind spot the ratchet exists to remove,
reintroduced in its own input.

The general rule this fixes: the baseline must describe the platform that gates. A
baseline taken somewhere else describes a different suite.

What this does not establish

  • Why they differ across platforms was not investigated. Endianness, float formatting,
    path separators and filesystem ordering are all plausible and none is confirmed. The four
    are spec_first_* inference-vs-reference comparisons, which suggests a numeric or ordering
    difference rather than a build problem, but that is a guess and is labelled as one.
  • Pruning them means a macOS developer running the suite locally will now see four
    failures the ratchet does not know about. That is the correct trade — CI is what gates —
    but it is a trade, not a free win, and it is worth a note in the file, which this adds.
  • It does not fix the four. If they should pass on macOS too, that is separate work.

Related: #2382, #2383, #2386.

The first test-ratchet run on master reported all four spec_first_* tests
under 'now PASS' -- they fail on macOS/arm64 where the baseline was taken
and pass on ubuntu-latest.

A CI-passing test left in the baseline is invisible if it breaks: the
ratchet fails only on names new relative to the baseline. The baseline
must describe the platform that gates.

Closes #2387
@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-22 08:24:44 UTC

Summary

Status Count
Total Open PRs 3
PRs with Failing Checks 2
PRs with All Checks Green 1
READY 1
FAILING 2
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=fba627661c36 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

@gHashTag
gHashTag enabled auto-merge (squash) August 22, 2026 08:25
@gHashTag
gHashTag merged commit 03c8908 into master Aug 22, 2026
24 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Four tests that CI passes are baselined as failing, so a real Linux regression in them would be invisible

1 participant