Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions docs/now/2026-08-22-prune-platform-baseline.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# NOW — a CI-passing test sat in the failing baseline

Last updated: 2026-08-22

## Prune four macOS-only failures from the ratchet baseline (Closes #2387)

- Branch: `fix/2387-prune-platform-baseline`
- Issue: #2387

### Что легло

`scripts/ci/test-baseline.txt`, 382 → 378 entries. The four `spec_first_*` tests in
`bitnet_layer`, `bitnet_mlp`, `bitnet_mlp3` and `bitnet_neuron_nchunk` fail on macOS/arm64,
where the baseline was generated, and **pass on the Linux runner** — the first
`test-ratchet` run on master reported all four under "now PASS".

A CI-passing test left in the baseline is invisible if it breaks: the ratchet fails only on
names new relative to the baseline, so these four could go red on Linux and the gate would
stay green. That is the blind spot the ratchet exists to remove, reintroduced in its own
input. **The baseline must describe the platform that gates.**

### Границы честности (BINDING)

- **The cause of the platform divergence was not investigated.** Endianness, float
formatting, path separators and filesystem ordering are all plausible; none is confirmed.
That they are all `spec_first_*` inference-vs-reference comparisons hints at a numeric or
ordering difference — a hint, not a finding.
- Pruning means a **macOS** developer now sees four local failures the ratchet does not know
about. Correct trade — CI is what gates — but a trade, noted in the file itself.
- **This fixes nothing.** The four still fail on macOS.
- The baseline is still generated on macOS. Regenerating it from a runner log would be
better and is not done here.
8 changes: 3 additions & 5 deletions scripts/ci/test-baseline.txt
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# Generated by scripts/ci/test_ratchet.py --emit-baseline.
# Do not hand-edit: regenerate from a real --no-fail-fast log.
# Each line is target<TAB>test for a test failing on master.
# 73 targets ran; 382 failing tests (one pruned by #2384).
# 73 targets ran; 378 failing tests on the Linux runner.
# Four macOS-only failures pruned by #2387: the baseline describes the
# platform that gates, and a CI-passing test left in it is invisible if it breaks.
src/main.rs compiler::tests_compiler_rejects::lowers_only_first_of_two_modules_characterization
src/main.rs compiler::tests_phase40_coverage::test_for_range_loop_unroll
src/main.rs compiler::tests_phase40_coverage::test_parse_for_range
Expand All @@ -15,10 +17,6 @@ src/main.rs compiler::tests_w458::tests_w459::array_param_bound_from_test_block
src/main.rs compiler::tests_w458::tests_w459::test_block_emits_real_function_call
src/main.rs lex_conform::tests::lexer_matches_its_conformance_table
src/main.rs parse_conform::tests::parser_matches_its_conformance_table
tests/bitnet_layer.rs spec_first_layer2_packs_two_neuron_trits
tests/bitnet_mlp.rs spec_first_mlp2_two_layer_inference_matches_reference
tests/bitnet_mlp3.rs spec_first_mlp3_three_layer_inference_matches_reference
tests/bitnet_neuron_nchunk.rs spec_first_neuron_n_accumulates_and_quantizes
tests/icarus_lowerable.rs accepts_known_lowerable_witnesses
tests/icarus_lowerable.rs accepts_w545_primitive_scalar_array_return
tests/icarus_lowerable.rs accepts_w547_signed_primitive_scalar_array_return
Expand Down
Loading