Skip to content

Close evidence-guard gaps: wire npm packaging tests into dist-validation, correct stale OSTW oracle CI claims - #203

Merged
Teakowa merged 3 commits into
mainfrom
audit/evidence-guard-gaps
Aug 22, 2026
Merged

Close evidence-guard gaps: wire npm packaging tests into dist-validation, correct stale OSTW oracle CI claims#203
Teakowa merged 3 commits into
mainfrom
audit/evidence-guard-gaps

Conversation

@Teakowa

@Teakowa Teakowa commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Entropy-audit follow-up (Audit candidate 2): two declared evidence guards did not match reality.

  • Wire the npm packaging unit regressions into dist-validation. chore(compat): remove duplicated language-oracle infrastructure #183 moved the [Distribution correctness] Make npm package-mode validation portable on Windows #123 npm tarball determinism/executable-bit regressions from compatibility/tests to scripts/tests and stated they would "run through distribution validation", but no CI step ever executed them — the suite was dead weight since 2026-08-19. This adds the missing unittest discover step to the dist-validation matrix (3 OSes), which already sets up Python 3.12 and Node.js 22 and runs the adjacent verify-dist.py / test-npm.py steps.
  • Correct the stale OSTW oracle CI-guard claims. ci: remove legacy OSTW reference baseline gate #177 deliberately removed the ostw-reference CI job (upstream .NET oracle replay is evidence infrastructure owned by del-rs, not a Wright merge gate), but docs/ostw/compatibility-baseline.md and compatibility/ostw/README.md still claimed the run_oracle.py drift check "guards CI". The docs now state the reality: the drift check is a manual maintainer command, future oracle reproducibility workflows belong to del-rs (del-rs#49, tracked in Wright by Track removal of residual OSTW owner-style evidence after deltin-rs migration #182), and Wright CI consumes the recorded evidence via the [M13] Compile the accepted OSTW corpus slice to Workshop with reference differential validation #119 compile differential without re-deriving it.
  • Fix a portability defect the unwired suite was hiding. On its first-ever Windows execution, test_verify_tarball_still_rejects_non_executable_bin failed on a literal mode 0o644 assertion: Windows-staged exec-less files carry tar mode 0o666, and the production verify_tarball rejection is correct on both platforms. The assertion now checks the stable observable contract (rejection names the offending bin file and reports the observed mode) instead of Unix-specific mode digits.

Why not re-wire run_oracle.py into CI instead

PR #177 made an explicit ownership decision: Wright PRs must not acquire and execute the pinned upstream .NET OSTW runtime, and oracle reproducibility workflows should be owned by del-rs as maintainer/evidence infrastructure. Re-adding the gate would reverse that decision; the remaining problem was only the stale doc claims, which this PR fixes.

Evidence / validation

All results below come from CI run on this PR (run 32558927508 + the re-run for the last commit) and local branch runs:

  • python3 -m unittest discover -s scripts/testsRan 7 tests, OK (local, both before and after the assertion fix)
  • First CI round: dist-validation windows-latest failed on the mode 0o644 literal — the defect the unwired suite had been hiding; ubuntu/macos passed. After the fix, all three OSes pass.
  • python3 -m unittest discover -s compatibility/testsRan 2 tests, OK (untouched)
  • python3 -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))" — parses
  • git diff --check — clean
  • grep -rn "guards CI|CI drift guard" --include="*.md" — no remaining stale claims

Testing-policy notes

  • No expected result, fixture, or support classification changed; no guard was weakened. The doc edit weakens only a false claim.
  • The one changed test expectation (0o644 → platform-agnostic mode assertion) does not change the protected contract — verify_tarball must reject non-executable bin entries — which remains asserted; the removed literal encoded a Unix-specific incidental, evidenced by the Windows CI run observing 0o666 with a correct rejection.
  • The CI addition strictly adds coverage: a previously never-executed regression suite now gates distribution validation on all three OSes.

Known gaps that remain visible (not hidden)

#183 moved the npm tarball determinism/executable-bit regressions (#123)
from compatibility/tests to scripts/tests and declared them part of
distribution validation, but no CI step ever executed them. Add the
unittest discover step to the dist-validation matrix (Python and Node are
already set up there, and the suite exercises npm pack end-to-end when
available). Verified locally: 7 tests pass in ~0.6s.
PR #177 deliberately removed the ostw-reference CI job (upstream .NET
oracle replay is evidence infrastructure owned by del-rs, not a Wright
merge gate), but two documents still claimed the run_oracle.py drift
check guards CI. State the reality: the drift check is a manual
maintainer command, oracle reproducibility workflows belong to del-rs
(del-rs#49, tracked in Wright by #182), and Wright CI consumes the
recorded evidence via the #119 differential without re-deriving it.
The #123 regression suite had never executed on Windows (it was unwired
until the previous commit). CI on windows-latest showed the staged
exec-less bin files carry tar mode 0o666 there instead of POSIX 0o644,
so the literal 'mode 0o644' assertion failed while the production
verify_tarball rejection worked correctly. Assert the stable observable
contract instead: the rejection names the offending bin file and reports
the observed mode.
@Teakowa
Teakowa merged commit 68298cc into main Aug 22, 2026
11 checks passed
@Teakowa
Teakowa deleted the audit/evidence-guard-gaps branch August 22, 2026 07:27
@github-project-automation github-project-automation Bot moved this from Backlog to Done in Make Workshop Great Again Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

1 participant