Skip to content

Re-pin the CI verification contract to the shard-matrix gate - #26

Closed
Parad0x-Labs wants to merge 1 commit into
mainfrom
mission/ci-contract-repin
Closed

Parad0x-Labs wants to merge 1 commit into
mainfrom
mission/ci-contract-repin

Conversation

@Parad0x-Labs

Copy link
Copy Markdown
Owner

What failed

tests/test_ci_verification_contract.py — 4 of 6 cases red in the main CI census (run
35570948370) and every PR run since; deterministic locally:

  • test_push_and_pr_ci_use_the_exact_authoritative_gate — demands a verify-job step named
    "Run authoritative verification" running ops/verify.py
  • test_contract_mutations_are_rejected[collect_only], [step_disabled] — the sabotage
    mutations crash mutating that same nonexistent step
  • test_verification_dependencies_are_exactly_pinned — asserts ruff==0.15.16; the tree
    adopted ruff==0.16.7 with dependabot PR Bump ruff from 0.15.16 to 0.16.7 #5 (and tests/test_install_surface_contracts.py
    already pins 0.16.7 — the two contract files had drifted apart)

Root cause

The suite pins the pre-migration single-job gate. The workflow it describes — one verify
job running ops/verify.py --workers 4 … over the whole suite — was replaced by the owner's
shard architecture (commits 9fd7ee7, 006d874, ca9eb4d, plus merged PR #11's
hidden-file artifact uploads): a verify job that lints with the pinned ruff and produces the
canonical pytest collection, a ten-shard Linux matrix where every collected file runs in
exactly one shard, a routed macOS job for the macOS-only suites, and build gated on all
three. The suite was never re-pinned; standing failure since those workflow changes.

Repair (test-only; the workflow is untouched — it is what main authorizes)

The weakening law the old suite enforced, restated against the current structure:

  • push + pull_request triggers, no paths-ignore; no if on the verify/tests/macos jobs;
    exactly one setup-python per job pinned to 3.12.13
  • verify: exact python -m ruff check . and exact
    python ops/pytest_manifest.py … -- -q commands, no step-level if
  • tests: needs: verify, the full shard: ["0"…"9"] matrix (a dropped shard is a dropped
    tenth of the suite), the exact shard pytest prefix, and the run consuming exactly the
    resolver's own shard-<N>-files.txt
  • both test jobs upload .verification-logs/ unconditionally with
    include-hidden-files: true
    (PR Keep one collector per directory when pytest re-collects a parent package #11's law — without it the uploads are empty)
  • build needs: [verify, tests, macos]; global no continue-on-error, no || true /
    | tail / | tee
  • deps: pytest==9.1.0, ruff==0.16.7 (in agreement with the install-surface contract)

The sabotage mutations now target the shard RUN step — the modern equivalent of weakening
the gate (--collect-only appended executes nothing; a disabled shard step skips a tenth of
the suite).

Validation

  • Target suite: 4 failed → 6 passed (all four sabotage mutations still rejected).
  • Neighbors: tests/test_install_surface_contracts.py + tests/test_delivery_contracts.py
    (the macOS-list coupling) — 18 passed, unchanged.
  • ruff check . clean (pinned 0.16.7).

Base: main 8153a96. Head: mission/ci-contract-repin. One file changed
(tests/test_ci_verification_contract.py, +76/−27).

The suite still pinned the pre-migration single-job ops/verify.py gate:
the 'Run authoritative verification' step it demands has not existed
since the workflow became the ten-shard matrix + routed macOS job
(owner commits 9fd7ee7/006d874), so the gate case failed at the missing
step and the collect_only/step_disabled sabotage cases crashed mutating
a step that is not there. The ruff pin still asserted 0.15.16, two
versions behind the 0.16.7 the tree adopted with dependabot PR #5 and
that test_install_surface_contracts.py already pins.

The weakening law is restated against the current structure: exact lint
and manifest-collection commands in the verify job, the full ten-shard
matrix, the exact shard pytest prefix consuming the resolver's own file
list, unconditional hidden-file log uploads (PR #11's law) on both test
jobs, build gated on all three, and the same no-disabled-step,
no-swallowed-output scan. The sabotage cases mutate the shard RUN step
now -- a shard weakened to --collect-only executes nothing. 4 CI
failures (main run 35570948370, every PR run) -> 6 passed;
install-surface and delivery contracts unchanged and green.
@Parad0x-Labs
Parad0x-Labs deleted the mission/ci-contract-repin branch September 23, 2026 03:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant