ci(ratchet): let the baseline be re-measured where it is used - #729
Open
gHashTag wants to merge 2 commits into
Open
ci(ratchet): let the baseline be re-measured where it is used#729gHashTag wants to merge 2 commits into
gHashTag wants to merge 2 commits into
Conversation
Six gates are TIMEOUT at the recorded 120 s. Timed on a ten-core machine, five of the six finish -- audit_pack_layouts 99.6 s, verify_intrinsic_invariants 154.3 s, audit_ci_health 169.1 s (all three exit clean), audit_dsp_inference 185.2 s and audit_yosys_reads 212.8 s (findings). Only audit_selftest_sensitivity still exceeds 400 s. So half the TIMEOUT column is a property of the runner, and audit_pack_layouts fits inside 120 s here while missing it there. The obvious move is a per-gate timeout, and this module argues against it in its own words: the timeout is "part of the MEASUREMENT, not a convenience knob", and a script "TIMEOUT at 90s and clean at 300s has not changed". Handing slow gates more seconds would report fixes that did not happen -- moving the ruler until the board fits. There is already a broken-ruler guard here for the same reason: any TIMEOUT-involving degradation is re-run serially. What was actually missing is the ability to re-measure *where the number is used*. `--update` on a laptop writes a laptop's baseline over the runner's, and this repository has spent a good deal of effort on exactly that class of bug. So: a manual `rebaseline` input that runs `--update` on the runner, prints the gates that moved into the step summary, uploads the result as an artifact, and restores the committed file. Nothing is committed by the workflow. Raising the timeout does not repair a gate, it changes what its status means, and that is a decision for a person -- who now has the measurement to make it with.
… with audit_gf64_chain.py and audit_hardware_rows.py shell out to `gh`. Without a token `gh` exits non-zero, both scripts report "needs input" (exit 2), and the ratchet reads that as clean -> needs input against a baseline measured where a token existed. Every open PR shows it, including one that changes no gate, so it is the environment and not the tree. GITHUB_TOKEN is the built-in per-run token: no secret to add, scoped to this repository, and it expires with the job. The gates then either pass or find something -- either way they are looking, which is what a gate that cannot see its subject was never doing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Six gates sit at TIMEOUT in gate_status_baseline.json. Timed individually on a ten-core machine:
Five of six finish, three are clean, and audit_pack_layouts fits inside the recorded 120 s here while missing it there. At least one entry in that column is a fact about the runner, not the script.
What I did not do
Add a per-gate timeout. This module refuses it in its own words: the timeout is 'part of the MEASUREMENT, not a convenience knob', and 'a script that is TIMEOUT at 90s and clean at 300s has not changed'. Handing slow gates more seconds turns three green without anything being repaired — moving the ruler until the board fits. There is already a broken-ruler guard here for the same reason: TIMEOUT-involving degradations are re-run serially.
What was missing
A way to re-measure where the number is used.
--updateon a laptop writes a laptop's baseline over the runner's, which is the substitution half this repository's audits exist for.This adds a manual
rebaselineinput (with explicittimeout) that runs--updateon the runner, prints every gate whose status moved into the step summary, uploads the result as an artifact, and restores the committed file. The workflow commits nothing.Raising the timeout does not repair a gate; it changes what the status means. That is a decision for a person — who now has the measurement to make it with, taken on the machine the number describes.