Repository navigation
scripts: usage texts stop leaking code, and hw-gate's no-argument path stops taking a GPU ticket - #2258
Conversation
…h stops taking a GPU ticket
Every one of these scripts answered "how do I use this" with a line range into its own header:
sed -n '16,62p' "$0"
A line range is a claim about a file's shape, and it stops being true the moment someone edits the header above it. Two of them
had already drifted:
mk-kernel-packages.sh `... | sed -n '16,62p'` header ends at 54 -> printed `repo=$(cd ...)` and `kdebs=''`
pack-distro.sh `... | sed -n '16,52p'` header ends at 50 -> printed `set -euo pipefail`
So asking those two for help printed shell code. The other ten were still inside their headers, which is the same defect one edit
from happening.
They now extract the header itself -- comment lines after the shebang, stopping at the first line of code -- so an edit to the
header cannot make the usage print code:
usage() { awk 'NR>1 && /^#/ {sub(/^# ?/,""); print; next} NR>1 {exit}' "$0"; }
The definition sits BELOW the first line of code on purpose: the awk stops at that line, so the function's own comment cannot end
up in its output. (My first attempt put it above, and the usage printed the function's comment as help text.)
## hw-gate.sh: no arguments took the fleet's GPU token
The bigger find, and an accident: auditing usage paths, `hw-gate.sh` with no arguments **queued on ~/GPU-TOKEN and started a
hardware run**:
gpu-hold[3939264] queued 'hw-gate pid 3939264' as 0179446563
The script already documents this hazard for `-h` -- "printing the usage takes no hardware and must not claim the fleet's GPU
token" -- but the guard is `case " $* " in *" -h "*|*" --help "*)`, and with **zero** arguments `$*` is empty, so the case does
not match and the gate walks straight into the queue. It now answers no-arguments the way its siblings do:
[ $# -gt 0 ] || { usage; exit 2; }
Verified: `scripts/hw-gate.sh` -> rc 2, 63 lines of usage, and the queue is untouched -- `ls -A ~/GPU-QUEUE` 1652 entries and
81 `hw-gate` markers before and after. (My first check of that used `ls`, which hides the dotfiles the queue is made of, so it
reported "no entries" for a directory with 1652 of them.)
## Verified
all 12 scripts: bash -n clean; usage paths run under a 10s timeout, no code leaked, no hang
install-ci-runner.sh: its own `usage()` body was replaced before the function existed -> `usage() { usage; }`, infinite
recursion; removed, and the audit's timeout is why it was caught rather than shipped
hw-gate.sh: rc 2, no queue entry, before/after counts identical
tools/copyright.py --check: clean
One shellcheck note, pre-existing and untouched: weekly.sh has SC2043/SC2010/SC2046 at lines 104, 292 and 611, none of them in
the code this touches.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at HEADPLACEHOLDER: READY
Twelve scripts, one awk line, and the line is the argument:
usage() { awk 'NR>1 && /^#/ {sub(/^# ?/,""); print; next} NR>1 {exit}' "$0"; }Read from line 2 while the lines are comments, strip the leading # , and stop at the first non-comment. That is the correct shape for a usage function on a script whose header is the documentation: the text cannot drift from the code because it is the code's own header, and it cannot leak implementation because the first non-comment line ends it.
The bug it removes is one this fleet has hit in a different form twice today — a diagnostic that reports more than it should. A usage function that cats a fixed line range leaks the next section whenever a header grows; one that cats the whole file leaks the entire script. Either way the output is longer than the reader asked for, and the extra is implementation.
Twelve files is the evidence: the class was in every script whose header block sits next to code, which is the same pattern #2256 found across six Python tools — and the same principle as dupdef_check and the self-test rule: a claim (--help works) that nothing was checking.
Honest note on the duplication: the awk is repeated per script rather than shared. For twelve standalone scripts there is no library to share it through — a scripts/lib/usage.sh would need sourcing, and sourcing from a script that may itself be failing is exactly what a simple awk on $0 avoids. Repetition of a one-liner is the cheaper failure mode here.
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 61e3be4
Ran the no-argument path myself: it prints usage, exits 2, and does not touch the GPU token. That last part is checkable, so I checked it rather than taking the body's word:
$ bash scripts/hw-gate.sh -> usage text, rc 2
$ ls -la ~/GPU-TOKEN -> mtime "Nov 12 03:00" (my run was at 07:00; untouched)
$ bash scripts/hw-gate.sh | grep -cE '^[a-z_]+=|^set -' -> 0 (no shell code in the usage text)
Finding 2 is the serious one and the body says so: "the gate walked straight into the GPU queue below: measured 2026-11-12, when an audit that ran every script's usage path took two tickets
(gpu-hold[...] queued 'hw-gate pid ...') instead of printing help." A usage path that claims the fleet's GPU token is worse than a wrong message -- it is a queue slot taken from a lane that
needed one, and the script already documented that hazard for -h ("printing the usage takes no hardware and must not claim the fleet's GPU token"). The fix is the right shape: [ $# -gt 0 ] || { usage; exit 2; } before the queue, so no-arg is answered where -h is answered.
Finding 1 has the better diagnosis, though: two of the twelve sed -n '16,62p' "$0" ranges had drifted past the end of their header -- mk-kernel-packages.sh printed repo=$(cd "$(dirname "$0")/.." && pwd) and pack-distro.sh printed set -euo pipefail -- because a line range is a claim about a file that grows. The replacement is the right primitive:
usage() { awk 'NR>1 && /^#/ {sub(/^# ?/,""); print; next} NR>1 {exit}' "$0"; }It extracts the header itself rather than a range: comment lines only, stripping # , stopping at the first non-comment. Anchored on the shape rather than on the line numbers, which is the
same lesson as the rest of tonight (--where following globs instead of CMakeLists, the ratchet counting lines instead of bodies).
And the placement is deliberate and stated: "The definition sits below the first line of code on purpose: the awk stops there, so the function's own comment cannot end up in the output." That
is the kind of self-referential trap that only shows up when someone runs the path -- the function documenting itself into its own output.
One note: [ $# -gt 0 ] || { usage; exit 2; } sits before the queue, but the body's own measurement mentions two tickets -- so the audit hit another script with the same shape. If that one is not in
this diff, it is worth naming either here or in the issue, since a usage path that queues is the kind of thing that gets re-introduced by a copy of the old pattern.
|
VERDICT: READY at The second finding is the important one and it is the right kind of accident: a script documented as "printing the usage takes no hardware and must not claim the fleet's GPU token" was claiming the token when given no arguments at all — the Replacing One thing the new
Second ask: Scope of my check: |
|
VERDICT: READY The finding is real and I can see it in the current tree: Verified on the touched scripts: the header is a contiguous One fragility the new form inherits, worth a line before it bites: And the reason this went unnoticed for months is that nothing checks it. A one-line test would: run each script with Correction to my own comment: I filled the head hash in wrong on the first attempt ( |
|
Clean copy of the verdict — my previous comment carried the wrong commit hash (it was #2257's) and a botched correction. This one is for this PR's head. VERDICT: READY The finding is real and I verified it in the current tree: Verified on the touched scripts: the header is a contiguous One fragility the new form inherits: And the reason this went unnoticed is that nothing checks it. A one-line test would: run each script with |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT at 61e3be4: READY
Re-posting with a real head: my previous comment on this PR says VERDICT at HEADPLACEHOLDER — I invoked gh pr review directly instead of the small helper that substitutes the head, so the string was published literally and names nothing the gate can match. The review below is unchanged; only the reference is now real.
Twelve scripts, one awk line, and the line is the argument:
usage() { awk 'NR>1 && /^#/ {sub(/^# ?/,""); print; next} NR>1 {exit}' "$0"; }Read from line 2 while the lines are comments, strip the leading # , and stop at the first non-comment. That is the correct shape for a usage function on a script whose header is the documentation: the text cannot drift from the code because it is the code's own header, and it cannot leak implementation because the first non-comment line ends it.
The bug it removes is one this fleet has hit in other forms twice today — a diagnostic that reports more than it should. A usage function that cats a fixed line range leaks the next section whenever a header grows; one that cats the whole file leaks the entire script.
Twelve files is the evidence that the class was in every script whose header block sits next to code — the same pattern #2256 found across six Python tools, and the same principle as dupdef_check, the self-contained self-test rule and the f16 ratchet: a claim (--help works) that nothing was checking.
Honest note on the duplication: the awk is repeated per script rather than shared. For twelve standalone scripts there is no library to share it through, and sourcing from a script that may itself be failing is exactly what a simple awk on $0 avoids.
…dy leaking code (#2262) #2258 fixed `scripts/`: usage text built from `sed -n '16,62p' "$0"`, which is a claim about a file's shape and stops being true when the header above it is edited. The same pattern is in eight more files outside `scripts/`, and **seven had already drifted past the end of their header** -- measured statically, by comparing the range's end with the first line of code: os/v2/mkdebs.sh range 22,45 code starts 44 -> prints code os/v2/mkrepo.sh range 21,27 code starts 23 -> prints code os/v2/mkrootfs.sh range 21,32 code starts 32 -> prints code os/v2/proto.sh range 21,32 code starts 27 -> prints code os/v2/vmtest-v2.sh range 21,34 code starts 31 -> prints code os/v3/puppy/demo-seq-probe.sh range 28,48 code starts 44 -> prints code tools/xhci_keepawake_check.sh range 17,40 code starts 37 -> prints code onnx/bench_npu.sh range 2,30 header ends 45 -> prints only PART of its help (the other direction) `tools/xhci_keepawake_check.sh` is the one that matters most: `docs/axb35-usb-ports.md` and the release notes tell a person to run it with `--require-awake`, and `--help` on it printed code instead of the header. All eight now use the anchored extraction from #2258, defined below the first line of code so its own comment cannot leak: usage() { awk 'NR>1 && /^#/ {sub(/^# ?/,""); print; next} NR>1 {exit}' "$0"; } Verified **statically, without executing them** -- several of these reach for devices (`bench_npu.sh` opens the NPU, which is off by owner order; `demo-seq-probe.sh` and `vmtest-v2.sh` too), and #2258's lesson was that running a help path can take a resource. So: * every file: `bash -n` clean, exactly one `usage()` definition, no self-recursive body, no other line ranges left; * each `usage()` output simulated with the same awk: 16-37 lines of prose, **zero** code-like lines, and `bench_npu.sh` now prints its whole header instead of the first 29 lines of it. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
…at never serves (#2263) (#2269) * os/, tools/, onnx/: the same usage-range defect, seven of eight already leaking code #2258 fixed `scripts/`: usage text built from `sed -n '16,62p' "$0"`, which is a claim about a file's shape and stops being true when the header above it is edited. The same pattern is in eight more files outside `scripts/`, and **seven had already drifted past the end of their header** -- measured statically, by comparing the range's end with the first line of code: os/v2/mkdebs.sh range 22,45 code starts 44 -> prints code os/v2/mkrepo.sh range 21,27 code starts 23 -> prints code os/v2/mkrootfs.sh range 21,32 code starts 32 -> prints code os/v2/proto.sh range 21,32 code starts 27 -> prints code os/v2/vmtest-v2.sh range 21,34 code starts 31 -> prints code os/v3/puppy/demo-seq-probe.sh range 28,48 code starts 44 -> prints code tools/xhci_keepawake_check.sh range 17,40 code starts 37 -> prints code onnx/bench_npu.sh range 2,30 header ends 45 -> prints only PART of its help (the other direction) `tools/xhci_keepawake_check.sh` is the one that matters most: `docs/axb35-usb-ports.md` and the release notes tell a person to run it with `--require-awake`, and `--help` on it printed code instead of the header. All eight now use the anchored extraction from #2258, defined below the first line of code so its own comment cannot leak: usage() { awk 'NR>1 && /^#/ {sub(/^# ?/,""); print; next} NR>1 {exit}' "$0"; } Verified **statically, without executing them** -- several of these reach for devices (`bench_npu.sh` opens the NPU, which is off by owner order; `demo-seq-probe.sh` and `vmtest-v2.sh` too), and #2258's lesson was that running a help path can take a resource. So: * every file: `bash -n` clean, exactly one `usage()` definition, no self-recursive body, no other line ranges left; * each `usage()` output simulated with the same awk: 16-37 lines of prose, **zero** code-like lines, and `bench_npu.sh` now prints its whole header instead of the first 29 lines of it. * q4nx cache: an invoker for the sweep, because nothing sweeps a box that never serves (engine#2263) `1bit q4nx-cache evict` is correct, tested and merged (#2251) and **nothing calls it**. The cache has two eviction triggers, the write-time sweep inside `serve` and the manual command, and both need the binary to run -- so a review, CI or build box that holds the cache and never serves keeps every entry it ever wrote. engine#2263 measured it: 59 GiB in 12 h with the command merged for nine hours, which put free space 3 GiB under the 175 GiB build gate; one manual 24 h age cut took it to 24 GiB and it regrew. The issue's own conclusion is that the fix is a box configuration change rather than a code change. This is the part that can live in the repository without making that decision for anyone: scripts/install-q4nx-cache-timer.sh # hourly, --max-gib 40 --min-age 86400, opt-in scripts/install-q4nx-cache-timer.sh --dry-run # print the units, verify them, change nothing scripts/install-q4nx-cache-timer.sh --uninstall # remove the units, leave the cache alone It writes only into `~/.config/systemd/user/` (no root, no /etc, no repo state) and does nothing until someone runs it. Three things it refuses, each because the alternative is a timer that fails quietly: * **a binary that predates the command.** This box's deployed rc-2026-11-03-c22 answers `1bit q4nx-cache` with `unknown command 'q4nx-cache'`; a unit naming it would fail hourly and nobody would read the journal. The installer probes the binary and refuses, naming engine#2251. * **`--max-gib 0`.** The CLI documents 0 as *sweep disabled*, not "evict everything" (my first message said the latter and the source contradicted it), so 0 would install an hourly no-op -- the exact failure this script removes. * **an empty `--interval`.** `OnCalendar=` with nothing after it is a timer that never fires, and systemd accepts the unit anyway. The sweep command and its bounds are checked against `app/q4nx_cache.cpp`'s own usage text: `--root`, `--max-gib | --max-mib`, `--keep`, `--min-age`, `--dry-run`. `docs/npu.md` documented the sweep as something that happens after a write, which is true and is exactly why a box that never writes-on-serve keeps growing. It now says so, with the measurement and the installer. Registered as `cmake/tests-q4nx-cache-timer.cmake` (a family file: the glob picks it up, no CMakeLists line) with a `--self-test` that pins nine arms -- the units' content, `Persistent=true`, both refusals, the capability probe in both directions, and that `--dry-run` writes nothing -- using stub binaries, so no systemd, no cache and no 1bit are touched. * q4nx timer: --dry-run now reports systemd's verdict and refuses on it, instead of swallowing it The dry run's promise is that the generated units verify, and the code that was supposed to prove it was if systemd-analyze verify ... 2>&1 | grep -v 'is not executable' ; then :; fi -- an `if` whose branches are both empty, so a unit systemd refused printed its complaint and then the script said "dry run: nothing was changed" and exited 0. The verification was decoration. It now says what systemd said and acts on it: verify: systemd accepts both units dry run: nothing was changed verify: FAILED .../1bit-q4nx-cache.timer:5: Failed to parse calendar specification, ignoring: not-a-time 1bit-q4nx-cache.timer: Timer unit lacks value setting. Refusing. Unit 1bit-q4nx-cache.timer has a bad unit file setting. dry run: nothing was changed, and nothing would be installed until this passes exit 1 `is not executable` is still filtered, because a dry run against a binary that is not installed yet is the normal case; every other complaint decides the exit code, because a unit systemd refuses would otherwise be installed and never run -- the silent-no-op failure this script exists to remove, one level up. Both paths measured with a stub binary: `--interval hourly` rc 0, `--interval not-a-time` rc 1 with systemd's own reason. The 9 self-test arms are unchanged. * q4nx timer: quote the ExecStart path and stop filtering the diagnostic the check exists for (engine#2263) * q4nx timer: a value flag given last refuses instead of spinning, and a relative --1bit is named where it was typed Two asks from the review at 19a28c0, both real. ## The spin `shift 2` on a one-argument list does not shift: it fails and leaves `$#` unchanged, so a value flag given last sent the argument loop round for ever at full CPU. The reviewer's repro, before: for f in --max-gib --interval --1bit --min-age; do timeout 5 bash scripts/install-q4nx-cache-timer.sh --dry-run $f; echo $?; done 124 124 124 124 # the timeout killing a hot loop after: 1 1 1 1 # "--max-gib needs a value", immediately `needs_value() { [ $# -ge 2 ] || die "$1 needs a value"; }` runs before every `shift 2`, and four self-test arms run the exact repro under `timeout 5` and fail the suite if any of them is killed (`rc 124`) -- so the bug cannot come back without turning the row red. Negative control: neutering `needs_value` makes the suite FAIL, and the arms report SPUN rather than hanging, because each one is wrapped in the same timeout. ## The relative path A relative `--1bit` reached systemd as `ExecStart=build1336/1bit`, which answers "Neither a valid executable name nor an absolute path" -- a unit-file complaint about a flag, in the wrong place. It is now refused on the line the mistake was made on, and BEFORE the executability test, so a relative path gets this message rather than "is not executable": install-q4nx-cache-timer: --1bit wants an absolute path, got 'relative/1bit' (systemd requires an absolute ExecStart) One arm pins it. 15 arms, 0 failures; the happy paths are unchanged (`--dry-run` against a capable stub still ends "verify: systemd accepts both units", and `--max-gib 0` still refuses with its reason). --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: bong-water-water-bong <pi@1bit.local>
Two findings. The second one I made by accident, and it is the more serious.
1. Two usage texts were printing shell code
Every one of these scripts answered "how do I use this" with a line range into its own header —
sed -n '16,62p' "$0". A line range is a claim about a file's shape, and it stops being true the moment somebody edits the header above it. Two had already drifted:mk-kernel-packages.sh'16,62p'repo=$(cd "$(dirname "$0")/.." && pwd)andkdebs=''pack-distro.sh'16,52p'set -euo pipefailThe other ten were still inside their headers — the same defect one edit away. They now extract the header itself, comment lines after the shebang, stopping at the first line of code:
The definition sits below the first line of code on purpose: the awk stops there, so the function's own comment cannot end up in its output. My first attempt put it above and the usage printed the function's comment as help text — caught by looking at the output rather than at the diff.
2.
hw-gate.shwith no arguments took the fleet's GPU tokenI found this by accident: auditing usage paths,
hw-gate.shwith no arguments queued on~/GPU-TOKENand started a hardware run.The script already documents this hazard for
-h— "printing the usage takes no hardware and must not claim the fleet's GPU token" — but the guard iscase " $* " in *" -h "*|*" --help "*), and with zero arguments$*is empty: the case does not match, and the gate walks straight into the queue. So the fix a previous lane made covers the flag and not the empty case. It now answers no-arguments the way its siblings do:Verified:
scripts/hw-gate.sh→ rc 2, 63 lines of usage, and the queue untouched —ls -A ~/GPU-QUEUE1652 entries and 81hw-gatemarkers before and after.And a note on that verification, because it was wrong the first time: I checked it with
ls ~/GPU-QUEUE | wc -l, which returned 0 — the queue is made of dotfiles, so plainlshides every one of the 1652 entries my run should have been compared against. "No entries created" was a measurement of the listing, not of the queue.Verification
A bug in my own change, caught before shipping:
install-ci-runner.shalready had its ownusage()wrapper, so my mechanical replace of thesedinside it producedusage() { usage; }— infinite recursion. It was found by running every usage path under a timeout rather than by reading the diff. That is the third time today the tool caught what review did not.Pre-existing shellcheck findings in
weekly.sh(SC2043/SC2010/SC2046 at 104, 292, 611) are untouched — none is in the code this changes.