Promotion: develop → main (pinned 38eeea27 — GT-671 closes, its exemption expired by itself) - #593
Closed
beyondnetPeru wants to merge 3 commits into
Closed
Promotion: develop → main (pinned 38eeea27 — GT-671 closes, its exemption expired by itself)#593beyondnetPeru wants to merge 3 commits into
beyondnetPeru wants to merge 3 commits into
Conversation
…nd the gate called that a pass (#590) fix(mcp,release): --version starts the server instead of answering, and the gate called that a pass FOUR DEFECTS. THE FIRST WAS VISIBLE IN THE RELEASE LOG; EACH OF THE OTHERS WAS FOUND BY TESTING THE FIX FOR THE ONE BEFORE IT. 1. `evolith-mcp --version` boots the MCP server. parseArgs read the command as `args.find((a) => !a.startsWith('-')) ?? 'serve'`. Every flag spelling starts with `-`, so it fell through to 'serve'. Measured against the PUBLISHED 1.3.2: stdin closed -> exit 0, stdout EMPTY (the stdio transport takes EOF and leaves) stdin open -> never returns (killed at 10s, printed nothing) The second is what a terminal, a doctor script or a CI probe does, and asking a tool its version is the first thing anyone does after installing it. `evolith-mcp version` — the positional — worked the whole time, which is why nothing noticed. `--version`, `-v`, `-V`, `--help` and `-h` are now commands by intent, and `--help` prints the USAGE string that already existed and had no caller. 2. A flag's VALUE was read as the command. Found because a test written for defect 1 failed for an unrelated reason: `evolith-mcp --transport http` resolved command to 'http' and exited 1 with `Unknown command: http`, since `http` is the first token not starting with `-`. Argv is now classified in one pass that skips the value each value-taking flag consumes, and both answers are read off that. This also means `--api-key -v` keeps `-v` as the key instead of printing a version. 3. The smoke gate's boot check asserted only the exit status. const res = run(process.execPath, [bin, '--version'], { cwd: treeDir }); if (res.status !== 0) fail([...]); console.log(`✓ the installed binary boots: --version prints ${String(res.stdout).trim()}`); It PRINTED stdout it never inspected. Release run 31986300098 logged, verbatim: ✓ the installed binary boots: --version prints An empty observable, reported as a pass — while defect 1 was already on the registry. Now: a 30s timeout, so a binary that hangs fails instead of hanging CI; a non-empty assertion; and the printed version must CONTAIN the version its own manifest declares, which is what keeps it from passing on any non-empty string. Exercised against the real script with four fixtures: answers correctly -> exit 0 ✓ answers --version: 1.3.0 (matches manifest 1.3.0) prints nothing -> exit 1 `--version` exited 0 and printed NOTHING on stdout wrong version -> exit 1 printed "9.9.9", which does not contain 1.3.0 hangs -> exit 1 did not answer within 30s (30s elapsed) The second fixture is exactly the case the release passed. 4. In `--tree` mode the gate decided whether to boot from the WRONG TREE. Found because fixtures 2-4 above passed when they should have failed. `boot` is computed from `pkgRoot/dist/main.js` — this workspace's local build — and then handed to a verifyTree call that inspects a completely different directory. On a checkout that has not been built locally, `boot` is false, the boot check never runs, and the guard prints a pass anyway. Whether the strongest check runs depended on unrelated local state. In `--tree` mode it is now decided from the tree under test. WHAT IS AND IS NOT FIXED HERE The published `@beyondnet/evolith-mcp@1.3.2` still has defects 1 and 2 — npm versions are immutable and this does not republish. It ships on the next version bump. Defects 3 and 4 are in the gate and take effect on the next release run. Related and NOT addressed: the anti-vacuous specifier scan compares against the WORKSPACE manifest rather than the tree's, which is why a `--tree` fixture must declare this repo's four siblings to be accepted at all. Same class as defect 4, different call site. Verification: mcp-server suite 601 tests / 65 suites green. The 11 new parseArgs cases fail 10-to-8 against the shipped parser and pass 18/18 against this one. Guards 43, 03 and 01 exit 0. CI: 32 SUCCESS, 2 SKIPPED, 0 failing.
…actly as designed (#591) The canary's gate assertion was exempted on one symptom: the published MCP server shipped no ruleset corpus (split out as GT-705), so it could not produce a verdict to assert. The exemption was keyed on `installedPackageShipsNoCorpus()` -- a property of the INSTALLED TARBALL, not a version number and not a date. GT-705 shipped as `mcp@1.3.2`. Run 31987205590 against the registry, cli@1.3.1 + mcp@1.3.2, from a throwaway npm prefix with no repository on the resolution path: ✓ the documented `evolith` command is published — bin/evolith present ✓ `init` completes on a clean directory — evolith.yaml written ✓ `validate --format json` returns an ADR-0073 envelope carrying a verdict — failed, 41/159 rules checked ✓ the published MCP server answers a tools/call with a REAL GATE VERDICT — verdict asserted by gate-verdict.assert.js The log contains ZERO occurrences of "exempt". Nobody had to remember to delete anything, and that is the whole point: a blanket skip would still be green today, with the defect fixed by accident and the assertion never having run. This is what the row meant by keeping the assertion and exempting only the exact symptom. All five criteria now MET, including the two that measurement had to correct first -- `--help` is a weak oracle (every published version answers `--version` with exit 0, including the one GT-625 recorded as broken), and the row's named red fixture `cli@1.2.0` is not broken at all, so falsifiability was proven against `cli@1.1.0`. Board: 672/703 done, 2 in progress. Closure record added with its measurements. Verified: 08 / 41 / 57 / 66 / 04 / 01 green, 46 at a fixed point. Signed-off-by: aarroyo <beyondnet.peru@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
📊 Bilingual Coverage ImpactPR Changes
Repository Coverage
✅ Good: All EN changes have ES counterparts. Generated by GitHub Actions |
…s read outputs they never declared (#592) fix(ci): publish-npm asks the registry before publishing, and two jobs read outputs they never declared Closes #569. 1. A HARMLESS RE-RUN MANUFACTURED A BUG REPORT. `publish-npm` ran a bare `npm publish --provenance --access public` with no check that the version was absent. Any re-run against an already-published version — a retry after an unrelated flake, a workflow_dispatch, a re-pushed tag — died on "You cannot publish over the previously published versions". That failure reached `failure-notification`, which files a "Release Pipeline Failed" issue. Three accumulated that way (#492, #552, #553) before anyone read one, on a tracker that is among the first things a visitor sees. The publish is now gated on an explicit question asked BEFORE it: if [ "$(npm view "$NAME@$VERSION" version 2>/dev/null || true)" = "$VERSION" ]; then published=true -> ::notice:: already on the registry, skipping. Not a failure. Idempotence belongs before the publish, as a question with an answer, not after it as a swallowed error. `continue-on-error` or `|| true` ON THE PUBLISH would make a genuine failure indistinguishable from this one — the same defect wearing a different hat, and the issue says so explicitly. The `|| true` here is on a QUERY, where a non-zero exit means "not found", and the answer is then compared rather than ignored. Exercised against the live registry, both directions, under `set -euo pipefail`: @beyondnet/evolith-cli@1.3.0 -> published=true (on npm) @beyondnet/evolith-cli@9.9.9 -> published=false (never published) @beyondnet/evolith-mcp@1.3.2 -> published=true (published tonight) @beyondnet/does-not-exist@1.0.0 -> published=false (no such package) The two false cases are the ones that matter: they prove the query does not abort the step. 2. AN ECHO IS NOT A CONFIRMATION. The old final step was: echo "✅ Published @beyondnet/evolith-cli@${{ needs.release-gate.outputs.version }} to NPM" It reported success with no evidence — and printed it with an EMPTY version, because of defect 3. Replaced with a registry poll (5 attempts, 10s apart) that fails the job when npm does not hold what was just published, matching what npm-release.yml already does. 3. TWO JOBS READ `needs.<job>` FOR A JOB THEY DID NOT DECLARE. `needs.<job>` resolves only for jobs listed in that job's `needs`. `publish-npm` referenced `needs.release-gate.outputs.version` with `needs: build-and-test`, so the version was the empty string. Sweeping for the CLASS rather than the instance found a second, worse one in the same file: `upload-assets` interpolates `needs.release-gate.outputs.tag_name` into the GitHub Release's `tag_name` AND `name`, with `needs: [package-binaries, smoke-test, smoke-test-functional]`. That publishes a Release named "Release " against an empty tag. Both declared now. A sweep over every workflow in the repository reports zero dangling `needs.<job>` references remaining. NOT DONE HERE, and worth saying: nothing enforces this. The sweep above is a throwaway script in a commit message, not a guard — the 90-day freeze holds — so the next dangling reference will be found the same way this one was, by someone looking. CI: 30 SUCCESS, 2 SKIPPED, 0 failing.
Contributor
Author
|
Ceding the id to #594, which is pinned at |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotion of
develop→main, pinned at38eeea27.#591 — GT-671 is DONE. Its gate assertion was exempted on exactly one symptom: the published MCP server shipped no ruleset corpus (
GT-705), so it could not produce a verdict to assert. The exemption was keyed oninstalledPackageShipsNoCorpus()— a property of the installed tarball, not a version number and not a date.GT-705shipped asmcp@1.3.2, and canary run 31987205590 against the registry (cli@1.3.1+mcp@1.3.2, throwaway npm prefix, no repository on the resolution path) asserts a real gate verdict. The log contains zero occurrences ofexempt— nobody had to remember to delete anything. A blanket skip would still be green today with the defect fixed by accident.#590 fixes
--versionstarting the MCP server instead of answering, which the gate was scoring as a pass.Board: 672/703 done, 2 in progress.
🤖 Generated with Claude Code