Roadmap item 8: Add the inactive workflow-scope qualification evaluator - #254
Conversation
Step 8 enables one low-risk qualified workflow scope at a time, in its own operator-merged pull request, after that scope's shadow evidence passes. This first commit adds the thing that gets proposed: a written-down workflow scope, plus the gate policy and the validator that decides whether a scope record is even well formed. Nothing here enables anything. A workflow scope names the target repository, the workflow, the task class, the risk tier, the repo-relative paths the work may touch, the proof kinds and eval families it needs, the shadow environments it must have evidence from, and how many attempts it gets. It always records `enabled: false` and `push_allowed: false`; a record that claims otherwise is refused. Allowed paths are globs with deliberately small teeth: no absolute path, no traversal, no backslash, no `**`, and no wildcard in the first segment, because a wildcard there could expand into any top-level directory and the protected-path check could not bound it. The gate policy is a canonical, inactive document that names the only proposable tier (routine), the protected path prefixes, root files and segments taken from the Roadmap's high-risk list, which shadow outcomes count, and the full set of reason ids. It grants no authority. Proof on this commit: - /opt/homebrew/bin/shellcheck -x -S style scope/v1/validate-scope.sh scripts/test/scope-qualification.test.sh — clean - bash scripts/test/scope-qualification.test.sh — 4 focused checks passed - bash scripts/check-rename.sh — clean Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…oposal Second stacked commit for step 8. The first added the workflow-scope record and its validator; this one adds the deterministic evaluator that decides whether a scope may be proposed for enablement, and the proposal document an operator pull request would carry. Enabling a scope is still an independent operator-merged pull request after the operating-mode transition. This code enables nothing. The evaluator reads seven documents: the scope record, a set of shadow reproduction records in the shape step 7 emits, the eval dashboard, a risk-gates evaluation, a kill-switch evaluation, a duty-separation evaluation, and the operating-mode marker. It measures all of them itself, including a digest over each individual shadow record, so nothing a supplied document claims about its own identity is trusted. The answer is `proposable` only when the tier is routine and the risk gate agrees and reports no violation, no allowed path touches a protected path, every required shadow environment has a conclusive record for this repository and none is inconclusive, every required eval family is seeded with no failing or inconclusive grade, the kill switch is clear, duty separation is satisfied, and the mode is readable. Otherwise it is `not-proposable` with reason ids. `config/construction-mode.json` is read read-only and only ever compared. A supplied marker that disagrees with the committed one leaves the mode unknown and the evaluator refuses rather than guessing. Under construction mode the best possible answer is `proposable`; `enabled` is never an outcome in any mode, and the proposal records `authority: "none"`, `enabled: false`, `push_allowed: false`, and a blocked enablement state. The two commits together are about 600 net lines, over the soft budget, so the record and validator landed first and the evaluator lands here; the gate program and its driver are one concern and cannot be split further. Proof on this commit: - /opt/homebrew/bin/shellcheck -x -S style scope/v1/validate-scope.sh scope/v1/evaluate-scope.sh scripts/test/scope-qualification.test.sh — clean - bash scripts/test/scope-qualification.test.sh — 16 focused checks passed - bash scripts/check-rename.sh — clean Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Deploying ystack with
|
| Latest commit: |
25b9130
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://3d5815da.fabrica-6yx.pages.dev |
| Branch Preview URL: | https://ystack-roadmap-8-scope-quali.fabrica-6yx.pages.dev |
Codex reviewer (cross-vendor, read-only)Reviewed-head: 3d2b4c9 Posted verbatim by The new evaluator can produce a proposable result from malformed evidence in edge cases, so the qualification decision is not fail-closed as intended. These issues should be fixed before relying on the component. Full review comments:
|
Indexing dashboard families with from_entries let a later passing duplicate shadow a failing entry for a required family; duplicate family ids are now malformed. The mode marker treated any status other than active as operating, so a typo in a portable tree could make a scope proposable; only active (construction) and retired (operating) are meaningful and anything else refuses with scope.mode-construction. Proof: scope-qualification 18/18 (two new cases), shellcheck clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The previous commit closed the mode-status vocabulary to active and retired but left the portable-tree fixture on the old free-form value, so that check failed and the earlier commit's proof line overstated the count: the suite is 17 focused checks, all passing with this fixture change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex reviewer (cross-vendor, read-only)Reviewed-head: ac3c59a Posted verbatim by The evaluator can mark a scope proposable using unbound shadow evidence and can miss protected paths on case-insensitive filesystems. These are correctness/safety issues in the new qualification gate. Full review comments:
|
…otected-path checks Two P2 review findings on the workflow-scope qualification evaluator. A scope record now declares its own evidence. `shadow_evidence_refs` is a required bounded set (1-16) of document refs naming, by id and by digest, the shadow reproduction records the scope claims. A shadow record carries the repository, environment, outcome, and the incident it reproduced, but nothing about the workflow or task class, so before this any reproduced/no-change record for the same repository satisfied any scope and a new scope could ride in on evidence produced for a different workflow. The evaluator now counts a supplied record only when a ref names it exactly and its target repository is the scope's; records nobody claimed are ignored and reported by digest under the evaluation's evidence as `unclaimed_shadow_records`; a required environment covered only by ignored records, or a claimed ref no supplied record answers, refuses with the existing `scope.shadow-evidence-missing`. The reason-id set stays closed. Protected-path names are now compared case-insensitively on both sides: the glob's segments and the policy's prefix, root-file, and segment lists are all downcased before comparison, so on a case-insensitive checkout `agents.md`, `AGENTS.MD`, `src/Auth/login.ts`, and `.GitHub/workflows/x.yml` are protected just as their canonical spellings are. `glob_ok` already refused empty, `.`, and `..` segments; the write-up now states what it enforces. The test builds the shadow evidence set first and then writes the scope record to claim those exact digests, the way a real scope pull request would, and adds cases for an ignored record that would have covered a required environment, a claimed ref with no matching record, a missing or malformed `shadow_evidence_refs`, and the case-folded protected globs. Proof: /opt/homebrew/bin/shellcheck -x -S style clean on scope/v1/*.sh and the test; `bash scripts/test/scope-qualification.test.sh` exits 0 with 19 focused checks passed (was 17); `bash scripts/check-rename.sh` clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs: add the operating-mode transition proposal Construction mode says every output stays inactive until an operator-merged operating-mode transition, but nothing in the repo told the operator what that transition actually is. This adds docs/transition.md as the operator-facing proposal: it is the checklist and the exact proposed edits, not the transition. It records what construction mode built (roadmap step -> unit path -> merged or pending PR), what "implementation-complete" does and does not mean (nothing has run against a real target; the malicious-instructions and reviewer-severity eval families are declared with no seeds; cost/latency and live default-adapter regression evidence are absent), the exact config/construction-mode.json fields the operator's own PR would change and why, the post_transition_ruleset restore (which today is a verification, because the live ruleset already matches it), what steps 7-12 unlock in order, what the operator must supply, what stays human forever, and ten open questions. Claude may not edit config/construction-mode.json, AGENTS.md, REVIEW.md, or ROADMAP.md, and this commit does not: the document only proposes. Also indexes the new file: a one-line pointer next to README's construction-mode paragraph, an appended entry at the end of ci/required-files.txt under "# Operating-mode transition proposal", and a one-line mention in RESTORE.md beside docs/components.md. Proof: - bash scripts/check-rename.sh -> clean - every path in ci/required-files.txt resolves to a regular file (missing=0) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * State the rollout's current status honestly and keep docs in the transition PR The proposal called the twelve rollout units complete while its own table listed open PRs and units still being built; it now describes the table as current state, sets "every row merged" as the precondition for writing the transition PR, and drops the completion claim. The "nothing else" instruction for the transition PR left README and the docs saying construction mode was active after the record retired it; the PR now also updates those passages, as the repo's README/docs-in-sync rule requires. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Make the transition PR's scope and the pending-work questions consistent Section 3 said the transition PR touches nothing an agent could have authored, then required README and docs updates; the scope now names the record, the ruleset verification, and exactly those documentation passages. Open questions 6 and 7 offered to proceed with pending units or merge them under restored gates, contradicting the "every row merged" precondition; they now state that the pending PRs (#249–#255) land first and ask only whether any should be dropped from the roadmap. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Show steps 8 and 12 as their open PRs throughout The status table and the summary said steps 8 and 12 had no branch or PR while the open questions named PRs #254 and #255; every mention now reads the same. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: ci <ci@example.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…e-qualification-v1 # Conflicts: # ci/required-files.txt
Codex reviewer (cross-vendor, read-only)Reviewed-head: 69d6276 Posted verbatim by The evaluator can propose a scope while carrying claimed shadow evidence with an unrecognized outcome, as long as another accepted record covers the required environment. That violates the fail-closed intent of the qualification gate. Review comment:
|
The record shape check accepted any string as an outcome, so a claimed record with an unexpected outcome such as "failed" passed the shape check and was neither counted nor refused. Outcomes must now be one of the accepted or refused values the policy names; anything else makes the evidence set malformed. Proof: scope-qualification 19/19 (new case: an unknown outcome is malformed), shellcheck clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex reviewer (cross-vendor, read-only)Reviewed-head: 362073e Posted verbatim by The new evaluator can qualify scopes with globs that may touch protected path segments, so the core safety check is incomplete. This should be fixed before the patch is considered correct. Review comment:
|
The protected-path check inspected wildcards only in directory segments, so a leaf glob such as src/* or src/auth* passed although it can expand to a protected segment like auth. The leaf pattern is now matched against the protected segment names (and, for a single-segment glob, the protected root files), and a match makes the path protected. The test also names the input when the evaluator refuses a call it expected to succeed. Proof: scope-qualification 20/20 (new cases: src/*, src/auth*, lib/secret? are protected; src/*.ts stays proposable), shellcheck clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e-qualification-v1 # Conflicts: # README.md # RESTORE.md # ci/required-files.txt # docs/components.md
Codex reviewer (cross-vendor, read-only)Reviewed-head: 11577cf Posted verbatim by The new inactive evaluator under-binds both the scope record and the gate evidence it accepts. Those issues can allow proposals to rely on evidence from a different execution context or workflow scope. Full review comments:
|
… gate evidence Two P2 review findings on the workflow-scope qualification evaluator. A scope record could not record what its shadow evidence was gathered under, so a profile, adapter config, permission set, model, effort, prompt or skill version, verification instruction set, or target version could change after the evidence was gathered and the same record would still qualify. The record now carries a required `qualified_identity` — resolved profile ref, adapter config refs, model request, prompt and skill refs, verification instructions ref, and the target revision in the scope's own target repository — validated by shape, and the evaluator counts a claimed shadow record only when its repository and the revision it ran against are the ones that identity names. Anything the shadow slice does not record (profile, adapter config, model, prompt, skill, verification instructions) is stated plainly in the docs as not yet bindable. Risk, kill-switch, and duty evaluations were accepted on kind and verdict alone, so evaluations produced for another workflow or attempt could make a scope proposable. The scope now names them the way it names shadow evidence: a required `gate_evidence_refs` with the three refs by kind, id, and digest. The evaluator requires each supplied document's measured digest to equal the claim, the three to name the same policy set, the risk and duty evaluations to be about the same stage and about the scope's own resolved profile, and the risk and kill-switch evaluations to each name this duty evaluation. Any of those failing is `scope.malformed`; the verdict reason ids stay reserved for real verdict problems and the reason-id set stays closed. Proof: - /opt/homebrew/bin/shellcheck -x -S style: clean - bash scripts/test/scope-qualification.test.sh: 22 focused checks passed (was 20) - bash scripts/check-rename.sh: clean Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e-qualification-v1 # Conflicts: # README.md # RESTORE.md # ci/required-files.txt # docs/components.md
Codex reviewer (cross-vendor, read-only)Reviewed-head: f24ecf1 Posted verbatim by The scope evaluator can mark a scope proposable using insufficiently validated gate evidence, and its marker handling does not match the documented fail-closed input contract. The README table split is lower severity but also introduced by the patch. Full review comments:
|
Two P2 review findings on the workflow-scope qualification evaluator. The three gate evaluations were accepted on their envelope, inactive markers, policy set, verdict, and reason ids only, so a hand-written document with none of the real evaluator's evidence could stand in for a risk-gate, kill-switch, or duty-separation evaluation. Each is now checked against its own evaluator's complete output — control/v1/risk-gates.jq, control/v1/kill-switch.jq, and control/v1/duty-separation.jq — with an exact body key set, every nested reference validated by shape and by kind or media type, authority_effect "none" where that evaluator emits it, the evaluator's own verdict vocabulary, and its verdict-to-reason consistency: a kill switch is satisfied only as kill.cleared-current, a risk gate is inconclusive only when every reason is one of its two unknowns, and a duty separation is satisfied only as duty.satisfied and inconclusive only as actual.capability-unclassified. The shared predicates are the ones deploy/v1/deploy_contracts.jq and loop/v1/review-fix-planner.jq already use, kept as one definition each; anything short of the real shape is scope.malformed. Every other input was snapshotted, size-bounded, required to be exactly one JSON text, and required to be canonical, but the mode marker was skipped, so a pretty-printed or multi-root marker was accepted and its digest recorded. It now runs through the identical pipeline and refuses with the same error ids. The committed marker in config/construction-mode.json is pretty-printed and can never be rewritten, so it is put into canonical form in the scratch directory before the byte comparison; two markers are the same marker when their canonical bytes agree, and the committed file is still only ever read. Tests rebuild the three gate fixtures as full evaluator documents and add: a kill evaluation missing state_ref, a risk evaluation with an extra body key, a kill satisfied carrying an extra stop reason, a duty evaluation naming a wrong-kind stage ref, a pretty-printed marker, and a two-root marker. The verdict cases now carry the reasons their evaluators emit with those verdicts. Proof: /opt/homebrew/bin/shellcheck -x -S style clean on the driver and the test; bash scripts/test/scope-qualification.test.sh -> "scope qualification: 24 focused checks passed", exit 0; bash scripts/check-rename.sh clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Blank lines between rows split the README component table so later rows rendered outside it; the rows are one table again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
One blank line still split the table before the shadow slice row. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e-qualification-v1 # Conflicts: # README.md # RESTORE.md # ci/required-files.txt # docs/components.md
Codex reviewer (cross-vendor, read-only)Reviewed-head: d09d3c1 Posted verbatim by The scope evaluator can accept mismatched gate evidence because nested duty references are not bound by digest. This can incorrectly qualify a workflow scope with evidence assembled from different duty-evaluation bytes. Review comment:
|
The risk-gate and kill-switch evaluations were tied to the duty evaluation by id alone, so outputs computed over different duty bytes could combine into one proposal. Both must now name the duty evaluation's digest as well; the test's duty-verdict fixtures recompute the risk and kill documents over the mutated duty bytes so the refusal under test stays the duty verdict alone. Proof: scope-qualification 25/25 (new case: a kill-switch evaluation naming the duty id over other bytes is malformed), shellcheck clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e-qualification-v1
The merge brought back blank lines between rows; the rows are one table again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex reviewer (cross-vendor, read-only)Reviewed-head: 8ba23f5 Posted verbatim by The new evaluator can mark a scope proposable using evidence documents that do not match the actual producer outputs it is supposed to trust. These validation gaps undermine the qualification decision even though the component remains inactive. Full review comments:
|
…ns agree Two P2 review findings on the workflow-scope qualification evaluator. The dashboard check accepted a hand-picked subset — the envelope, the activation state, and a families array — so a stub carrying nothing else could drive a proposable outcome even though the evals framework emits one fixed shape. The shape predicate and every helper it calls are now copied into the top of scope/v1/scope-gates.jq out of evals/v1/evals.jq and the core v2 schema module, under a header naming the commit, and the dashboard is judged against that whole shape plus the scope-specific reads it already made. Two adaptations are noted in that header: the schema:: module qualifiers are dropped, because this program is run without a module path and the copied helpers are those module functions byte for byte, and evaluator_shape is read structurally instead of against the pinned closure and run digests, which this evaluator replays nothing to know and which would otherwise make an unrelated edit anywhere in the tree refuse every scope. The four dashboard-shape parameters only an evals run holds are read back out of the candidate, and what that cannot establish — the catalog reference, the evaluator digest, and the observation time — is checked beside it. The three gate documents were bound by digest but free to contradict each other: a duty evaluation saying violated beside a risk evaluation whose reasons carry no duty.violated, or a kill evaluation without kill.duty-violated, is a combination no run produced. The duty verdict must now match the reasons the other two evaluators emit for it, mirroring control/v1/risk-gates.jq and control/v1/kill-switch.jq, including that kill.duty-unverifiable never belongs beside a duty evaluation this program has already accepted whole and that the kill switch's own attempt-invalid and state-invalid reasons stand alone. Any violation is scope.malformed; the reason-id set stays closed. The test's dashboard fixture is now a whole eval dashboard field for field, and the duty-verdict cases carry the risk and kill reasons that follow from each verdict, so they stay consistent — a violated duty therefore also reports scope.tier-not-routine and scope.kill-switch, which is what the real evaluators produce together. Proof: shellcheck -x -S style clean; bash scripts/test/scope-qualification.test.sh → 27 focused checks passed; bash scripts/check-rename.sh clean; a dashboard built by evals/v1/run-evals.sh passes the copied shape. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex reviewer (cross-vendor, read-only)Reviewed-head: f4be860 Posted verbatim by The evaluator can accept evidence that is not actually bound to the reviewed scope, and it accepts shadow records that do not preserve the shadow slice's no-deploy invariant. These gaps can make a scope proposable with stale or forged evidence. Full review comments:
|
… authority Gate evaluations from another workflow or attempt that used the same resolved profile still qualified a scope, because nothing tied them to this scope's stage; the qualified identity now names the stage request the evidence was produced for, and the risk and duty evaluations must be about it. A shadow record claiming any deploy authority passed the shape check; it must carry deploy_authority none like every record the shadow slice emits. Proof: scope-qualification 29/29 (new cases: gate outputs about another stage request are malformed; a shadow record claiming deploy authority is malformed; a scope without a stage request is refused by the validator), shellcheck clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex reviewer (cross-vendor, read-only)Reviewed-head: 6024217 Posted verbatim by The new qualification evaluator can count incomplete shadow evidence as successful reproduction, which undermines a core gate for proposing workflow scopes. There is also a stale-read hole for the committed mode marker. Full review comments:
|
The qualification gate checked a hand-picked subset of the shadow record (the envelope, the inactive/none markers, the outcome, the repository, the revision, and an environment id), so a minimal hand-written document carrying just those fields counted as shadow evidence and could back a scope. It now mirrors shadow/v1/reproduce.sh field for field: the exact top-level and body key sets, the incident, environment claim, registry, sandbox evaluation and trace ledger references by content id and media type, the materialization block with the materializer receipt's source and candidate bound to this record's own repository and revision, the check block with the incident's failing check and the executed digest, and the table of reason id, outcome, and present-or-absent sections that the slice's control flow actually produces. The failing-check shape is copied verbatim from shadow/v1/incident-record.jq; everything the slice builds in bash is written as an exact-key predicate with the line range cited. Anything else is malformed, so the set is refused rather than read. The evaluator also rechecks the committed operating-mode marker after the gate program runs. It snapshots that file early to decide the mode and the repository_marker field, and the post-run stale checks covered the policy, the programs, and the explicit inputs but not the marker, so marker bytes that changed during the run could still be reported as matched. Proof: shellcheck -x -S style clean on the driver and the test; bash scripts/test/scope-qualification.test.sh passes, exit 0, 31 focused checks (was 29); bash scripts/check-rename.sh clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex reviewer (cross-vendor, read-only)Reviewed-head: 3356af5 Posted verbatim by The evaluator can accept malformed shadow reproduction evidence as qualifying evidence for a scope. That undermines the main safety decision this patch adds. Review comment:
|
A record could say reproduced or no-change while its materialization carried another stage-result outcome such as changed or failed; the shadow slice can only ever record a no-change materialization because it applies an empty patch to the incident revision. Anything else is refused as malformed. Proof: scope-qualification 31/31 (new mutation: a changed materialization is malformed), shellcheck clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex reviewer (cross-vendor, read-only)Reviewed-head: 5eecc77 Posted verbatim by The evaluator can accept an inconsistent or incomplete eval dashboard and still mark a scope proposable, which undermines the qualification gate added by this patch. Review comment:
|
A seeded family with cases but no passing case, or with a passed count the total does not account for, was not an eval failure, so a scope could be proposed without any passing evidence. The family now needs at least one case, no failed or inconclusive case, and passed equal to total. Proof: scope-qualification 31/31 (new case: a family with one case and no passing case is eval-failing), shellcheck clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex reviewer (cross-vendor, read-only)Reviewed-head: aaad53a Posted verbatim by The evaluator can accept malformed dashboard evidence as qualifying input, and it can fail open when the repository mode marker exists but is non-regular. These are safety-gate correctness issues in the new scope qualification component. Full review comments:
|
…marker A family's cases block was accepted with only three counters checked, so an extra or missing counter passed and eval reasons could be skipped; the block must now be exactly the four counters, each a count, adding up to the total. A committed mode marker present as a symlink was treated as absent, letting a supplied marker decide the mode; anything but a regular file now refuses. Proof: scope-qualification 32/32 (new cases: an extra counter is malformed; a symlinked committed marker refuses), shellcheck clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex reviewer (cross-vendor, read-only)Reviewed-head: abb7ab8 Posted verbatim by The component has edge-case correctness issues: a valid maximum-length scope can generate an invalid proposal id, and the mode-marker stale check misses non-regular path appearances. These should be fixed before treating the patch as correct. Full review comments:
|
…tale A scope id near the id limit produced a "proposal."-prefixed proposal id longer than the repository's id rule allows; the scope id must now leave room for the prefix. The final marker check only refused a regular file that appeared mid-run, so a symlink or directory appearing at the marker path fell through; anything that appears there now makes the run stale. Proof: scope-qualification 32/32 (new cases: an over-long scope id is refused by the validator; a symlink appearing at the marker path mid-run is stale), shellcheck clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex reviewer (cross-vendor, read-only)Reviewed-head: 69b901a Posted verbatim by The evaluator can accept gate evidence shapes that the underlying control evaluators cannot emit, which weakens the qualification gate's malformed-evidence protection. Review comment:
|
A hand-edited evaluation whose envelope id no longer matched the document it described could be re-claimed by regenerating the scope's refs, because the binding compared only the renamed envelope. The risk and duty ids must now be their stage result's id and the kill-switch id its attempt's id (or the fixed kill-attempt.invalid), so a renamed envelope is malformed. Proof: scope-qualification 33/33 (new cases: a renamed risk id and a renamed kill-switch id are malformed), shellcheck clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex reviewer (cross-vendor, read-only)Reviewed-head: e0a24ec Posted verbatim by The new evaluator can accept eval dashboards that do not match the eval framework's emitted family metadata shape, allowing invalid evidence to qualify a scope. This is a correctness issue in the gate logic. Review comment:
|
A dashboard family marked seeded with no seed source, or with grader kinds or a trial policy no valid catalog could hold, passed the dashboard shape check because only the id, runs, and cases were read. The family metadata the dashboard copies through is now checked with the eval catalog's family rules, copied verbatim (grader kinds, trial policy, seed status, seed sources, and the seeded-implies-source and model-only-implies-multi-trial relations). The test fixture's seeded families name a real seed source. Proof: scope-qualification 33/33 (new case: a seeded family without a seed source is malformed), shellcheck clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex reviewer — DEGRADED, REVIEW DID NOT RUN (cross-vendor, read-only)Attempted-head: 25b9130 This Codex run FAILED TO RUN a genuine review — this is NOT a clean verdict. Do not Reason: codex exited non-zero (1). Posted by |
Codex reviewer (cross-vendor, read-only)Reviewed-head: 25b9130 Posted verbatim by No prioritized, actionable correctness issues were identified in the diff. The new scope evaluator, validator, tests, manifest entries, and documentation appear consistent with the stated inactive/restore-critical behavior. |
1 participant
Roadmap step 8 asks to enable one low-risk qualified workflow scope at a time, in an independent PR, after that scope's own shadow evidence passes. Qualification is authority attached to an exact recorded scope, never reputation. This adds the deterministic evaluator that decides whether a scope may be proposed for enablement and emits the proposal document an operator PR would carry. It never enables anything.
scope/v1/:scope-policy.json(routine-only tier, protected prefixes and files from the roadmap's high-risk list, accepted shadow outcomes, the reason-id set),workflow-scope.jq+validate-scope.sh(the scope record: target repository, workflow, task class, tier, allowed paths, required proof kinds, required eval families, required shadow environments, max attempts, push never allowed),scope-gates.jq+evaluate-scope.sh(seven inputs: scope record, a shadow evidence set in the shape of PR #251's shadow records, the evals dashboard, risk-gates, kill-switch, and duty-separation evaluations, and the operating-mode marker → one canonicalscope_qualification_evaluation).proposablerequires routine tier with the risk gate classifying routine, no protected path, every required environment covered by a reproduced or no-change shadow record with none inconclusive, every required eval family seeded with no failing grade, kill switch clear, duty satisfied. Otherwisenot-proposablewith a reason id (tier-not-routine, protected-path, shadow-evidence-missing, shadow-inconclusive, eval-family-unseeded, eval-failing, kill-switch, duty-violation, mode-construction, malformed). A proposable result nests a completescope_enablement_proposalwith authority none,enabled:false, enablement blocked while the committed mode marker says construction, and the sentence that enabling is an independent operator-merged PR after the transition.enabledis never an outcome.Notes: the mode marker is a supplied input that must match the committed
config/construction-mode.jsonbyte for byte (read only); the shadow evidence set is a small envelope so each record's digest is measured rather than trusted; a violated risk-gate verdict folds into tier-not-routine because that evaluator never emits satisfied. Depends on the shadow record shape from #251 by shape only.Review-size note: two stacked commits (record + validator, 320 lines; evaluator + docs), about 604 net product lines.
Proof run on 3d2b4c9:
bash scripts/test/scope-qualification.test.sh→ 16 focused checks passed (happy path byte-for-byte against an independently built expected document, every refusal, malformed inputs, portable tree without a marker)-x -S styleclean;bash scripts/check-rename.shcleanbash scripts/test/portable-core-schema.test.sh→ failures: 0Construction mode: inactive output, no human gate.
🤖 Generated with Claude Code