Roadmap item 12: Add the inactive maintenance loop (control bands, scan to intents, incident to eval) - #255
Conversation
… part 1)
Roadmap item 12 asks for deterministic control bands and scans that create new
intents for a service owner to triage. This is the first half: the band policy,
the pure band evaluator, and the scan driver that turns crossed bands and
high-severity scan findings into intent documents.
What it is:
- `maintenance/v1/control-bands.json` — seven named bands with fixed thresholds
and a plain-English line each: no failed eval case, no inconclusive eval case,
no refused event in the sealed trace ledger, at least one suppressed repeat,
at least one recovered stranded attempt, a rollback rehearsal no older than
thirty days, and a zero stale-artifact rate.
- `maintenance/v1/bands.jq` — the shapes every input must have and one pure
function from the supplied documents to a per-band in-band/out-of-band
observation. A metric that cannot be counted from the documents is out of
band, never quietly in band.
- `maintenance/v1/scan.jq` — builds one canonical `maintenance_scan` record
listing every band with its evidence digests, plus one canonical `intent`
document per crossed band and per high-severity finding.
- `maintenance/v1/scan.sh` — the driver. It pins jq 1.6 by digest, snapshots
every input, requires one canonical JSON text per file, proves the trace
ledger is sealed through the existing telemetry validator, and writes only
into a caller-owned empty output directory.
What it never does: it files no issue, opens no change request, deploys
nothing, activates nothing, invokes no model, reads no credential, and grants
no authority. Every intent it writes is a document with `owner: unassigned`,
`triage_state: pending`, and `deploy_authority: none` for a human to triage.
An engaged kill switch (any verdict that is not `satisfied`) makes the scan
record the bands and write no intent at all.
Size: this unit is larger than the ~300-400 line budget, so it lands as stacked
commits on this branch. The incident-to-eval half and the docs follow.
Proof on this tree:
/opt/homebrew/bin/shellcheck -x -S style maintenance/v1/scan.sh
scripts/test/maintenance-loop.test.sh # clean, shellcheck 0.11.0
bash scripts/test/maintenance-loop.test.sh # PASS: 31 maintenance loop checks
bash scripts/check-rename.sh # clean
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…oadmap item 12, part 2)
This is the second half of Roadmap item 12. The scan half landed in the parent
commit; this adds the path back into evals and the documentation for the whole
component.
What it is:
- `maintenance/v1/incident-to-eval.jq` and `incident-to-eval.sh` — given one
shadow incident record and the shadow reproduction record for it, write one
eval seed case skeleton for the family that incident belongs to. The map from
an incident's failing check to a family is closed: a file-digest check belongs
to the stale-and-moved-artifacts family, anything else is refused with
`E_FAMILY` rather than guessed at. The two records must belong together — the
shadow record's incident digest, id, failing check, revision, and repository
all have to match the incident it was handed.
- The expectation comes from the shadow outcome: a reproduced run becomes
`{accepted, stale}`, a run that found nothing becomes the passing baseline
`{accepted, completed}`, and an inconclusive run proves nothing and is
refused. The case's field list, its request role, and the set of expectations
it may use are read out of that family's real seed set in `evals/v1/`, so the
skeleton is in the shape that seed set actually uses rather than a guess.
- The skeleton names the fields it filled and the fields still pending, and
carries a provenance block binding the incident and shadow digests. It never
modifies a seed set: the file is written to the caller's empty output
directory for a later reviewed change.
Docs: one write-up appended to `docs/components.md`, one row in README's
components table, one restore block in `RESTORE.md`, and the seven paths
appended to `ci/required-files.txt`. All of them say the same plain thing: the
intents are documents for a human owner to triage, and nothing here is filed,
deployed, or activated.
Proof on this tree:
/opt/homebrew/bin/shellcheck -x -S style over every tracked *.sh # clean, 0.11.0
bash scripts/test/maintenance-loop.test.sh # PASS: 47 maintenance loop checks
bash scripts/check-rename.sh # clean
ci/required-files.txt structure check # ok, every path present and +x
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Deploying ystack with
|
| Latest commit: |
34520d3
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://391860fc.fabrica-6yx.pages.dev |
| Branch Preview URL: | https://ystack-roadmap-12-maintenanc.fabrica-6yx.pages.dev |
Section 3 said the transition PR touches nothing an agent could have authored, then required README and docs updates; the scope now names the record, the ruleset verification, and exactly those documentation passages. Open questions 6 and 7 offered to proceed with pending units or merge them under restored gates, contradicting the "every row merged" precondition; they now state that the pending PRs (#249–#255) land first and ask only whether any should be dropped from the roadmap. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex reviewer (cross-vendor, read-only)Reviewed-head: fe7d68b Posted verbatim by The maintenance scanner rejects a valid class of telemetry ledgers and one new driver refuses a normal invocation spelling. These are functional issues in the newly added maintenance loop. Full review comments:
|
The bands accepted only recorded result facts, so a sealed ledger the telemetry validator accepts with a computed result fact was refused as malformed after validation; computed facts now count like recorded ones. Both drivers also failed their self-check when invoked with a dotted relative path because the directory was normalized but the file path was not; the path is normalized first, as the sibling evaluators do. Proof: maintenance-loop 49/49 (new cases: a computed refused result crosses its band; a dotted relative invocation runs), shellcheck clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex reviewer (cross-vendor, read-only)Reviewed-head: 0679bc1 Posted verbatim by The maintenance loop can both miss real stale-artifact failures and raise false rollback-age intents for valid inputs. These affect the correctness of the new deterministic control-band behavior. Full review comments:
|
Flooring the per-thousand stale rate let one unresolved case in a large family read as zero and stay in band; it now rounds up. The rehearsal age subtracted calendar dates, overstating the age by a day whenever the dashboard's time of day was earlier than the rehearsal's; it now divides the full-timestamp difference into whole days. Proof: maintenance-loop 51/51 (new cases: one unresolved case in 2000 crosses the band; 23:59 on the 5th to 00:00 a month later is 30 days), shellcheck clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* docs: add the operating-mode transition proposal Construction mode says every output stays inactive until an operator-merged operating-mode transition, but nothing in the repo told the operator what that transition actually is. This adds docs/transition.md as the operator-facing proposal: it is the checklist and the exact proposed edits, not the transition. It records what construction mode built (roadmap step -> unit path -> merged or pending PR), what "implementation-complete" does and does not mean (nothing has run against a real target; the malicious-instructions and reviewer-severity eval families are declared with no seeds; cost/latency and live default-adapter regression evidence are absent), the exact config/construction-mode.json fields the operator's own PR would change and why, the post_transition_ruleset restore (which today is a verification, because the live ruleset already matches it), what steps 7-12 unlock in order, what the operator must supply, what stays human forever, and ten open questions. Claude may not edit config/construction-mode.json, AGENTS.md, REVIEW.md, or ROADMAP.md, and this commit does not: the document only proposes. Also indexes the new file: a one-line pointer next to README's construction-mode paragraph, an appended entry at the end of ci/required-files.txt under "# Operating-mode transition proposal", and a one-line mention in RESTORE.md beside docs/components.md. Proof: - bash scripts/check-rename.sh -> clean - every path in ci/required-files.txt resolves to a regular file (missing=0) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * State the rollout's current status honestly and keep docs in the transition PR The proposal called the twelve rollout units complete while its own table listed open PRs and units still being built; it now describes the table as current state, sets "every row merged" as the precondition for writing the transition PR, and drops the completion claim. The "nothing else" instruction for the transition PR left README and the docs saying construction mode was active after the record retired it; the PR now also updates those passages, as the repo's README/docs-in-sync rule requires. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Make the transition PR's scope and the pending-work questions consistent Section 3 said the transition PR touches nothing an agent could have authored, then required README and docs updates; the scope now names the record, the ruleset verification, and exactly those documentation passages. Open questions 6 and 7 offered to proceed with pending units or merge them under restored gates, contradicting the "every row merged" precondition; they now state that the pending PRs (#249–#255) land first and ask only whether any should be dropped from the roadmap. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Show steps 8 and 12 as their open PRs throughout The status table and the summary said steps 8 and 12 had no branch or PR while the open questions named PRs #254 and #255; every mention now reads the same. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: ci <ci@example.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…ntenance-loop-v1 # Conflicts: # ci/required-files.txt
Codex reviewer (cross-vendor, read-only)Reviewed-head: aea856c Posted verbatim by The maintenance loop can treat malformed stale-family counters as healthy, and the incident converter can emit invalid seed case IDs for accepted inputs. These are edge cases, but they affect correctness of the new deterministic outputs. Full review comments:
|
A dashboard whose stale-family counters did not add up (total 8, passed 9) produced a negative per-thousand rate that read as in band; totals must now equal the sum of their parts for the quality block and every family. An incident id near the length limit produced a "incident."-prefixed case id longer than the seed-case id rule allows; the incident id must now leave room for the prefix. Proof: maintenance-loop 53/53 (new cases: inconsistent counters are refused; an over-long incident id is refused), shellcheck clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex reviewer (cross-vendor, read-only)Reviewed-head: 34520d3 Posted verbatim by I found no actionable correctness issues in the diff. The new maintenance-loop scripts, jq filters, docs, and manifest entries appear consistent with the stated inactive/data-only behavior. |
1 participant
Roadmap step 12 asks for deterministic control bands and scans that create new intents, owners who triage them, and shipped incidents that become evals. This adds the inactive maintenance loop. Everything writes documents into a caller-owned empty directory only: it files no issue, opens no PR, invokes no model, and grants no deploy authority.
maintenance/v1/:control-bands.json(seven named bands with fixed thresholds and plain-English descriptions) andbands.jq(the pure band evaluator over the evals dashboard counts and a sealed telemetry ledger);scan.jq+scan.sh(inputs: dashboard, sealed trace ledger, kill-switch evaluation, empty output directory, and zero or more scan-finding and rollback-rehearsal documents → one canonicalmaintenance_scanrecord plus one intent document per crossed band and per high-severity finding, named deterministically, deduplicated per band, carrying the repo's intent sections with owner unassigned, triage pending, deploy authority none; a kill switch that is not satisfied records the bands and writes no intent);incident-to-eval.jq+incident-to-eval.sh(one incident record plus its shadow record → one eval seed case skeleton in the matching family's exact case shape, read from the family's seed set, with a provenance block binding both digests; it never modifies a seed set, and refuses an unmatched family withE_FAMILY).The ledger must prove sealed through the existing telemetry validator before any band counts an event. Refusals cover relative paths, symlinks, pipes, malformed, multi-root, non-canonical, or oversized inputs, an unsealed or tampered ledger, unknown document kinds, duplicate finding ids, unknown severities, a non-empty output directory, and a driver copied out of its directory.
Notes:
events_refused_maxreads refused results from the sealed ledger rather than the dashboard's passing refusal count; a family with no cases is out of band (fail closed); shapes from #251 (incident and shadow records) and #252 (rollback rehearsal record) are mirrored by kind and field set, copying no code.Review-size note: two stacked commits (bands + scan; incident-to-eval + docs), about 931 net product lines.
Proof run on fe7d68b:
bash scripts/test/maintenance-loop.test.sh→ 47 maintenance loop checks passed (two crossed bands → two deterministic intents, in-band → none, kill switch → scan record only, high-severity finding → intent, byte-identical repeat, every refusal, incident→eval skeleton matches the seed set's expectation set, unmatched family refused)-x -S styleclean;bash scripts/check-rename.shcleanbash scripts/test/portable-core-schema.test.sh→ failures: 0Construction mode: inactive output, no human gate.
🤖 Generated with Claude Code