fix(docs): pre-main-sync — the runbook told operators there is no rollback - #214
Merged
Conversation
…lback Three review iterations on the development->main sync. Each found defects in the previous iteration's fixes; the pattern held all three times. The one that mattered - docs/operations/correction_procedure.md said withdrawal "has no mechanism today" and "there is no rollback". Both false. views-faoapi ships an operator quarantine (f1a59bf, on its main): add a store document's file-id to APPWRITE_UNFAO_QUARANTINED_FILE_IDS and selection drops it, falling the consumer back to the previous run. Read at selection time, no redeploy, nothing deleted, reversible. The runbook instead sent an operator to rebuild and republish an entire corrected run, for hours, while wrong data stayed live. - Iteration 2 found that fix incomplete: a run has TWO selection entry points. The historical artifact is a separate document selected by category="historical" and never consults the manifest. Quarantining the manifest alone leaves it serving — wrong way round for a geography fault, since lookup_version lives only there. §3 now gives a per-leg table. - The env vars are partner-specific. Setting FAO's name against a CRAF'd deployment is a silent no-op. Said in the runbook and in CLONING.md, which had told a new partner these steps "transfer unchanged". The guard that protected the error - test_the_procedure_distinguishes_intended_policy_from_what_is_implemented had been mutation-proven against the false claim: it asserted the doc must say withdrawal costs "an ADR-013 amendment". ADR-014 §2 asks whether a guard bites, never whether what it bites on is true. Replaced with one that pins the mechanism -- checkable against another repository rather than against our own prose. Other corrections - README's delivered-schema table had the NORMATIVE column order reversed and typed the four *_code columns int where the wire is float64. A consumer building a reader from it ships a broken one. Corrected and now guarded against gaul_schema.COLUMNS. - README copied four real registry VALUES into a public file, two lines below the sentence saying they never are. The value-copy guard scanned only .py; it now scans tracked markdown, matching NAME=value assignments (a substring draft cried wolf on prose that merely names a store). - The UNFAOPostProcessorManager CIC named GaulLookupEnricher as its enrichment collaborator in six places; the manager has zero references and a test asserts its absence. Its sibling CIC said the opposite. Five further CIC claims described deleted code. - Runbook: run_id and lookup_version are on different artifacts and never co-occur; the fidelity count is 24, not 26; region-set equality is gated, not always-on. - ADR-014 said "no test enforces this and none is proposed" while two partial guards shipped in the same branch. - test_no_partner_contact_details_... scanned the repo root, so a CI sibling checkout could turn main red with a message naming another repo's file. Scoped to tracked files -- via a directory pathspec, because git's ** does not match top-level files and the first fix silently narrowed a security-adjacent scan. CI - run_pytest.yml checks out views-crafdapi. Recovers exactly ONE test. views- datafactory was tried and reverted: its tests need raw parquets absent from its git repo. views-appwrite and views-faoapi are private -- C-81. Register: C-80, C-81, C-82 added; C-07, C-15, C-22, C-24, C-57, Cluster H, D-12 corrected. 79/19/60 -> 82/22/60. 402 collected; 362 passed / 40 xfailed locally, 347/17/38 in a bare checkout, 348/16/38 with the crafdapi sibling. ruff clean. Every new guard mutation-proven. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-flight for the
development→mainsync. Three review iterations over the 23-commit diff; each found defects in the previous iteration's fixes.The one that mattered
docs/operations/correction_procedure.md— the runbook someone follows at 22:00 after a bad delivery — said withdrawal "has no mechanism today" and "there is no rollback." Both false.views-faoapi ships an operator quarantine (
f1a59bf, on itsmain). Add a store document's file-id toAPPWRITE_UNFAO_QUARANTINED_FILE_IDSand selection drops it, falling the consumer back to the previous run — read at selection time, no redeploy, nothing deleted, reversible by unsetting. The runbook instead sent an operator to rebuild and republish an entire corrected run, for hours, while wrong data stayed live.Iteration 2 found that fix incomplete. A run has two selection entry points: the historical artifact is a separate document selected by
category="historical"and never consults the manifest. Quarantining the manifest alone leaves it serving — exactly the wrong way round for a geography fault, sincelookup_versionlives only on the historical document. §3 now carries a per-leg table.The env var names are partner-specific. Setting FAO's against a CRAF'd deployment is a silent no-op. Now said in the runbook and in
CLONING.md, which had told a new partner these steps "transfer unchanged".The guard that protected the error
test_the_procedure_distinguishes_intended_policy_from_what_is_implementedhad been mutation-proven against the false claim — it asserted the document must say withdrawal costs "an ADR-013 amendment", unbuilt work that was in fact built.ADR-014 §2 asks whether a guard bites. It never asks whether the thing it bites on is true, and a guard can hold a falsehood in place perfectly. Replaced with one that pins the mechanism, which is checkable against another repository rather than against our own prose.
Other corrections
*_codecolumnsint(wire isfloat64)gaul_schema.COLUMNS, mutation-proven.py; now scans tracked markdown, matchingNAME=value(a substring draft cried wolf on prose that merely names a store)UNFAOPostProcessorManagerCIC namedGaulLookupEnricheras its collaborator in six placesrun_idandlookup_versionnever co-occur; fidelity count 26→24; region-set equality is gated, not always-ontest_no_partner_contact_details_...scanned the repo rootmainred with a message naming another repo's file. Scoped to tracked files — via a directory pathspec, because git's**does not match top-level files and the first fix silently narrowed a security-adjacent scan on a public repoCI
run_pytest.ymlnow checks out views-crafdapi. It recovers exactly one test — the §4.1a consumer-name pin for CRAF'd. views-datafactory was tried and reverted: its tests need raw GAUL parquets absent from its git repo, so checking it out converts an honest skip into aFileNotFoundError. views-appwrite and views-faoapi are private — see C-81.Register
C-80 (ADRs and CICs are exempt from the doc-accuracy scan — the root cause of most of the above), C-81 (CI verifies 17 fewer tests than local, nine new in this arc including both drift detectors, and
mainhas no required check), C-82 (governance-artifact prose drift; one instance under-scopes the pipeline-core bump). C-07, C-15, C-22, C-24, C-57, Cluster H and D-12 corrected. 79/19/60 → 82/22/60.Verification
Every new or changed guard is mutation-proven. The count table above was itself wrong twice — the second time because it was measured on a
git clone, which carries committed state while the new tests were still in the working tree. Recorded in C-81, since that entry is about miscounted tests.Two operator decisions surfaced
main'sprotect_mainruleset has an empty ref list, so a red CI would not block a merge.