Skip to content

fix(docs): pre-main-sync — the runbook told operators there is no rollback - #214

Merged
Polichinel merged 1 commit into
developmentfrom
fix/main-sync-preflight
Aug 3, 2026
Merged

fix(docs): pre-main-sync — the runbook told operators there is no rollback#214
Polichinel merged 1 commit into
developmentfrom
fix/main-sync-preflight

Conversation

@Polichinel

Copy link
Copy Markdown
Collaborator

Pre-flight for the developmentmain sync. Three review iterations over the 23-commit diff; each found defects in the previous iteration's fixes.

The one that mattered

docs/operations/correction_procedure.md — the runbook someone follows at 22:00 after a bad delivery — said withdrawal "has no mechanism today" and "there is no rollback." Both false.

views-faoapi ships an operator quarantine (f1a59bf, on its main). Add a store document's file-id to APPWRITE_UNFAO_QUARANTINED_FILE_IDS and selection drops it, falling the consumer back to the previous run — read at selection time, no redeploy, nothing deleted, reversible by unsetting. The runbook instead sent an operator to rebuild and republish an entire corrected run, for hours, while wrong data stayed live.

Iteration 2 found that fix incomplete. A run has two selection entry points: the historical artifact is a separate document selected by category="historical" and never consults the manifest. Quarantining the manifest alone leaves it serving — exactly the wrong way round for a geography fault, since lookup_version lives only on the historical document. §3 now carries a per-leg table.

The env var names are partner-specific. Setting FAO's against a CRAF'd deployment is a silent no-op. Now said in the runbook and in CLONING.md, which had told a new partner these steps "transfer unchanged".

The guard that protected the error

test_the_procedure_distinguishes_intended_policy_from_what_is_implemented had been mutation-proven against the false claim — it asserted the document must say withdrawal costs "an ADR-013 amendment", unbuilt work that was in fact built.

ADR-014 §2 asks whether a guard bites. It never asks whether the thing it bites on is true, and a guard can hold a falsehood in place perfectly. Replaced with one that pins the mechanism, which is checkable against another repository rather than against our own prose.

Other corrections

what why it mattered
README's delivered-schema table had the normative column order reversed and typed the four *_code columns int (wire is float64) a consumer building a reader from it ships a broken one — now guarded against gaul_schema.COLUMNS, mutation-proven
README copied four real registry values into a public file, two lines below the sentence saying they never are the value-copy guard scanned only .py; now scans tracked markdown, matching NAME=value (a substring draft cried wolf on prose that merely names a store)
The UNFAOPostProcessorManager CIC named GaulLookupEnricher as its collaborator in six places the manager has zero references and a test asserts its absence; its sibling CIC said the opposite. Five further CIC claims described deleted code
Runbook: run_id and lookup_version never co-occur; fidelity count 26→24; region-set equality is gated, not always-on an operator would have believed they verified something they did not
ADR-014 said "no test enforces this and none is proposed" two partial guards shipped in the same branch and are green
test_no_partner_contact_details_... scanned the repo root a CI sibling checkout could turn main red with a message naming another repo's file. Scoped to tracked files — via a directory pathspec, because git's ** does not match top-level files and the first fix silently narrowed a security-adjacent scan on a public repo

CI

run_pytest.yml now checks out views-crafdapi. It recovers exactly one test — the §4.1a consumer-name pin for CRAF'd. views-datafactory was tried and reverted: its tests need raw GAUL parquets absent from its git repo, so checking it out converts an honest skip into a FileNotFoundError. views-appwrite and views-faoapi are private — see C-81.

Register

C-80 (ADRs and CICs are exempt from the doc-accuracy scan — the root cause of most of the above), C-81 (CI verifies 17 fewer tests than local, nine new in this arc including both drift detectors, and main has no required check), C-82 (governance-artifact prose drift; one instance under-scopes the pipeline-core bump). C-07, C-15, C-22, C-24, C-57, Cluster H and D-12 corrected. 79/19/60 → 82/22/60.

Verification

402 collected
362 passed / 40 xfailed          local, all siblings present
347 passed / 17 skipped / 38 xf  bare checkout (today's CI)
348 passed / 16 skipped / 38 xf  with the crafdapi sibling
ruff check .  ->  All checks passed!

Every new or changed guard is mutation-proven. The count table above was itself wrong twice — the second time because it was measured on a git clone, which carries committed state while the new tests were still in the working tree. Recorded in C-81, since that entry is about miscounted tests.

Two operator decisions surfaced

  1. Pre-Release Note 07 Topic B.2 tells FAO withdrawal is "not available today" and would need contract plus API work. That is now false. The decision is real; the cost quoted is not.
  2. C-81 — the CI gap needs a token for two private repos, and main's protect_main ruleset has an empty ref list, so a red CI would not block a merge.

…lback

Three review iterations on the development->main sync. Each found defects in the
previous iteration's fixes; the pattern held all three times.

The one that mattered
- docs/operations/correction_procedure.md said withdrawal "has no mechanism today"
  and "there is no rollback". Both false. views-faoapi ships an operator quarantine
  (f1a59bf, on its main): add a store document's file-id to
  APPWRITE_UNFAO_QUARANTINED_FILE_IDS and selection drops it, falling the consumer
  back to the previous run. Read at selection time, no redeploy, nothing deleted,
  reversible. The runbook instead sent an operator to rebuild and republish an entire
  corrected run, for hours, while wrong data stayed live.
- Iteration 2 found that fix incomplete: a run has TWO selection entry points. The
  historical artifact is a separate document selected by category="historical" and
  never consults the manifest. Quarantining the manifest alone leaves it serving —
  wrong way round for a geography fault, since lookup_version lives only there. §3
  now gives a per-leg table.
- The env vars are partner-specific. Setting FAO's name against a CRAF'd deployment
  is a silent no-op. Said in the runbook and in CLONING.md, which had told a new
  partner these steps "transfer unchanged".

The guard that protected the error
- test_the_procedure_distinguishes_intended_policy_from_what_is_implemented had been
  mutation-proven against the false claim: it asserted the doc must say withdrawal
  costs "an ADR-013 amendment". ADR-014 §2 asks whether a guard bites, never whether
  what it bites on is true. Replaced with one that pins the mechanism -- checkable
  against another repository rather than against our own prose.

Other corrections
- README's delivered-schema table had the NORMATIVE column order reversed and typed
  the four *_code columns int where the wire is float64. A consumer building a reader
  from it ships a broken one. Corrected and now guarded against gaul_schema.COLUMNS.
- README copied four real registry VALUES into a public file, two lines below the
  sentence saying they never are. The value-copy guard scanned only .py; it now scans
  tracked markdown, matching NAME=value assignments (a substring draft cried wolf on
  prose that merely names a store).
- The UNFAOPostProcessorManager CIC named GaulLookupEnricher as its enrichment
  collaborator in six places; the manager has zero references and a test asserts its
  absence. Its sibling CIC said the opposite. Five further CIC claims described
  deleted code.
- Runbook: run_id and lookup_version are on different artifacts and never co-occur;
  the fidelity count is 24, not 26; region-set equality is gated, not always-on.
- ADR-014 said "no test enforces this and none is proposed" while two partial guards
  shipped in the same branch.
- test_no_partner_contact_details_... scanned the repo root, so a CI sibling checkout
  could turn main red with a message naming another repo's file. Scoped to tracked
  files -- via a directory pathspec, because git's ** does not match top-level files
  and the first fix silently narrowed a security-adjacent scan.

CI
- run_pytest.yml checks out views-crafdapi. Recovers exactly ONE test. views-
  datafactory was tried and reverted: its tests need raw parquets absent from its git
  repo. views-appwrite and views-faoapi are private -- C-81.

Register: C-80, C-81, C-82 added; C-07, C-15, C-22, C-24, C-57, Cluster H, D-12
corrected. 79/19/60 -> 82/22/60.

402 collected; 362 passed / 40 xfailed locally, 347/17/38 in a bare checkout,
348/16/38 with the crafdapi sibling. ruff clean. Every new guard mutation-proven.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Polichinel
Polichinel merged commit 1c606c7 into development Aug 3, 2026
4 checks passed
@Polichinel
Polichinel deleted the fix/main-sync-preflight branch August 3, 2026 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant