From c9524846f98f0b62f967109c8a5a81150992b023 Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sat, 1 Aug 2026 19:44:33 +0200 Subject: [PATCH 01/23] =?UTF-8?q?chore(register):=20review-rr=20strategic?= =?UTF-8?q?=20=E2=80=94=20curation,=20and=20a=20drift=20class=20the=20guar?= =?UTF-8?q?d=20cannot=20catch=20(#179)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two entries sat under Open with their defects already fixed: - C-63 (the silent config fork) was fixed by #149 on 2026-07-31 — the manager carries zero wire_contract branches and two launch_config asserts — but the entry never moved. - C-47 (the stale reconciliation/ bytecode) was fixed by #177 hours ago. Worth naming precisely: test_register_integrity.py did NOT catch either, and correctly so. It asserts that no entry whose HEADING says RESOLVED sits under Open. Neither heading said so — the code was fixed and the record was not. The guard verifies internal consistency; only reading the tree catches this. C-62's trigger had already fired. It read "when cutting this repo's first release (#125)" — 1.0.0 was tagged on 2026-08-01 and the dependency drag shipped with it. Rewritten to the half that is still ahead: the 3.0.0 bump, plus a re-check before any PyPI publish, which is the point the footprint reaches someone else. Cluster M added: five open concerns, one upstream publish. C-44, C-62, C-72, C-73 and C-07 all resolve on the pipeline-core 3.0.0 pin bump and none can be taken before it. They had been readable as five separate backlog items; they are one blocked action. What this repo owes at the bump is a single verification — confirm the delivery selects the run it expects — recorded in C-73's trigger. Register: 73 concerns, 24 open, 49 resolved. Integrity guard green. Co-authored-by: Claude Fable 5 --- reports/technical_risk_register.md | 90 +++++++++++++++++------------- 1 file changed, 51 insertions(+), 39 deletions(-) diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 3898349..c5567e0 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -6,8 +6,8 @@ | Owner | Dylan Pinheiro / PRIO MD&D Team | | Last Updated | 2026-08-01 | | Total Concerns | 73 | -| Open Concerns | 26 | -| Resolved Concerns | 47 | +| Open Concerns | 24 | +| Resolved Concerns | 49 | --- @@ -78,6 +78,14 @@ covered a single open entry (see Historical clusters below). **The lesson worth carrying, and it is D-11's:** WET-before-DRY was applied *correctly* — the pandas and frame seams ran as deliberate siblings through the migration, and a premature abstraction would have outlived the implementation it existed to unify. What went wrong was not the duplication; it was that the removal condition (*"until run 0 proves the contract path live"*) was written down without a **named trigger to act on**, so the box expired on 2026-07-27 and nobody opened it. D-11 predicted exactly this. **A deferral needs an owner and a trigger, not just a reason.** +### Cluster M: Five open concerns, one upstream publish +**Root cause:** this repo pins `views-pipeline-core >=2.1.3,<3.0.0`, which resolves 2.3.0 from PyPI. Every fix and every removal below exists **only** on pipeline-core's unreleased 3.0.0. None is engineering work here; all five arrive together with one pin bump, and none can be taken before that bump. +**Entries:** **C-44** (the bump itself, deliberately held), **C-62** (the transitive drag — 3.4 GB venv, 31 of 32 Dependabot alerts), **C-72** (the pyarrow CVE whose fix our ceiling excludes), **C-73** (the Tier-2 stale-run selection defect, fixed upstream in their #341), **C-07** (the undeclared `appwrite` dependency, whose transitive path their #345 withdraws). +**Highest tier:** 2 (C-73) +**Fix strategy:** none here. The chain is **views-evaluation 0.5.0 → views-pipeline-core 3.0.0 → this repo's pin bump**, and it moves on the maintainer's platform-wide release signal, not on engineering. What this repo owes at the bump is one verification, recorded in C-73's trigger: **confirm the delivery selects the run it expects**, comparing the resolved `run_id` against the producer's newest published run. +**Resolution scope:** Full for C-62, C-72, C-73, C-07; C-44 closes as the act itself. +**Why this cluster is worth having:** it stops five entries reading as five backlog items. They are one blocked action, and the register should say so rather than let a reader triage them separately five times. + ### Historical clusters (mapper era — all resolved or moot) Clusters **A** (cache architecture never unified), **B** (silent error-hiding @@ -451,24 +459,6 @@ See also C-36 (the resolved strict-xfail conversion this extends), C-44 (the dat --- -### C-47: Stale untracked `reconciliation/__pycache__/` survives the module's retirement and misrepresents the package tree `[backlog]` - -| Field | Value | -|-------|-------| -| ID | C-47 | -| Tier | 4 — pure hygiene: not importable (no `__init__.py`, no sources), untracked, no correctness or reliability impact; its only effect is misleading humans and tools that inventory the tree | -| Source | `manual` (2026-07-19) — maintainer question "I thought reconciliation had moved out?" during the ADR-013 read-through; directory listing showed a phantom `reconciliation/` package | -| Trigger | When the D-12 repo-rename assessment (or any repo-structure audit / fresh assimilation) next inventories `views_postprocessing/` and takes the phantom `reconciliation/` dir as evidence the module still lives here — as happened in-session 2026-07-19 | -| Location | `views_postprocessing/reconciliation/__pycache__/` (untracked bytecode leftovers; sources deleted in #62 / PR #63, `6af2020`) | - -The reconciliation retirement (C-42 cutover leg C2) deleted all tracked sources, but the untracked `__pycache__/` bytecode directory survived on the working machine. Directory listings therefore still show a `views_postprocessing/reconciliation/` package, which already misled one in-session inspection into reporting the migration unfinished. Deletion is a one-liner (`rm -rf views_postprocessing/reconciliation`) deferred by maintainer decision; tracked as a GitHub issue. Resolves on deletion (verify `git status` stays clean and the vpp suite green — trivially expected). - -Cross-refs: C-42 (RESOLVED — the migration this is residue of), D-12 (the rename assessment it could mislead), issue #103 (the live tracker). - -**Verified still present 2026-07-31 (review-rr):** `views_postprocessing/reconciliation/__pycache__/` holds 6 stale `.pyc` files (`proportional`, `grouping`, `module`, `frames`, `validation`, `__init__` — all `cpython-310`). Directory listings still show a phantom `reconciliation/` package. **Tagged `[backlog]`:** Tier 4, one-line fix, already tracked as issue #103 — kept here for completeness, not active risk management. Resolves on deletion. - ---- - ### C-57: PLATFORM-001 coordinate registry is referenced by URL, so nothing detects drift between it and this repo's declared environment | Field | Value | @@ -588,7 +578,7 @@ Cross-refs: C-35 (RESOLVED — the `-1` defect class this guards against), C-59 | ID | C-62 | | Tier | 3 — no correctness or reliability impact: the packages are installed but never imported. The cost is **measured at 3.4 GB of virtualenv** for a repo that writes parquet files, plus an architectural excision that is **real in the source but incomplete in the environment**, landing on the first-ever release. | | Source | `manual` (2026-07-31) — maintainer challenge during the development→main sweep ("Is geopandas back? Is it still here?"), verified against `poetry.lock` and the sibling checkouts | -| Trigger | When cutting this repo's first release (**#125**), or when taking the views-pipeline-core 3.0.0 bump (**C-44**) — verify `geopandas` and `torch` have left the resolved dependency tree. Until 3.0.0 is published, they cannot. | +| Trigger | **The first half FIRED on 2026-08-01** — `1.0.0` was tagged and this drag shipped with it. Remaining trigger: when taking the views-pipeline-core 3.0.0 bump (**C-44**), verify `geopandas` and `torch` have left the resolved dependency tree. Also re-check before any *PyPI* publish, which is the point at which the footprint reaches someone other than this team. | | Location | `poetry.lock` (`geopandas 1.0.1`, `optional = false`); `pyproject.toml:13` (`views-pipeline-core = ">=2.1.3,<3.0.0"`, which resolves to 2.3.0) | This repo declares exactly three dependencies — `views-pipeline-core`, `views-frames`, `pyarrow` — and imports **zero** geospatial libraries. Verified 2026-07-31: the only three mentions of `geopandas`/`shapely` in `.py`/`.toml` are assertions of its *absence* (`enrichment.py:9`, `build_gaul_lookup.py:11`) and a doc-accuracy test that **bans the word** (`tests/test_doc_accuracy.py:29`). C-39's deletion held completely at the source level. @@ -629,24 +619,6 @@ Cross-refs: **C-44** (the held bump — same action, opposing rationale), **C-39 --- -### C-63: A launch config that omits `wire_contract` silently routes into retired code instead of failing - -| Field | Value | -|-------|-------| -| ID | C-63 | -| Tier | 2 — the repo that authored ADR-003 ("authority of declarations over inference") infers its own delivery mode from the *absence* of a config key. A clone, a config refactor, or a typo selects the retired pandas path with no signal; that path's uploads also discard their failure result (#145), so the second failure is silent too. Not Tier 1: the retired path still produces a valid artifact, so this is wrong-path-taken, not wrong-data-shipped. | -| Source | `repo-assimilation` (2026-07-31) — clone-readiness pass | -| Trigger | When writing the launch config for **views-crafdapi** or **views-productionapi**, or when refactoring views-models' `config_meta.py` — verify the manager *raises* on a missing `wire_contract`/`data_format` rather than falling back. It does not today. | -| Location | `views_postprocessing/unfao/managers/unfao.py:233`, `:294`, `:323`, `:409`, `:520` (the `wire_contract` forks); `:130` (the `data_format` fork); views-models `postprocessors/un_fao/configs/config_meta.py:26-27`, `config_queryset.py:62` (the only place both are declared) | - -Two **independent** dispatch axes give four theoretical delivery modes, of which production uses exactly one: `declared_data_format(queryset) == "feature_frame"` (`:130`) selects the frame-native historical read, and `configs.get("wire_contract")` (five sites) selects the ADR-013 contract delivery. Production declares both. **Omitting either silently selects the retired half** — `.get()` returning `None` is indistinguishable from a deliberate `False`. - -This is the inference this repo's own ADR-003 forbids, in the manager that orchestrates the delivery. The correct shape is one path plus a loud refusal naming the missing key. - -Cross-refs: **C-40** (the manager this lives in), **#145** (the retired path's silent upload failures — the second half of the same hazard), **D-11** (the concrete-siblings-and-delete decision whose "delete" step is outstanding), **Cluster L**. - ---- - ### C-71: `appwrite_env.assert_env_declared` raises without logging — ADR-008 non-compliance in an entry-validation seam | Field | Value | @@ -787,6 +759,46 @@ See also C-40 (the inheritance/representation coupling this migration unwinds), ## Resolved Concerns +### C-47: Stale untracked `reconciliation/__pycache__/` survives the module's retirement and misrepresents the package tree `[backlog]` — RESOLVED + +| Field | Value | +|-------|-------| +| ID | C-47 | +| Resolved | 2026-08-01 | +| Resolution | **Resolved by #177 (2026-08-01).** `views_postprocessing/reconciliation/` and its stale bytecode are deleted. The directory had survived the module's retirement in #62 (2026-06-26) and had already misled one in-session inspection into reporting the migration unfinished — which is the harm this entry recorded. Surfaced for action by the `/falsify` audit that disproved "there is nothing more to do in this repo"; it had been known and walked past for six weeks. | +| Tier | 4 — pure hygiene: not importable (no `__init__.py`, no sources), untracked, no correctness or reliability impact; its only effect is misleading humans and tools that inventory the tree | +| Source | `manual` (2026-07-19) — maintainer question "I thought reconciliation had moved out?" during the ADR-013 read-through; directory listing showed a phantom `reconciliation/` package | +| Trigger | When the D-12 repo-rename assessment (or any repo-structure audit / fresh assimilation) next inventories `views_postprocessing/` and takes the phantom `reconciliation/` dir as evidence the module still lives here — as happened in-session 2026-07-19 | +| Location | `views_postprocessing/reconciliation/__pycache__/` (untracked bytecode leftovers; sources deleted in #62 / PR #63, `6af2020`) | + +The reconciliation retirement (C-42 cutover leg C2) deleted all tracked sources, but the untracked `__pycache__/` bytecode directory survived on the working machine. Directory listings therefore still show a `views_postprocessing/reconciliation/` package, which already misled one in-session inspection into reporting the migration unfinished. Deletion is a one-liner (`rm -rf views_postprocessing/reconciliation`) deferred by maintainer decision; tracked as a GitHub issue. Resolves on deletion (verify `git status` stays clean and the vpp suite green — trivially expected). + +Cross-refs: C-42 (RESOLVED — the migration this is residue of), D-12 (the rename assessment it could mislead), issue #103 (the live tracker). + +**Verified still present 2026-07-31 (review-rr):** `views_postprocessing/reconciliation/__pycache__/` holds 6 stale `.pyc` files (`proportional`, `grouping`, `module`, `frames`, `validation`, `__init__` — all `cpython-310`). Directory listings still show a phantom `reconciliation/` package. **Tagged `[backlog]`:** Tier 4, one-line fix, already tracked as issue #103 — kept here for completeness, not active risk management. Resolves on deletion. + +--- + +### C-63: A launch config that omits `wire_contract` silently routes into retired code instead of failing — RESOLVED + +| Field | Value | +|-------|-------| +| ID | C-63 | +| Resolved | 2026-08-01 | +| Resolution | **Resolved by #149 (epic #148), and this entry simply never moved.** The fix landed on 2026-07-31: the manager carries **zero** `configs.get("wire_contract")` branches and **two** `launch_config.assert_*` calls, so an incomplete launch config is refused by name rather than routed into retired code. Verified against the tree during the 2026-08-01 `review-rr` pass. The entry sat under Open for a day with its defect already gone — a drift class `test_register_integrity.py` cannot catch, because that guard checks whether a *heading* says RESOLVED, not whether the *code* is fixed. | +| Tier | 2 — the repo that authored ADR-003 ("authority of declarations over inference") infers its own delivery mode from the *absence* of a config key. A clone, a config refactor, or a typo selects the retired pandas path with no signal; that path's uploads also discard their failure result (#145), so the second failure is silent too. Not Tier 1: the retired path still produces a valid artifact, so this is wrong-path-taken, not wrong-data-shipped. | +| Source | `repo-assimilation` (2026-07-31) — clone-readiness pass | +| Trigger | When writing the launch config for **views-crafdapi** or **views-productionapi**, or when refactoring views-models' `config_meta.py` — verify the manager *raises* on a missing `wire_contract`/`data_format` rather than falling back. It does not today. | +| Location | `views_postprocessing/unfao/managers/unfao.py:233`, `:294`, `:323`, `:409`, `:520` (the `wire_contract` forks); `:130` (the `data_format` fork); views-models `postprocessors/un_fao/configs/config_meta.py:26-27`, `config_queryset.py:62` (the only place both are declared) | + +Two **independent** dispatch axes give four theoretical delivery modes, of which production uses exactly one: `declared_data_format(queryset) == "feature_frame"` (`:130`) selects the frame-native historical read, and `configs.get("wire_contract")` (five sites) selects the ADR-013 contract delivery. Production declares both. **Omitting either silently selects the retired half** — `.get()` returning `None` is indistinguishable from a deliberate `False`. + +This is the inference this repo's own ADR-003 forbids, in the manager that orchestrates the delivery. The correct shape is one path plus a loud refusal naming the missing key. + +Cross-refs: **C-40** (the manager this lives in), **#145** (the retired path's silent upload failures — the second half of the same hazard), **D-11** (the concrete-siblings-and-delete decision whose "delete" step is outstanding), **Cluster L**. + +--- + ### C-09: Publish workflow validates version against wrong PyPI package — RESOLVED | Field | Value | From 5fd15c1b8fd939370fd2ccd222ae7a3b14da576c Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sat, 1 Aug 2026 19:54:34 +0200 Subject: [PATCH 02/23] chore(register): give the upstream-owned concerns their homes, and move C-58 into Cluster M (#180) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Answering "are the things in other repos tracked by gh issues there?" — mostly yes, two were not, and one turned out to be already fixed upstream. ALREADY TRACKED, verified rather than assumed: C-13, C-28 timeouts -> pipeline-core #248, #347 C-73 stale run -> pipeline-core #341 (closed) C-72 pyarrow CVE -> vpp #174, faoapi #348, comment on #280 C-62, C-44 dependency drag -> comment on pipeline-core #319 C-43 area math -> views-datafactory #387 GAPS, now filed: C-26 (Tier 1) -> pipeline-core #366. Our only Tier 1, open since 2026-06-12 with no home in the repo that owns the code. Filed carrying the question this seat cannot answer: does get_feature_frame inherit the unconditional fillna(0.0), or does the frame path propagate NaN? That decides whether C-26 is live — run-0 shipped 28.4M historical rows through the frame path — or historical. Entry stays Tier 1 until answered, not downgraded on a guess. C-27 -> pipeline-core #367, cross-referenced to their #168 as the same defect class on a different call path. C-24 -> cross-referenced on faoapi #222 (their output-schema epic) asking directly whether the Topic-C rename is in scope, with an offer to close our entry pointing there if so. Open fourteen months without a home in the repo that owns the fix. FOUND WHILE CHECKING: C-58's fix has already LANDED upstream. pipeline-core #322 (write-path raise-by-default), #331 (relocate the four create_* sites) and #332 (assert the delivery path does not import provisioning) are all closed — that is þing-02's D5 implemented. But all three are on their 3.0.0, and we resolve 2.3.0, so the defect is fixed upstream and LIVE here until the bump. Same shape as C-73. C-58 joins Cluster M, which is now six entries — two of them Tier 2 — resolving on one publish. The integrity guard caught me writing "C-166" unnamespaced, which is my own convention. Fixed before commit. Co-authored-by: Claude Fable 5 --- reports/technical_risk_register.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index c5567e0..6222797 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -80,10 +80,10 @@ covered a single open entry (see Historical clusters below). ### Cluster M: Five open concerns, one upstream publish **Root cause:** this repo pins `views-pipeline-core >=2.1.3,<3.0.0`, which resolves 2.3.0 from PyPI. Every fix and every removal below exists **only** on pipeline-core's unreleased 3.0.0. None is engineering work here; all five arrive together with one pin bump, and none can be taken before that bump. -**Entries:** **C-44** (the bump itself, deliberately held), **C-62** (the transitive drag — 3.4 GB venv, 31 of 32 Dependabot alerts), **C-72** (the pyarrow CVE whose fix our ceiling excludes), **C-73** (the Tier-2 stale-run selection defect, fixed upstream in their #341), **C-07** (the undeclared `appwrite` dependency, whose transitive path their #345 withdraws). +**Entries:** **C-44** (the bump itself, deliberately held), **C-62** (the transitive drag — 3.4 GB venv, 31 of 32 Dependabot alerts), **C-72** (the pyarrow CVE whose fix our ceiling excludes), **C-73** (the Tier-2 stale-run selection defect, fixed upstream in their #341), **C-58** (the Tier-2 auto-provision-instead-of-raise, fixed upstream in their #322/#331/#332), **C-07** (the undeclared `appwrite` dependency, whose transitive path their #345 withdraws). **Highest tier:** 2 (C-73) **Fix strategy:** none here. The chain is **views-evaluation 0.5.0 → views-pipeline-core 3.0.0 → this repo's pin bump**, and it moves on the maintainer's platform-wide release signal, not on engineering. What this repo owes at the bump is one verification, recorded in C-73's trigger: **confirm the delivery selects the run it expects**, comparing the resolved `run_id` against the producer's newest published run. -**Resolution scope:** Full for C-62, C-72, C-73, C-07; C-44 closes as the act itself. +**Resolution scope:** Full for C-62, C-72, C-73, C-58, C-07; C-44 closes as the act itself. **Six entries, two of them Tier 2, on one publish.** **Why this cluster is worth having:** it stops five entries reading as five backlog items. They are one blocked action, and the register should say so rather than let a reader triage them separately five times. ### Historical clusters (mapper era — all resolved or moot) @@ -211,6 +211,8 @@ The FAO API contract (Release Note 01, Topic C, confirmed and locked) specifies: **D-06 resolved (2026-06-03):** Investigation of views-faoapi confirms NO renaming layer exists. The `FAOApiManager` passes postprocessor column names through to the HTTP response unmodified. FAO receives `country_iso_a3`, `admin1_gaul1_code`, `pg_xcoord` — not the contract-specified names. The column renaming from Release Note 01 Topic C was never implemented in any repo. +**Cross-referenced upstream 2026-08-01** on views-faoapi **#222** (their output-schema epic, which mentions column renaming) asking directly whether the Topic-C rename is in its scope — with an explicit offer to close this entry pointing there if so, or to file it properly if not. Open fourteen months without a home in the repo that owns the fix. + **This is NOT this repo's responsibility to fix.** The schema mismatch is between the API layer (views-faoapi) and the FAO contract. The postprocessor should keep its current column names — changing them now would break views-faoapi's `FAO_PGMDataset._METADATA_COLS` validation. The renaming belongs in views-faoapi as a response-formatting step, coordinated with FAO. See also C-17 (RESOLVED — implicit column naming between mapper and manager), D-06 (resolved: no renaming layer exists). @@ -233,6 +235,8 @@ See also C-17 (RESOLVED — implicit column naming between mapper and manager), Location is in views-pipeline-core, but the impact lands on this repo's FAO delivery; registered here because the consuming call and the delivery responsibility are here. +**Filed upstream 2026-08-01 as views-pipeline-core#366**, carrying the open question this entry could not answer from this seat: **does `get_feature_frame` inherit the same unconditional `fillna(0.0)`, or does the frame-native fetch propagate NaN?** That decides whether C-26 is live (run-0 shipped 28.4M historical rows through the frame path) or historical (it describes only the branch #149 retired). The entry stays Tier 1 until answered — deliberately not downgraded on a guess. + See also C-25 (same data path, wrong-file variant), C-15 (upload provenance would aid post-hoc detection). **OPEN VERIFICATION QUESTION (review-rr 2026-07-31) — tier held at 1 pending an answer.** `fillna` has **zero occurrences in this repo**; the fabrication site is entirely upstream. Since #126, the historical path run-0 actually used is `get_feature_frame` (`_read_historical_frame`), **not** the pandas `get_data` branch that reaches `dataloaders.py:1208`. It could not be verified from this seat (views-pipeline-core is deliberately absent from test environments, per repo convention). **Question for the pipeline-core seat: does `get_feature_frame` inherit the same unconditional `fillna(0.0)`, or does the frame-native fetch propagate NaN?** If it propagates NaN, this Tier 1 now describes only the legacy branch (retirement is the named post-run-0 follow-up) and should be re-tiered. **Do not downgrade on inspection of this repo alone** — the deliverable ran through the unverified path at global scale on 2026-07-27. @@ -249,6 +253,8 @@ See also C-25 (same data path, wrong-file variant), C-15 (upload provenance woul | Trigger | When bumping views-pipeline-core, or changing this postprocessor's queryset/config — verify a `ViewsDataLoader` construction failure surfaces its real exception rather than a downstream `AttributeError`; today it is caught bare, logged as "No Queryset detected" with `exc_info=False`, and replaced with `self._data_loader = None` | | Location | views-pipeline-core `managers/model/model.py:883-902`; crash sites `views_postprocessing/unfao/managers/unfao.py:105` (`_read_historical_frame`), `:134` (`_read_historical_data`) | +**Filed upstream 2026-08-01 as views-pipeline-core#367**, cross-referenced to their **#168** (views-pipeline-core C-166, narrow Appwrite exception handling) as the same defect class on a different call path — catch broadly, guess at the cause, discard the evidence — worth deciding once rather than twice. + `_initialize_data_loader()` catches bare `Exception`, discards the traceback, and nulls the loader. The failure then surfaces as `AttributeError: 'NoneType' object has no attribute 'get_data'` in `_read_historical_data` — the operator debugs the postprocessor while the cause (import error, malformed config, path issue) was erased at construction time. Cost is time-to-diagnosis during exactly the runs where time matters. --- @@ -497,6 +503,10 @@ The practical exposure here is bounded but real. Coordinates are validated for * Tier 2 rather than 1: no *value* is corrupted — the payload is exactly right, it lands in the wrong container — and the consequence is visible downstream (FAO serves nothing) rather than being wrong-but-plausible data. The precedent is already on record: six stranded `orange_ensemble` forecast documents sat invisible in `unfao_bucket` for months (ADR-013 Post-adoption, 2026-07-15) because a *name* filter mismatched — the same class of silent mis-addressing, discovered only by a deliberate read-only audit. +**Update 2026-08-01 — the upstream fix has LANDED, and this entry now rides the 3.0.0 bump (Cluster M).** views-pipeline-core **#322** (*"[þing-01] ADR-046 §5 + write-path raise-by-default"*) is **CLOSED**, as are **#331** (relocate the four `create_*` sites into a dedicated provisioning module) and **#332** (assert the delivery path does not import provisioning). That is D5's ruling implemented: provisioning moved out of the ordinary write path, and the write path raises by default. + +**We do not have it yet.** All three landed on their `development` (3.0.0); our pin resolves 2.3.0 from PyPI, which still auto-creates. So this entry is **fixed upstream and live here** until the bump — the same shape as C-73. Added to **Cluster M**; verify at the bump that a wrong coordinate now raises rather than creating an empty target. + **Not this repo's code to fix.** The fix belongs in views-pipeline-core (make provisioning an explicit opt-in parameter defaulting to off, per D5), and D5's drill ordering is fixed verbatim by the verdict: amend → ship raise → drill the raise path → stand up a test project → drill provisioning. This repo's available mitigations are a post-upload target assertion in `_ContractStorePort`, or a read-back count check after the manifest commits. Cross-refs: C-57 (registry drift — the most likely way a coordinate goes wrong), C-25 (the sibling wrong-*source* selection risk, mitigated by identity assertion), C-13 (the same store calls, timeout dimension), C-40 (the inherited pipeline-core surface this arrives through — **Cluster G**), C-22 (no recall procedure if a mis-delivery is discovered late). From 923667d10832a5d2edc79d807b35d1062e79f60d Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 2 Aug 2026 00:34:00 +0200 Subject: [PATCH 03/23] =?UTF-8?q?fix(env):=20S1=20#182=20=E2=80=94=20appwr?= =?UTF-8?q?ite=5Fenv=20logs=20before=20it=20raises,=20and=20a=20guard=20ov?= =?UTF-8?q?er=20both=20entry=20validators=20(#193)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-008:48 requires raised structural failures to be logged at ERROR; :51 that "raising is not a substitute for logging." `assert_env_declared` raised `EnvironmentError` naming every missing variable and wrote no log record, so an operator reading logs after a refused run found nothing (register C-71). The two-line fix matters less than why it was missing. `contract/launch_config.py` was written in #149 by mirroring this module, inherited the flaw, was fixed in review, and left the module it copied as the odd one out. The new test therefore parametrises the ADR-008 obligation over BOTH validators rather than pinning one: adding a third is one line, and the pair cannot drift again. Deliberately NOT extracted into a shared helper. WET before DRY — they answer different questions (did the launcher assemble the environment / did it declare the delivery mode) against different sources (os.getenv / a passed dict). Named trigger to revisit: a third entry-validation module. Recorded in epic #181. Logging here is only safe because `missing` holds NAMES: membership is decided by `os.getenv(name)` being falsy and the resolved value is never read. CONNECTION_ENV carries the APPWRITE_DATASTORE_API_KEY secret slot, so a second test sets a sentinel value and asserts it never reaches a record — a future "log the current environment for debuggability" cannot land quietly. Mutation-tested: removing the logger.error line fails 2 of the new tests. Also registers C-74, found by this story's own review. tests/test_redaction_guard.py pins the þing-01 #135 redaction audit over five module roots; #153 moved the machinery from unfao/ to contract/ and four were never re-pointed. rglob on a nonexistent directory yields nothing rather than raising, so the guard scans 6 files instead of ~16 and passes. Not a leak — the relocated modules are still credential-blind — but C-57 and Cluster I both vouched for that guard in writing, and both are corrected in place. Tracked as #192. Co-authored-by: Claude Opus 5 (1M context) --- reports/technical_risk_register.md | 48 +++++++++-- tests/test_env_declaration.py | 93 +++++++++++++++++++++- views_postprocessing/unfao/appwrite_env.py | 25 +++++- 3 files changed, 154 insertions(+), 12 deletions(-) diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 6222797..5ed255a 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -4,9 +4,9 @@ |-------------------|--------------------------------------| | Project | views-postprocessing | | Owner | Dylan Pinheiro / PRIO MD&D Team | -| Last Updated | 2026-08-01 | -| Total Concerns | 73 | -| Open Concerns | 24 | +| Last Updated | 2026-08-02 | +| Total Concerns | 74 | +| Open Concerns | 25 | | Resolved Concerns | 49 | --- @@ -45,9 +45,9 @@ covered a single open entry (see Historical clusters below). ### Cluster I: Governance-artifact drift **Root cause:** the register, ADR prose, and issue bodies are hand-maintained mirrors of cross-repo state that moves under them. -**Entries:** C-44, C-46, C-47, C-57 (the cross-repo instance: a registry referenced by URL cannot be diffed by a local test) — plus this register's own findings at review-rr 2026-07-31 (header miscount, two RESOLVED entries misfiled under Open, eight stale `unfao.py` line ranges after the manager grew 273→636 lines, two unnamespaced foreign-register IDs). Historical precedent: the entire C-48–C-55 ADR-013 audit series, and C-42/C-47. +**Entries:** C-44, C-46, C-47, C-57, C-74 (a guard whose declared scan roots silently stopped existing — the cluster's disease inside the cluster's own prescription) — plus this register's own findings at review-rr 2026-07-31 (header miscount, two RESOLVED entries misfiled under Open, eight stale `unfao.py` line ranges after the manager grew 273→636 lines, two unnamespaced foreign-register IDs). Historical precedent: the entire C-48–C-55 ADR-013 audit series, and C-42/C-47. **Highest tier:** 3 -**Fix strategy:** this repo already solved this disease once — the ADR-013 audit series ended with **40 permanent guard tests** (`tests/test_falsify_adr013_*.py`), and the same pattern now guards the þing-01 invariants (`tests/test_env_declaration.py`, `tests/test_redaction_guard.py`). There is **no equivalent for the register**. A small `tests/test_register_integrity.py` — header counts match section counts; no RESOLVED body under `## Open Concerns`; every `C-\d+`/`D-\d+` reference resolves or is namespaced to a foreign register — would make this class self-detecting. +**Fix strategy:** this repo already solved this disease once — the ADR-013 audit series ended with **40 permanent guard tests** (`tests/test_falsify_adr013_*.py`), and the same pattern now guards the þing-01 invariants (`tests/test_env_declaration.py`, `tests/test_redaction_guard.py` — the latter **only over the roots that still exist**, see C-74: a guard is only as good as the assertion that its inputs are real). There is **no equivalent for the register**. A small `tests/test_register_integrity.py` — header counts match section counts; no RESOLVED body under `## Open Concerns`; every `C-\d+`/`D-\d+` reference resolves or is namespaced to a foreign register — would make this class self-detecting. **Resolution scope:** Full for the mechanical half. ### Cluster J: Delivery aftercare has no mechanism @@ -481,9 +481,11 @@ This repo's half is well guarded. `tests/test_env_declaration.py` pins that ever Two named changes are already anticipated and will fire this trigger: the **retirement of the legacy `APPWRITE_DATASTORE_API_KEY`** in favour of the three-tier read/write/provision slots (D4), and any target-coordinate addition for the second store (issue #97). Tier 3 — coordination and cost-of-change across a repo boundary; the failure is loud, not silent, and D6's entry validation is the backstop that keeps it that way. -**Deliberately out of scope here:** the þing-01 redaction clause is already mechanically enforced (`tests/test_redaction_guard.py` — the delivery modules stay credential-blind and the provenance description is a closed keyset), and D2's ruling that **integration tests against the production Appwrite project are FORBIDDEN** (no non-production project exists) is a standing prohibition, not a drift risk. +**Deliberately out of scope here:** the þing-01 redaction clause is mechanically enforced (`tests/test_redaction_guard.py` — the delivery modules stay credential-blind and the provenance description is a closed keyset), and D2's ruling that **integration tests against the production Appwrite project are FORBIDDEN** (no non-production project exists) is a standing prohibition, not a drift risk. -Cross-refs: C-33 (store identity still hardcoded per store — the same env surface, different concern), C-58 (what happens when a coordinate is wrong rather than missing), C-44 (the pipeline-core version coupling that would carry a registry change), issues #134/#135/#138 (this repo's discharged þing-01 obligations), #104 (README env block placeholders). +**⚠ CORRECTED 2026-08-02.** The word *already* above overclaimed: **C-74** showed that guard scans one of its five declared roots, four having pointed at paths that #153 moved. The keyset half is enforced; the credential-blindness half is enforced over `delivery/` only until C-74 lands. + +Cross-refs: C-74 (the guard this paragraph vouched for), C-33 (store identity still hardcoded per store — the same env surface, different concern), C-58 (what happens when a coordinate is wrong rather than missing), C-44 (the pipeline-core version coupling that would carry a registry change), issues #134/#135/#138 (this repo's discharged þing-01 obligations), #104 (README env block placeholders). --- @@ -701,6 +703,38 @@ Cross-refs: **C-25** (whose resolution this corrects), **C-40** (the inherited p --- +### C-74: The þing-01 redaction guard scans four paths that stopped existing — it reports success while covering one root of five + +| Field | Value | +|-------|-------| +| ID | C-74 | +| Tier | 3 — **not a leak today**: the relocated modules were checked directly and are still credential-blind (zero hits for `os.environ`, `getenv`, `load_dotenv`, `API_KEY`, `credentials`). What is gone is the thing that would notice them ceasing to be. A security-adjacent control that cannot fail is a maintainability defect until the day it is a correctness one. | +| Source | `review-diff` (2026-08-02) — S1/#182 review; found while reading the redaction discipline the new ADR-008 test cites as "the wider rule" | +| Trigger | When any module under `contract/wire`, `contract/historical.py`, `contract/track_a_source.py` or `contract/frame_extraction.py` gains environment access — the #135 guard will not report it. Also fires on **the next package move**: `rglob` on a vanished root yields silence, not an error, so any future relocation narrows the scan again with no signal | +| Location | `tests/test_redaction_guard.py:25-31` (`_CREDENTIAL_BLIND`), `:34-39` (`_python_sources`) | + +The þing-01 delivery-log redaction audit (#135, orð_09 §3) certified five module trees as credential-blind and pinned that finding as a permanent guard. Epic #148's S5 (#153) then moved the machinery out from under `unfao/` into `contract/`. **Four of the five roots were never re-pointed:** + +| declared root | exists | files scanned | +|---|---|---| +| `unfao/wire` | no | 0 | +| `delivery` | yes | 6 | +| `unfao/historical.py` | no | 0 | +| `unfao/track_a_source.py` | no | 0 | +| `unfao/frame_extraction.py` | no | 0 | + +**The failure is silent by construction.** `_python_sources` branches `if path.is_file(): … else: path.rglob("*.py")`, and `rglob` on a **nonexistent** directory yields an empty iterator rather than raising. A missing root and a clean root are therefore indistinguishable to the test, which passes either way. The guard reports an audited fact as pinned while pinning roughly a third of it. + +**Why this is Cluster I and not merely a stale path.** The register's own text asserts the opposite. **C-57** says, in its "deliberately out of scope" paragraph, that *"the þing-01 redaction clause is **already mechanically enforced** (`tests/test_redaction_guard.py` — the delivery modules stay credential-blind…)"* — and **Cluster I's fix strategy cites this very file** as an example of the guard pattern working. Both were written in good faith and both are now overclaims. That is the cluster's disease reproducing inside the cluster's own prescription. + +It is also the **sixth** instance of the stale-claim class found in two days, after C-63 and C-47 (filed Open with the defect fixed), C-43/C-59/C-61 (filed Open with their stated closing conditions met), #158 (closed with the code half undone) and `tests/test_validation.py` (claiming fidelity to a method it no longer resembles — **C-03**). The common shape is not carelessness: it is that a *move* leaves prose and paths behind, and nothing in this repo asserts that a declared path exists. + +**Mitigation:** re-point the four roots at `contract/`, and add a root-existence assertion so a future relocation fails loudly instead of silently narrowing. The second half is the load-bearing one — re-pointing fixes today, asserting existence fixes the class. Tracked as a story under **epic #181**. + +Cross-refs: **C-57** (whose "already mechanically enforced" claim this falsifies — corrected in place), **C-03** (the same class in `test_validation.py`), **C-46** (a different guard that also does not run, by a different mechanism), **C-63**, **C-47**, **Cluster I**, #135, #153, #182. + +--- + ## Disagreements ### D-12: Post-Run-0 infrastructure & naming intents — repo rename, internal-store transport, compute co-location diff --git a/tests/test_env_declaration.py b/tests/test_env_declaration.py index 5ad57f7..8ba81f6 100644 --- a/tests/test_env_declaration.py +++ b/tests/test_env_declaration.py @@ -7,16 +7,20 @@ verdict retired it — the launcher declares its env sourcing (views-models M3, merged), and this package only validates (verdict D6). -Imports only `unfao.appwrite_env` (dependency-light by design); the manager -module itself needs views-pipeline-core, absent in test environments, so the -manager-side facts are pinned by source scan — the repo's standing pattern. +Imports `unfao.appwrite_env` and its sibling `contract.launch_config` — both are +dependency-light by design (no views-pipeline-core, no pandas), which is what +lets them be exercised directly here. The manager module is not: it needs +views-pipeline-core, absent in test environments, so the manager-side facts are +pinned by source scan instead — the repo's standing pattern. """ +import logging import re from pathlib import Path import pytest +from views_postprocessing.contract import launch_config from views_postprocessing.unfao import appwrite_env @@ -96,6 +100,89 @@ def test_declared_names_match_the_manager_reads(): assert text.count("appwrite_env.assert_env_declared(") == 2 +# ── ADR-008 across BOTH entry validators (S1 / #182, register C-71) ────────── +# +# `appwrite_env` asserts the launcher assembled the *environment*; `launch_config` +# asserts it declared the *delivery mode*. They are deliberate siblings, NOT a +# shared abstraction (WET before DRY — the trigger to extract is a THIRD such +# module, #181). Siblings drift: `launch_config` was written in #149 by mirroring +# `appwrite_env`, inherited its missing log call, was fixed in review, and left +# the module it copied as the odd one out for two weeks. +# +# Parametrising the ADR-008 obligation over both is what stops that recurring — +# and adding a third validator is one line here, not a new test. +_REFUSALS = ( + pytest.param( + lambda: appwrite_env.assert_env_declared(("VPP_S1_ABSENT_VAR",), store="test_store"), + EnvironmentError, + "VPP_S1_ABSENT_VAR", + id="appwrite_env.assert_env_declared", + ), + pytest.param( + lambda: launch_config.assert_contract_mode({}), + launch_config.LaunchConfigError, + launch_config.WIRE_CONTRACT_KEY, + id="launch_config.assert_contract_mode", + ), + pytest.param( + lambda: launch_config.assert_frame_native_historical("pandas_dataframe"), + launch_config.LaunchConfigError, + launch_config.FEATURE_FRAME_FORMAT, + id="launch_config.assert_frame_native_historical", + ), +) + + +@pytest.mark.parametrize("refuse,exc,expected_token", _REFUSALS) +def test_every_entry_validator_logs_before_it_raises(refuse, exc, expected_token, caplog): + """ADR-008:48/51 — a structural refusal must leave a persistent record. + + :48 requires raised structural failures to be logged at ERROR or higher; :51 + that *"raising is not a substitute for logging."* A launcher misconfiguration + is a structural failure by any reading, and these are the two seams whose + whole job is to make one visible. An operator reading logs after a refused + run must find the reason there, not only in a traceback they no longer have. + """ + with caplog.at_level(logging.ERROR): + with pytest.raises(exc): + refuse() + + errors = [r for r in caplog.records if r.levelno >= logging.ERROR] + assert len(errors) == 1, ( + f"expected exactly one ERROR record from the refusal, got {len(errors)}: " + f"{[r.getMessage() for r in errors]}" + ) + assert expected_token in errors[0].getMessage(), ( + "the log record must name what was missing — a record that says only " + "'refused' sends the operator back to the traceback it was meant to replace" + ) + + +def test_the_environment_refusal_logs_names_and_never_values(monkeypatch, caplog): + """The one place ADR-008 and the redaction discipline could collide. + + `CONNECTION_ENV` carries `APPWRITE_DATASTORE_API_KEY` — a secret slot. Logging + a refusal is only safe because `missing` holds NAMES: membership is decided by + `os.getenv(name)` being falsy and the resolved value is never read. This test + pins that, so a future "let's log the current environment for debuggability" + cannot land quietly. See tests/test_redaction_guard.py for the wider rule. + """ + sentinel = "s1-sentinel-secret-value-must-never-be-logged" + monkeypatch.setenv("APPWRITE_DATASTORE_API_KEY", sentinel) + monkeypatch.delenv("APPWRITE_ENDPOINT", raising=False) + + with caplog.at_level(logging.ERROR): + with pytest.raises(EnvironmentError): + appwrite_env.assert_env_declared(appwrite_env.CONNECTION_ENV, store="unfao_bucket") + + logged = "\n".join(r.getMessage() for r in caplog.records) + assert "APPWRITE_ENDPOINT" in logged, "the missing NAME must be reported" + assert sentinel not in logged, "a secret VALUE reached a log record" + assert "APPWRITE_DATASTORE_API_KEY" not in logged, ( + "the key was set, so it is not missing — it must not appear at all" + ) + + def test_secret_env_names_follow_the_platform_naming_rule(): # PLATFORM-001 D3: suffix _API_KEY/_PASSWORD/_TOKEN ⇒ secret. Exactly one # declared name is a secret; every other declared name is a coordinate. diff --git a/views_postprocessing/unfao/appwrite_env.py b/views_postprocessing/unfao/appwrite_env.py index 6e2724e..b60a28d 100644 --- a/views_postprocessing/unfao/appwrite_env.py +++ b/views_postprocessing/unfao/appwrite_env.py @@ -13,8 +13,11 @@ from __future__ import annotations +import logging import os +logger = logging.getLogger(__name__) + CONNECTION_ENV = ( "APPWRITE_ENDPOINT", "APPWRITE_DATASTORE_PROJECT_ID", @@ -40,12 +43,30 @@ def assert_env_declared(names: tuple, *, store: str) -> None: """Entry validation (þing-01 D6): every required name resolved and non-empty, - or raise naming ALL missing variables — never a partial config that half-works.""" + or raise naming ALL missing variables — never a partial config that half-works. + + Logs at ERROR before raising (ADR-008): the refusal must survive in the run's + log, not only in a traceback the operator no longer has. Names only — the + resolved value of a variable is never read, and never logged. + + Args: + names: the environment-variable names this store requires. + store: the store being configured, for the message. + + Raises: + EnvironmentError: naming every missing variable. + """ missing = [name for name in names if not os.getenv(name)] if missing: - raise EnvironmentError( + # `missing` holds NAMES, never values: membership is decided by + # `os.getenv(name)` being falsy, and the resolved value is never read. + # That is what makes this loggable at all — CONNECTION_ENV carries the + # APPWRITE_DATASTORE_API_KEY secret slot. + err_msg = ( f"{store}: the launcher did not assemble the required environment — " f"missing {missing}. Coordinates come from the PLATFORM-001 registry " "(views-models run.sh declares its sourcing); the secret is the " "operator slot. This package no longer loads any dotenv (#134)." ) + logger.error(err_msg) # ADR-008: logged persistently AND raised + raise EnvironmentError(err_msg) From 2284be407acd83a40f6eb13933f4896eb0105603 Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 2 Aug 2026 01:08:48 +0200 Subject: [PATCH 04/23] =?UTF-8?q?chore(register):=20S2=20#183=20=E2=80=94?= =?UTF-8?q?=20close=20Cluster=20K's=20three=20finished=20entries,=20and=20?= =?UTF-8?q?make=20closing=20conditions=20machine-checkable=20(#194)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit C-43, C-59 and C-61 were filed under Open with their own stated closing conditions already met. C-43's body said it plainly — "C-43 closes when tests/test_gaul_lookup_fidelity.py is committed and green" — and that file was committed in #141 with 18 tests passing. C-59 and C-61 both recorded "Mitigation — landed 2026-07-31" and named four guards that exist and pass. Nothing was outstanding. The entries simply never moved. All three moved to Resolved, each citing the test rather than the session, which is what C-43 instructed. Its transcription-fidelity-vs-assignment-correctness scope split is preserved verbatim: this closes faithful carriage of the producer's answer, NOT whether that answer is right. The latter is views-datafactory#387 and is not this repo's to close. The guard this adds is the point of the story. test_register_integrity.py could already catch a heading that SAYS resolved; it could not catch an entry saying open while the thing it waited for had arrived — which turned out to be this register's actual failure mode, six times in two days. Two new checks: - an Open entry naming a "closes when `tests/…`" artifact that exists - an Open entry recording "Mitigation — landed" without "Partial mitigation" Mutation-tested against the real 2026-08-01 register: both fail, naming 'C-43 -> tests/test_gaul_lookup_fidelity.py' and ['C-59','C-61']. Restored, all 10 pass. Deliberately narrow, and narrowed further in review. _ARTIFACT_ROOTS covers tests/ and scripts/ only. For an executable artifact, existence is close to the condition because the suite then runs it; for docs/ the two come apart entirely — an ADR file has existed since July while the section an entry waits on may never have been written. Including doc paths would fire on correctly-open entries, and the guard would be deleted rather than fixed. That is the failure this check exists to avoid, so it must not commit it. Cluster K carries a closure note: one test file discharged three entries exactly as its fix strategy predicted, and the cost was not the geography bug it was opened for — it was that the engineering took one session and the record took two more days. C-43's own residual said the forward-check needed attaching to something the interpreter runs; the entries then made the identical error one level up. A guarantee needs a check, including the register's own. Header 25 -> 22 open, 49 -> 52 resolved. Cluster M heading corrected to six. Co-authored-by: Claude Opus 5 (1M context) --- reports/technical_risk_register.md | 223 ++++++++++++++++------------- tests/test_register_integrity.py | 96 ++++++++++++- 2 files changed, 213 insertions(+), 106 deletions(-) diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 5ed255a..8e5a23e 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -6,8 +6,8 @@ | Owner | Dylan Pinheiro / PRIO MD&D Team | | Last Updated | 2026-08-02 | | Total Concerns | 74 | -| Open Concerns | 25 | -| Resolved Concerns | 49 | +| Open Concerns | 22 | +| Resolved Concerns | 52 | --- @@ -65,6 +65,14 @@ covered a single open entry (see Historical clusters below). **Fix strategy:** one test file — `tests/test_gaul_lookup_fidelity.py` — split into an always-on half (gid uniqueness, region-set equality, coordinate formula against a committed ground-truth sample, no nulls, no `-1` codes) and a `skipif`-gated half comparing all 7 GAUL columns against the datafactory sibling. Plus two one-line hardenings in `scripts/build_gaul_lookup.py`: assert index uniqueness, and convert the bare `assert`s to explicit raises. **Resolution scope:** Full for C-59/C-61/C-43's residual guard; partial for C-60 (needs the flat declared `lookup_version` key) and C-46 (needs the hardcoded path removed). +**✅ MOSTLY CLOSED 2026-08-02 (S2 / #183).** The prediction held: **one test file discharged three entries.** `tests/test_gaul_lookup_fidelity.py` (18 tests) closed **C-43**, **C-59** and **C-61** together, exactly as the fix strategy above said it would — the always-on half against the committed artifact, the `skipif` half against views-datafactory, plus the builder's bare `assert`s converted to `LookupBuildError` raises. + +**What the cluster actually cost, and it is not what the entries said.** All three were **fixed on 2026-07-31 and stayed filed under Open until 2026-08-02**, C-43 with its own closing condition written into its body and already met. The engineering took one session; the *record* took two more days and a direct question from the maintainer to correct. That asymmetry is the finding — not the geography bug the cluster was opened for, which never existed. + +**The lesson, and it generalises past this cluster.** C-43's residual said the forward-check *"was a one-off session result, not a standing guarantee"* — and the fix was to attach it to something the interpreter runs. The entries then reproduced the identical error one level up: they stated their closing conditions in prose and attached them to nothing. S2 therefore added a closing-condition check to `test_register_integrity.py`. **A guarantee needs a check, and that applies to the register's own guarantees too.** + +**Remaining in this cluster:** **C-60** (the flat declared `lookup_version` key — S5 / #186) and **C-46** (the hardcoded datafactory path — S7 / #188). Both unblocked, both in epic #181. + ### Cluster L: The won migration was never cleaned up **Root cause:** the frame-native contract path replaced the pandas path and **won** — run-0 delivered global-land on 2026-07-27 and FAO has been served from it since. The replaced path was deliberately kept behind a config fork "until run 0 proves the contract path live" (C-40) and was then never removed. Everything below is residue of that one omission, not independent defects. **Entries:** C-63 (the fork is silent), C-64 (an invariant nothing calls), C-65 (a seam four-fifths dead), C-66 (an object built and unused), C-68 (a private name holding the survivors together) — plus **#145** (the retired path's uploads discard their failure result) and **C-40**'s residual scope. @@ -78,7 +86,7 @@ covered a single open entry (see Historical clusters below). **The lesson worth carrying, and it is D-11's:** WET-before-DRY was applied *correctly* — the pandas and frame seams ran as deliberate siblings through the migration, and a premature abstraction would have outlived the implementation it existed to unify. What went wrong was not the duplication; it was that the removal condition (*"until run 0 proves the contract path live"*) was written down without a **named trigger to act on**, so the box expired on 2026-07-27 and nobody opened it. D-11 predicted exactly this. **A deferral needs an owner and a trigger, not just a reason.** -### Cluster M: Five open concerns, one upstream publish +### Cluster M: Six open concerns, one upstream publish **Root cause:** this repo pins `views-pipeline-core >=2.1.3,<3.0.0`, which resolves 2.3.0 from PyPI. Every fix and every removal below exists **only** on pipeline-core's unreleased 3.0.0. None is engineering work here; all five arrive together with one pin bump, and none can be taken before that bump. **Entries:** **C-44** (the bump itself, deliberately held), **C-62** (the transitive drag — 3.4 GB venv, 31 of 32 Dependabot alerts), **C-72** (the pyarrow CVE whose fix our ceiling excludes), **C-73** (the Tier-2 stale-run selection defect, fixed upstream in their #341), **C-58** (the Tier-2 auto-provision-instead-of-raise, fixed upstream in their #322/#331/#332), **C-07** (the undeclared `appwrite` dependency, whose transitive path their #345 withdraws). **Highest tier:** 2 (C-73) @@ -365,57 +373,6 @@ See also C-07/C-27/C-29 (pipeline-core coupling symptoms), C-39 (the dead-mapper --- -### C-43: ADR-011 enrichment swap shipped without its output-equivalence proof — and the proof is now unrecoverable - -| Field | Value | -|-------|-------| -| ID | C-43 | -| Tier | 2 | -| Source | `manual` (2026-06-26) — user-flagged rigor loss on accepting option A; verified against git history (`eba1df8` / PR #42) | -| Trigger | **This trigger has FIRED — see the 2026-07-31 update.** Forward-looking replacement: when FAO or faoapi reports geographic metadata that looks wrong for specific cells, **or** before the next global delivery — forward-check a sample of `land_gaul` assignments against views-datafactory's GAUL parquet. The protective pre-go-global gate this entry originally described has passed. | -| Location | `views_postprocessing/unfao/enrichment.py` (`GaulLookupEnricher`); `views_postprocessing/unfao/managers/unfao.py:129` (`_append_metadata`), `:147-172` (`_validate` — the 9-column NULL gate, checks presence not correctness); umbrella #20 / issues #21, #23, #24 (the baseline+diff procedure, now unrunnable); deleted in `eba1df8` (PR #42): `mapping.py` + both ADR-011 diff scripts | - -ADR-011 swapped FAO geo-enrichment from the runtime geopandas mapper to the GAUL lookup enricher (commit `65635b6`). The swap's own plan (umbrella #20) required an **output-equivalence proof** before trusting it in production: Stage 0 (#21) run the OLD mapper on real `africa_me_legacy` data to archive a ground-truth baseline; Stage 2 (#23) diff the new enricher against it with *"zero unexplained differences."* That proof was **never produced** — no `baseline_schema.md` or baseline parquet was ever committed — and on 2026-06-24 the old mapper **and both diff scripts** were deleted (`eba1df8`, PR #42, C-39). So the equivalence check is now **unrecoverable** short of `git revert`-ing the mapper back. - -The accepted path forward (**option A**) is a single smoke-test delivery: "the run is green and the output looks sane," which proves the path *runs*, not that it produces the *same / correct* values the trusted mapper did. The manager's `_validate` enforces only that the 9 GAUL columns are **non-null** — it does not check value correctness — so a latent bug in the lookup build or the merge-by-gid (wrong join key, stale `lookup_version`, gid misalignment) would ship **wrong-but-non-null** geographic metadata to FAO with **no error signal**. - -**Why not Tier 1:** the lookup is built from views-datafactory's authoritative area-majority GAUL parquets — the canonical *producer* source (D-07). The new path sources from the gold standard; the old mapper was the *less*-trusted path being retired (C-31, C-23). So the missing diff is a lost cross-check, not "unverified code," and the Stage-1 enricher unit tests + coverage guards (C-30/C-34) cover part of the build. **Why Tier 2:** the residual silent-wrong-value path is real, the null gate cannot catch it, the one guard that would have is gone for good, and the trigger (go-global to 64k cells) is concrete and imminent. - -**Mitigation if assurance is wanted before go-global** (cheaper than reverting the mapper): forward-check a sample of `land_gaul` cell assignments directly against the datafactory GAUL parquet, or add a lightweight value-level assertion into the enricher path (a forward check against the producer source — *not* a resurrection of the deleted old-mapper diff). - -**TRIGGER FIRED 2026-07-27 — the risk changed tense (review-rr 2026-07-31).** Run-0 delivered the first FAO global-land forecast: `region=land_gaul`, 64,742 cells, 28,356,996 historical rows, 108 arrow shards + sidecar + manifest committed to `unfao_bucket`. The go-global run this entry was written to warn about **has happened**, and it happened with **no value-level equivalence check** — exactly as predicted. The concern is therefore no longer "risk of shipping unverified enrichment" but **"unverified enrichment has shipped, at global scale, and the forward-check is outstanding."** - -This is the most important consequence of the run-0 cluster (Cluster H). Run-0 discharged the *availability* half of the go-global debt — the path runs, memory is bounded (C-32: 5.6 GB), coverage is proven (C-30: 64,742 correct). It discharged **none of the correctness half**, because proving the path *runs* at scale was never what C-43 asked for. **This entry now stands alone and un-gated**, with delivered data in the partner store and `_validate`'s null gate still checking presence rather than value. Tier held at 2: the lookup is still built from views-datafactory's authoritative area-majority parquets (the gold-standard producer), which is why this is a lost cross-check rather than unverified code. - -**Recommended action (unchanged, now overdue rather than pre-emptive):** forward-check a sample of delivered `land_gaul` cell assignments directly against the datafactory GAUL parquet — cheap, and it is the mitigation this entry proposed from the start. Folds naturally into #131 q1 (run-0 delivery-integrity verification). - ---- - -**TRANSCRIPTION FIDELITY DISCHARGED 2026-07-31 (`expert-code-review`) — the forward-check was run, offline, against committed artifacts. Four checks, zero mismatches:** - -| Link in the chain | Ground truth | Result | -|---|---|---| -| Coordinate formula (`gaul_schema.xcoord`/`ycoord`) | views-datafactory `data/raw/priogrid/shapefile/priogrid_cell.dbf` — **all 259,200 cells** | **max abs error 0.00e+00**, 0 mismatches | -| Lookup values, all 7 GAUL columns | the 7 `data/raw/gaul_admin/*.parquet` | **0 mismatches** across 64,742 cells | -| Lookup gid set + key uniqueness | `src/datafactory_query/land_gaul_pgids.json` | **exactly equal**; 64,742 unique of 64,742 | -| **The delivered run-0 sidecar** (`rusty_bucket_forecasting_20260727_095355__sidecar.parquet` — the real bytes on FAO's shelf) | the lookup | **0 mismatches** on all 9 columns; SHA-256 matches the manifest's declaration | - -The chain producer → lookup → delivered bytes is verified end to end. Note the leading hypothesis going in — that the gid→lat/lon formula might be flipped or off-by-one, producing wrong-but-non-null coordinates on *every* cell, invisible to every existing gate — was **falsified**: the formula is exact for the entire global grid. - -**SCOPE — what this does and does not prove** (the Kleppmann-vs-Nygard split in the 2026-07-31 review, adjudicated to *both, scoped*)**.** It proves **transcription fidelity**: this repo faithfully carries the producer's area-majority GAUL assignment through to the partner. It does **not** prove **assignment correctness** — if views-datafactory's area-majority join puts a cell in the wrong country, every check above still passes and FAO still receives a confidently wrong label. That is a separate concern belonging to **views-datafactory** — the degree-based (square-degree) area math its area-majority join uses, which distorts by up to ~2× at 60°N and could flip the winning polygon for high-latitude border cells now that the region is global. This distinction must survive retelling: C-43 was registered as *a lost old-vs-new cross-check inside this repo*, and that is what has been discharged. - -**⚠ CORRECTION, same day (2026-07-31).** This paragraph originally asserted the upstream half was *"a separate, already-registered concern (C-08, relocated to views-datafactory)."* **That was false and is corrected here.** Verified by direct inspection: views-datafactory's register carries 32 concerns and mentions "area-majority" nine times, but has **no entry** for the degree-based area calculation. C-08 was resolved *here* on 2026-06-24 with the note *"Tracked there, not here"* — and nobody ever opened it there. **The concern has been untracked platform-wide since that date**, and run-0 shipped the affected high-latitude cells to FAO on 2026-07-27. - -Filed upstream as **views-platform/views-datafactory#387** so it is tracked where the code and the geopandas toolchain actually live. This repo cannot verify it: the forward-check above confirms faithful *transcription* of the producer's answer and is structurally incapable of judging whether that answer is right. - -This is a textbook instance of **C-42**'s registered hazard (acting on a mis-stated cross-repo state) and of **Cluster I** — and it was reproduced *while writing the very paragraph describing it*. Concrete lesson for the "relocated" convention added to the Register Conventions this same day: **relocation is not complete until the destination issue or entry exists and is cited by number.** A relocation note naming only a repo is an assumption, not a handoff. - -**Residual (why this entry stays open):** the verification was a one-off session result, not a standing guarantee. Nothing in CI re-runs it, so a future lookup rebuild against a wrong or stale datafactory would ship silently exactly as before. **C-43 closes when `tests/test_gaul_lookup_fidelity.py` is committed and green** — the entry should then cite the test, not the session. Tracked as **Cluster K**; the same test discharges C-59 and C-61. - -See also C-03 (the sibling enrich→validate test-coverage gap), C-22 (no post-delivery correction/recall process — **now acute: the consequence path is live**), C-39 / C-31 / C-23 (the resolved mapper-deletion cluster this emerged from), C-30 (coverage — discharged by the same run that left this standing), C-32 / C-34 (RESOLVED — the go-global scale risks that fired cleanly), D-08 (the swap-to-lookup-first decision whose verification debt this is), #131 (run-0 delivery-integrity verification). - ---- - ### C-44: views-pipeline-core 3.0.0 dependency bump is pending and must not land until the platform runs on development across all repos | Field | Value | @@ -515,30 +472,6 @@ Cross-refs: C-57 (registry drift — the most likely way a coordinate goes wrong --- -### C-59: The GAUL lookup build asserts no key uniqueness — a duplicate gid silently inflates the legacy delivery - -| Field | Value | -|-------|-------| -| ID | C-59 | -| Tier | 3 — **recalibrated from 2 the same day, see the correction below.** A duplicated key would multiply rows through the legacy pandas merge invisibly to every gate, but reaching the artifact requires `--region all`: the production `--region land_gaul` path de-duplicates first. Unguarded fragility on a non-default code path, not present corruption. | -| Source | `expert-code-review` (2026-07-31) — Kleppmann lens; verified empirically in the same pass | -| Trigger | When views-datafactory regenerates the `gaul_admin` parquets, or when `build_gaul_lookup.py` is re-run against a new datafactory version — verify the resulting lookup index is unique before committing the artifact; nothing checks it today | -| Location | `scripts/build_gaul_lookup.py:146-154` (the invariant block, which checks nulls and `-1` but never uniqueness); consumed at `views_postprocessing/unfao/enrichment.py:117` (pandas left-merge — the inflating path), `unfao/historical.py:60` and `unfao/wire/sidecar.py:56` (deterministic-pick paths) | - -`build(...)` sets `df.index = df.index.astype("int64")`, names it `priogrid_gid`, sorts, and then asserts only that no nulls and no `-1` sentinels survive. It never asserts `df.index.is_unique`. The seven source parquets are joined via `pd.DataFrame({...})` over gid-indexed Series (`build_gaul_lookup.py:59-73`), so uniqueness is inherited from upstream data rather than enforced here. - -Downstream, `GaulLookupEnricher.enrich_dataframe_with_pg_info` does `base.merge(self._lookup, left_on=pg_id_col, right_index=True, how="left")`. A duplicated key produces **N rows per affected cell**. The delivery then carries more rows than cells, with every metadata value present and correct — invisible to the null gate, invisible to the distinct-cell coverage gate, and invisible to the `country_iso_a3` proxy at `enrichment.py:122`. - -**⚠ TIER RECALIBRATED 2 → 3, same day (2026-07-31), on empirical evidence.** Registering this at Tier 2 assumed a duplicate could reach the committed artifact through the normal build. Mutation-testing the builder showed it cannot, on the production path: `build()` applies `src.loc[src.index.intersection(sorted(region_gids))]` (`build_gaul_lookup.py:114-116`), and pandas' `Index.intersection` **de-duplicates**, so an injected duplicate is silently removed before the invariant block ever sees it. The guard is reachable only with `--region all`, which bypasses that filter — verified: it raises there, and raises under `python -O` too. - -Two consequences, both kept: the explicit raise still earns its place, because the de-duplication is an *accidental pandas behaviour* rather than a declared guard (and silently absorbing upstream duplication is itself undesirable — it hides a producer defect); and the tier drops to 3, because the realistic exposure is a non-default flag, not routine regeneration. *Recalibrated during the same session that registered it — the original Tier 2 rationale was written from code reading before the mutation test was run.* - -**Mitigation — landed 2026-07-31:** explicit `LookupBuildError` on a non-unique index in `build_gaul_lookup.py` (not `assert`, per C-61), pinned by `tests/test_gaul_lookup_fidelity.py::test_builder_rejects_a_duplicate_gid`, plus a standing uniqueness check on the committed artifact (`test_lookup_key_is_unique`). - -Cross-refs: C-43 (the value-correctness debt this shares a fix with), C-61 (the same invariant block's strippable asserts), C-30 (the distinct-cell coverage gate that cannot see this), C-40 (the legacy pandas path whose deletion would remove the inflating consumer), **Cluster K**. - ---- - ### C-60: The lookup provenance stamp reaches into the producer's ledger schema and degrades to `"unknown"` on a bare except | Field | Value | @@ -559,30 +492,6 @@ Cross-refs: C-15 (the provenance this field serves), C-22 (the recall process th --- -### C-61: The lookup build's hard invariants are bare `assert`s — stripped under `python -O`, and `-1` sentinels are caught nowhere else - -| Field | Value | -|-------|-------| -| ID | C-61 | -| Tier | 3 | -| Source | `expert-code-review` (2026-07-31) — Feathers lens | -| Trigger | When `build_gaul_lookup.py` is run under `python -O` (or from a wheel/CI step that sets `PYTHONOPTIMIZE`), or when the build is wrapped in any tooling that optimizes bytecode — verify the invariant block still executed; a stripped run writes an unvalidated lookup that looks identical | -| Location | `scripts/build_gaul_lookup.py:152-154` (`assert df.isna().sum().sum() == 0`, `assert (df[c] != -1).all()`) | - -The builder's docstring and the enricher both rely on the lookup being "clean by construction" — no nulls, no `-1` sentinels. That guarantee is enforced by three bare `assert` statements, which Python removes entirely under `-O`. - -The asymmetry the original registration leaned on: **nulls have a downstream backstop** (`_validate`, `historical.assert_metadata_complete`) but **`-1` codes have none** — `-1` is non-null, so it would pass every delivery gate, which is the resolved **C-35** defect (invalid country codes shipped to FAO for Somaliland cells) returning through a different door. - -**⚠ EXPOSURE CORRECTED, same day (2026-07-31), on empirical evidence.** That framing overstated the risk. Mutation-testing the builder showed the **primary protection against `-1` is not the `assert` at all** — it is the completeness filter at `build_gaul_lookup.py:125-131` (`complete &= df[c].notna() & (df[c] != -1)`), which drops sentinel rows outright. That filter is **plain code, untouched by `python -O`**, so the strippable-assert exposure never applied to the `-1` case. In practice the `assert` was unreachable: no ordinary input can get past the filter to reach it. - -What remains true, and why the entry stays open at Tier 3: the invariant block was the only *explicit statement* of "this artifact is clean," it was strippable, and the same block also carried the null and (now) uniqueness checks where the argument does bite. Stating invariants in a form the interpreter can delete is the defect; the `-1` severity was not. - -**Mitigation — landed 2026-07-31:** all three invariants converted from bare `assert` to explicit `LookupBuildError` raises with diagnostic messages. The `-1` raise is retained deliberately as a backstop should the filter ever change, and is **deliberately left untested** — reaching it requires stubbing pandas internals, and a test that fragile is worse than the invariant it guards. What *is* pinned is the behaviour that actually protects the partner: `tests/test_gaul_lookup_fidelity.py::test_a_sentinel_code_is_dropped_rather_than_shipped` (the cell is excluded, so it later fails loud as *absent* rather than shipping as wrong-but-non-null) and `test_lookup_carries_no_sentinel_codes` on the committed artifact. - -Cross-refs: C-35 (RESOLVED — the `-1` defect class this guards against), C-59 (same invariant block), C-43 (the fidelity test that would catch a bad artifact regardless), **Cluster K**. - ---- - ### C-62: The pinned pipeline-core release still installs geopandas and torch into a repo that architecturally excised them | Field | Value | @@ -802,6 +711,116 @@ See also C-40 (the inheritance/representation coupling this migration unwinds), --- ## Resolved Concerns +### C-43: ADR-011 enrichment swap shipped without its output-equivalence proof — and the proof is now unrecoverable — RESOLVED + +| Field | Value | +|-------|-------| +| ID | C-43 | +| Resolved | 2026-08-02 | +| Resolution | **Closed by `tests/test_gaul_lookup_fidelity.py` (18 tests, committed in #141) — which is precisely what this entry named as its own closing condition:** *"C-43 closes when `tests/test_gaul_lookup_fidelity.py` is committed and green — the entry should then cite the test, not the session."* It now cites the test. Verified 2026-08-02: `pytest -q tests/test_gaul_lookup_fidelity.py` → **18 passed**. The residual this entry stayed open over — that the forward-check was a one-off session result rather than a standing guarantee — is discharged: the always-on half pins gid uniqueness, region-set equality, the coordinate formula and the absence of nulls and `-1` sentinels against the committed artifact, and the datafactory-gated half compares all seven GAUL columns against the producer's parquets. **Scope is unchanged and load-bearing — see the SCOPE paragraph below.** This closes **transcription fidelity**, not **assignment correctness**; the latter is views-datafactory#387 and is not this repo's to close. The entry sat under Open with its own stated condition met — the drift class that prompted S2 (#183) to make closing conditions machine-checkable. **Cluster K.** | +| Tier | 2 | +| Source | `manual` (2026-06-26) — user-flagged rigor loss on accepting option A; verified against git history (`eba1df8` / PR #42) | +| Trigger | **This trigger has FIRED — see the 2026-07-31 update.** Forward-looking replacement: when FAO or faoapi reports geographic metadata that looks wrong for specific cells, **or** before the next global delivery — forward-check a sample of `land_gaul` assignments against views-datafactory's GAUL parquet. The protective pre-go-global gate this entry originally described has passed. | +| Location | `views_postprocessing/unfao/enrichment.py` (`GaulLookupEnricher`); `views_postprocessing/unfao/managers/unfao.py:129` (`_append_metadata`), `:147-172` (`_validate` — the 9-column NULL gate, checks presence not correctness); umbrella #20 / issues #21, #23, #24 (the baseline+diff procedure, now unrunnable); deleted in `eba1df8` (PR #42): `mapping.py` + both ADR-011 diff scripts | + +ADR-011 swapped FAO geo-enrichment from the runtime geopandas mapper to the GAUL lookup enricher (commit `65635b6`). The swap's own plan (umbrella #20) required an **output-equivalence proof** before trusting it in production: Stage 0 (#21) run the OLD mapper on real `africa_me_legacy` data to archive a ground-truth baseline; Stage 2 (#23) diff the new enricher against it with *"zero unexplained differences."* That proof was **never produced** — no `baseline_schema.md` or baseline parquet was ever committed — and on 2026-06-24 the old mapper **and both diff scripts** were deleted (`eba1df8`, PR #42, C-39). So the equivalence check is now **unrecoverable** short of `git revert`-ing the mapper back. + +The accepted path forward (**option A**) is a single smoke-test delivery: "the run is green and the output looks sane," which proves the path *runs*, not that it produces the *same / correct* values the trusted mapper did. The manager's `_validate` enforces only that the 9 GAUL columns are **non-null** — it does not check value correctness — so a latent bug in the lookup build or the merge-by-gid (wrong join key, stale `lookup_version`, gid misalignment) would ship **wrong-but-non-null** geographic metadata to FAO with **no error signal**. + +**Why not Tier 1:** the lookup is built from views-datafactory's authoritative area-majority GAUL parquets — the canonical *producer* source (D-07). The new path sources from the gold standard; the old mapper was the *less*-trusted path being retired (C-31, C-23). So the missing diff is a lost cross-check, not "unverified code," and the Stage-1 enricher unit tests + coverage guards (C-30/C-34) cover part of the build. **Why Tier 2:** the residual silent-wrong-value path is real, the null gate cannot catch it, the one guard that would have is gone for good, and the trigger (go-global to 64k cells) is concrete and imminent. + +**Mitigation if assurance is wanted before go-global** (cheaper than reverting the mapper): forward-check a sample of `land_gaul` cell assignments directly against the datafactory GAUL parquet, or add a lightweight value-level assertion into the enricher path (a forward check against the producer source — *not* a resurrection of the deleted old-mapper diff). + +**TRIGGER FIRED 2026-07-27 — the risk changed tense (review-rr 2026-07-31).** Run-0 delivered the first FAO global-land forecast: `region=land_gaul`, 64,742 cells, 28,356,996 historical rows, 108 arrow shards + sidecar + manifest committed to `unfao_bucket`. The go-global run this entry was written to warn about **has happened**, and it happened with **no value-level equivalence check** — exactly as predicted. The concern is therefore no longer "risk of shipping unverified enrichment" but **"unverified enrichment has shipped, at global scale, and the forward-check is outstanding."** + +This is the most important consequence of the run-0 cluster (Cluster H). Run-0 discharged the *availability* half of the go-global debt — the path runs, memory is bounded (C-32: 5.6 GB), coverage is proven (C-30: 64,742 correct). It discharged **none of the correctness half**, because proving the path *runs* at scale was never what C-43 asked for. **This entry now stands alone and un-gated**, with delivered data in the partner store and `_validate`'s null gate still checking presence rather than value. Tier held at 2: the lookup is still built from views-datafactory's authoritative area-majority parquets (the gold-standard producer), which is why this is a lost cross-check rather than unverified code. + +**Recommended action (unchanged, now overdue rather than pre-emptive):** forward-check a sample of delivered `land_gaul` cell assignments directly against the datafactory GAUL parquet — cheap, and it is the mitigation this entry proposed from the start. Folds naturally into #131 q1 (run-0 delivery-integrity verification). + +--- + +**TRANSCRIPTION FIDELITY DISCHARGED 2026-07-31 (`expert-code-review`) — the forward-check was run, offline, against committed artifacts. Four checks, zero mismatches:** + +| Link in the chain | Ground truth | Result | +|---|---|---| +| Coordinate formula (`gaul_schema.xcoord`/`ycoord`) | views-datafactory `data/raw/priogrid/shapefile/priogrid_cell.dbf` — **all 259,200 cells** | **max abs error 0.00e+00**, 0 mismatches | +| Lookup values, all 7 GAUL columns | the 7 `data/raw/gaul_admin/*.parquet` | **0 mismatches** across 64,742 cells | +| Lookup gid set + key uniqueness | `src/datafactory_query/land_gaul_pgids.json` | **exactly equal**; 64,742 unique of 64,742 | +| **The delivered run-0 sidecar** (`rusty_bucket_forecasting_20260727_095355__sidecar.parquet` — the real bytes on FAO's shelf) | the lookup | **0 mismatches** on all 9 columns; SHA-256 matches the manifest's declaration | + +The chain producer → lookup → delivered bytes is verified end to end. Note the leading hypothesis going in — that the gid→lat/lon formula might be flipped or off-by-one, producing wrong-but-non-null coordinates on *every* cell, invisible to every existing gate — was **falsified**: the formula is exact for the entire global grid. + +**SCOPE — what this does and does not prove** (the Kleppmann-vs-Nygard split in the 2026-07-31 review, adjudicated to *both, scoped*)**.** It proves **transcription fidelity**: this repo faithfully carries the producer's area-majority GAUL assignment through to the partner. It does **not** prove **assignment correctness** — if views-datafactory's area-majority join puts a cell in the wrong country, every check above still passes and FAO still receives a confidently wrong label. That is a separate concern belonging to **views-datafactory** — the degree-based (square-degree) area math its area-majority join uses, which distorts by up to ~2× at 60°N and could flip the winning polygon for high-latitude border cells now that the region is global. This distinction must survive retelling: C-43 was registered as *a lost old-vs-new cross-check inside this repo*, and that is what has been discharged. + +**⚠ CORRECTION, same day (2026-07-31).** This paragraph originally asserted the upstream half was *"a separate, already-registered concern (C-08, relocated to views-datafactory)."* **That was false and is corrected here.** Verified by direct inspection: views-datafactory's register carries 32 concerns and mentions "area-majority" nine times, but has **no entry** for the degree-based area calculation. C-08 was resolved *here* on 2026-06-24 with the note *"Tracked there, not here"* — and nobody ever opened it there. **The concern has been untracked platform-wide since that date**, and run-0 shipped the affected high-latitude cells to FAO on 2026-07-27. + +Filed upstream as **views-platform/views-datafactory#387** so it is tracked where the code and the geopandas toolchain actually live. This repo cannot verify it: the forward-check above confirms faithful *transcription* of the producer's answer and is structurally incapable of judging whether that answer is right. + +This is a textbook instance of **C-42**'s registered hazard (acting on a mis-stated cross-repo state) and of **Cluster I** — and it was reproduced *while writing the very paragraph describing it*. Concrete lesson for the "relocated" convention added to the Register Conventions this same day: **relocation is not complete until the destination issue or entry exists and is cited by number.** A relocation note naming only a repo is an assumption, not a handoff. + +**Residual (why this entry stays open):** the verification was a one-off session result, not a standing guarantee. Nothing in CI re-runs it, so a future lookup rebuild against a wrong or stale datafactory would ship silently exactly as before. **C-43 closes when `tests/test_gaul_lookup_fidelity.py` is committed and green** — the entry should then cite the test, not the session. Tracked as **Cluster K**; the same test discharges C-59 and C-61. + +See also C-03 (the sibling enrich→validate test-coverage gap), C-22 (no post-delivery correction/recall process — **now acute: the consequence path is live**), C-39 / C-31 / C-23 (the resolved mapper-deletion cluster this emerged from), C-30 (coverage — discharged by the same run that left this standing), C-32 / C-34 (RESOLVED — the go-global scale risks that fired cleanly), D-08 (the swap-to-lookup-first decision whose verification debt this is), #131 (run-0 delivery-integrity verification). + +--- + +--- + +### C-59: The GAUL lookup build asserts no key uniqueness — a duplicate gid silently inflates the legacy delivery — RESOLVED + +| Field | Value | +|-------|-------| +| ID | C-59 | +| Resolved | 2026-08-02 | +| Resolution | **Closed by the mitigation this entry already recorded as landed; nothing was outstanding, the entry simply never moved.** Verified 2026-08-02: `scripts/build_gaul_lookup.py` carries **zero** bare `assert` statements and raises an explicit `LookupBuildError` on a non-unique index, and both named guards exist and pass — `tests/test_gaul_lookup_fidelity.py::test_builder_rejects_a_duplicate_gid` (the builder refuses a duplicate) and `::test_lookup_key_is_unique` (the committed artifact is clean). The Tier 2→3 recalibration recorded below stands: the production `--region land_gaul` path de-duplicates before the guard is reached, so the explicit raise protects the `--region all` path and refuses to absorb a producer defect silently. **Cluster K.** | +| Tier | 3 — **recalibrated from 2 the same day, see the correction below.** A duplicated key would multiply rows through the legacy pandas merge invisibly to every gate, but reaching the artifact requires `--region all`: the production `--region land_gaul` path de-duplicates first. Unguarded fragility on a non-default code path, not present corruption. | +| Source | `expert-code-review` (2026-07-31) — Kleppmann lens; verified empirically in the same pass | +| Trigger | When views-datafactory regenerates the `gaul_admin` parquets, or when `build_gaul_lookup.py` is re-run against a new datafactory version — verify the resulting lookup index is unique before committing the artifact; nothing checks it today | +| Location | `scripts/build_gaul_lookup.py:146-154` (the invariant block, which checks nulls and `-1` but never uniqueness); consumed at `views_postprocessing/unfao/enrichment.py:117` (pandas left-merge — the inflating path), `unfao/historical.py:60` and `unfao/wire/sidecar.py:56` (deterministic-pick paths) | + +`build(...)` sets `df.index = df.index.astype("int64")`, names it `priogrid_gid`, sorts, and then asserts only that no nulls and no `-1` sentinels survive. It never asserts `df.index.is_unique`. The seven source parquets are joined via `pd.DataFrame({...})` over gid-indexed Series (`build_gaul_lookup.py:59-73`), so uniqueness is inherited from upstream data rather than enforced here. + +Downstream, `GaulLookupEnricher.enrich_dataframe_with_pg_info` does `base.merge(self._lookup, left_on=pg_id_col, right_index=True, how="left")`. A duplicated key produces **N rows per affected cell**. The delivery then carries more rows than cells, with every metadata value present and correct — invisible to the null gate, invisible to the distinct-cell coverage gate, and invisible to the `country_iso_a3` proxy at `enrichment.py:122`. + +**⚠ TIER RECALIBRATED 2 → 3, same day (2026-07-31), on empirical evidence.** Registering this at Tier 2 assumed a duplicate could reach the committed artifact through the normal build. Mutation-testing the builder showed it cannot, on the production path: `build()` applies `src.loc[src.index.intersection(sorted(region_gids))]` (`build_gaul_lookup.py:114-116`), and pandas' `Index.intersection` **de-duplicates**, so an injected duplicate is silently removed before the invariant block ever sees it. The guard is reachable only with `--region all`, which bypasses that filter — verified: it raises there, and raises under `python -O` too. + +Two consequences, both kept: the explicit raise still earns its place, because the de-duplication is an *accidental pandas behaviour* rather than a declared guard (and silently absorbing upstream duplication is itself undesirable — it hides a producer defect); and the tier drops to 3, because the realistic exposure is a non-default flag, not routine regeneration. *Recalibrated during the same session that registered it — the original Tier 2 rationale was written from code reading before the mutation test was run.* + +**Mitigation — landed 2026-07-31:** explicit `LookupBuildError` on a non-unique index in `build_gaul_lookup.py` (not `assert`, per C-61), pinned by `tests/test_gaul_lookup_fidelity.py::test_builder_rejects_a_duplicate_gid`, plus a standing uniqueness check on the committed artifact (`test_lookup_key_is_unique`). + +Cross-refs: C-43 (the value-correctness debt this shares a fix with), C-61 (the same invariant block's strippable asserts), C-30 (the distinct-cell coverage gate that cannot see this), C-40 (the legacy pandas path whose deletion would remove the inflating consumer), **Cluster K**. + +--- + +--- + +### C-61: The lookup build's hard invariants are bare `assert`s — stripped under `python -O`, and `-1` sentinels are caught nowhere else — RESOLVED + +| Field | Value | +|-------|-------| +| ID | C-61 | +| Resolved | 2026-08-02 | +| Resolution | **Closed by the mitigation this entry already recorded as landed; nothing was outstanding, the entry simply never moved.** Verified 2026-08-02: all three invariants in `scripts/build_gaul_lookup.py` are explicit `LookupBuildError` raises, which `python -O` cannot strip, and both named guards pass — `tests/test_gaul_lookup_fidelity.py::test_a_sentinel_code_is_dropped_rather_than_shipped` (the cell is excluded, so it later fails loud as *absent* rather than shipping wrong-but-non-null) and `::test_lookup_carries_no_sentinel_codes` (the committed artifact). The deliberate non-test of the `-1` raise stands as recorded: reaching it requires stubbing pandas internals, and a test that fragile is worse than the invariant it guards. **Cluster K.** | +| Tier | 3 | +| Source | `expert-code-review` (2026-07-31) — Feathers lens | +| Trigger | When `build_gaul_lookup.py` is run under `python -O` (or from a wheel/CI step that sets `PYTHONOPTIMIZE`), or when the build is wrapped in any tooling that optimizes bytecode — verify the invariant block still executed; a stripped run writes an unvalidated lookup that looks identical | +| Location | `scripts/build_gaul_lookup.py:152-154` (`assert df.isna().sum().sum() == 0`, `assert (df[c] != -1).all()`) | + +The builder's docstring and the enricher both rely on the lookup being "clean by construction" — no nulls, no `-1` sentinels. That guarantee is enforced by three bare `assert` statements, which Python removes entirely under `-O`. + +The asymmetry the original registration leaned on: **nulls have a downstream backstop** (`_validate`, `historical.assert_metadata_complete`) but **`-1` codes have none** — `-1` is non-null, so it would pass every delivery gate, which is the resolved **C-35** defect (invalid country codes shipped to FAO for Somaliland cells) returning through a different door. + +**⚠ EXPOSURE CORRECTED, same day (2026-07-31), on empirical evidence.** That framing overstated the risk. Mutation-testing the builder showed the **primary protection against `-1` is not the `assert` at all** — it is the completeness filter at `build_gaul_lookup.py:125-131` (`complete &= df[c].notna() & (df[c] != -1)`), which drops sentinel rows outright. That filter is **plain code, untouched by `python -O`**, so the strippable-assert exposure never applied to the `-1` case. In practice the `assert` was unreachable: no ordinary input can get past the filter to reach it. + +What remains true, and why the entry stays open at Tier 3: the invariant block was the only *explicit statement* of "this artifact is clean," it was strippable, and the same block also carried the null and (now) uniqueness checks where the argument does bite. Stating invariants in a form the interpreter can delete is the defect; the `-1` severity was not. + +**Mitigation — landed 2026-07-31:** all three invariants converted from bare `assert` to explicit `LookupBuildError` raises with diagnostic messages. The `-1` raise is retained deliberately as a backstop should the filter ever change, and is **deliberately left untested** — reaching it requires stubbing pandas internals, and a test that fragile is worse than the invariant it guards. What *is* pinned is the behaviour that actually protects the partner: `tests/test_gaul_lookup_fidelity.py::test_a_sentinel_code_is_dropped_rather_than_shipped` (the cell is excluded, so it later fails loud as *absent* rather than shipping as wrong-but-non-null) and `test_lookup_carries_no_sentinel_codes` on the committed artifact. + +Cross-refs: C-35 (RESOLVED — the `-1` defect class this guards against), C-59 (same invariant block), C-43 (the fidelity test that would catch a bad artifact regardless), **Cluster K**. + +--- + +--- ### C-47: Stale untracked `reconciliation/__pycache__/` survives the module's retirement and misrepresents the package tree `[backlog]` — RESOLVED diff --git a/tests/test_register_integrity.py b/tests/test_register_integrity.py index 2d94304..bdde1ad 100644 --- a/tests/test_register_integrity.py +++ b/tests/test_register_integrity.py @@ -65,6 +65,15 @@ def _headings(body: str) -> list[str]: return re.findall(r"^### ([CD]-\d+:.*)$", body, flags=re.M) +def _open_blocks(register: str) -> list[tuple[str, str]]: + """[(entry id, entry body)] for every concern under ``## Open Concerns``.""" + body = dict(_sections(register))["Open Concerns"] + return [ + (block.split(":", 1)[0].strip(), block) + for block in re.split(r"^### ", body, flags=re.M)[1:] + ] + + def _header_counts(text: str) -> dict[str, int]: return { field: int(value) @@ -137,11 +146,8 @@ def test_concerns_and_disagreements_are_not_filed_under_each_other(register): def test_every_concern_entry_declares_the_required_fields(register): # Open concerns carry Tier/Source/Trigger/Location; resolved ones may instead # carry Resolved/Resolution. Either shape is fine — silence is not. - sections = dict(_sections(register)) - blocks = re.split(r"^### ", sections["Open Concerns"], flags=re.M)[1:] missing = [] - for block in blocks: - entry_id = block.split(":", 1)[0].strip() + for entry_id, block in _open_blocks(register): for field in ("ID", "Tier", "Source", "Trigger", "Location"): if not re.search(rf"^\| {field} \|", block, flags=re.M): missing.append(f"{entry_id}: {field}") @@ -180,6 +186,88 @@ def test_internal_references_resolve_and_foreign_ones_are_namespaced(register): ) +# ── Closing conditions must not already be met (S2 / #183) ─────────────────── +# +# The guards above catch a heading that SAYS it is resolved. They cannot catch an +# entry that says it is open while the thing it is waiting for has arrived — and +# that turned out to be this register's actual failure mode. Six instances in two +# days: C-63 and C-47 (open, defect fixed), C-43, C-59 and C-61 (open, each +# naming a test that existed and passed), and C-74's cousin in `test_validation.py`. +# +# The stale-OPEN direction is the safe one — nothing is claimed fixed that is not — +# but it inflates the open set with finished work, so prioritisation lies. C-43 in +# particular carried its own closing condition in its body and sat under Open with +# it met, which is what makes this mechanisable at all: the entry told us what to +# check, in writing. +# +# Deliberately narrow. Two phrasings, both unambiguous past-tense claims, and the +# path-gated one only fires when the named file actually EXISTS. A guard that +# cries wolf gets deleted — this repo already deleted one over-fragile test this +# week for exactly that reason (C-61's `-1` monkeypatch). False negatives are the +# accepted cost. +_CLOSING_CONDITION = re.compile(r"closes when\b", re.I) + +#: Exact strings, and that is the point: these are CONVENTIONS the register writes +#: and this guard enforces by matching them. "Mitigation - landed" (hyphen) slips +#: through, deliberately — see the false-negatives note above. +_LANDED = "Mitigation — landed" +_PARTIAL = "Partial mitigation" + +#: A backticked token is treated as a closing artifact when it looks repo-relative. +#: +#: **Executable roots only.** For a test or a script, "the file exists" is close to +#: what a closing condition means, because the suite then RUNS it — C-43's condition +#: was "committed and green", and the 18 tests are what make the second half true. +#: For `docs/` the two come apart completely: an ADR file has existed since July, +#: while the section an entry is waiting on may never have been written. Including +#: doc paths here would fire on correctly-open entries and the guard would be +#: deleted rather than fixed, which is the failure this whole check exists to avoid. +_ARTIFACT_ROOTS = ("tests/", "scripts/") + + +def test_no_open_entry_names_a_closing_artifact_that_already_exists(register): + """An entry that says "closes when `tests/x.py` is committed" and finds it there. + + C-43's own words: *"C-43 closes when tests/test_gaul_lookup_fidelity.py is + committed and green."* The file was committed, the 18 tests passed, and the + entry stayed under Open for two days. Nothing objected, because nothing was + asked to. + """ + repo = _REGISTER.parent.parent + satisfied = [] + for entry_id, block in _open_blocks(register): + for match in _CLOSING_CONDITION.finditer(block): + window = block[match.end() : match.end() + 300] + for token in re.findall(r"`([^`]+)`", window): + path = token.split("::", 1)[0].strip() + if path.startswith(_ARTIFACT_ROOTS) and (repo / path).exists(): + satisfied.append(f"{entry_id} -> {path}") + assert not satisfied, ( + "open concern(s) whose own stated closing condition is already met — the " + "named artifact exists. Move the entry to Resolved citing it, or reword " + f"the condition to say what is actually outstanding: {sorted(set(satisfied))}" + ) + + +def test_no_open_entry_claims_its_mitigation_has_landed(register): + """A recorded landed mitigation is a past-tense claim that the fix shipped. + + C-59 and C-61 both carried it, both naming tests that existed and passed, and + both sat under Open. If a fix genuinely landed only in part, say so — the + escape hatch is the phrase "Partial mitigation", which is itself a declaration + rather than an ambiguity. + """ + claimed = [ + entry_id + for entry_id, block in _open_blocks(register) + if _LANDED in block and _PARTIAL not in block + ] + assert not claimed, ( + f"open concern(s) recording a landed mitigation: {claimed}. Either move the " + f"entry to Resolved, or write {_PARTIAL!r} and state what remains." + ) + + def test_the_register_is_dated_and_governed(register): assert re.search(r"\| Last Updated\s+\| \d{4}-\d{2}-\d{2}", register), ( "the header must carry an ISO Last Updated date" From 4e3100990e0ada33bea0848b682d53e36da8e22c Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 2 Aug 2026 02:17:27 +0200 Subject: [PATCH 05/23] =?UTF-8?q?docs(seam):=20S3=20#184=20=E2=80=94=20fin?= =?UTF-8?q?ish=20#158's=20rename,=20fix=20the=20URL=20it=20broke,=20and=20?= =?UTF-8?q?ban=20the=20retired=20name=20(#195)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #158 renamed the cross-repo contract from PLATFORM-001 to "the Appwrite Seam Contract" and was closed on 2026-08-01. Only the documentation half landed, and the half that did land was applied as a blanket find-and-replace. Four citations survived in .py. The one that mattered was inside the message raised by appwrite_env.assert_env_declared — the text an operator reads when a delivery refuses to launch, naming a retired contract in a repo they do not own, on the day their run failed. Worse, the find-and-replace rewrote the FILENAME INSIDE A URL. ADR-013 §7d pointed at ".../platform/the Appwrite Seam Contract_identity_secrets_ configuration_contract.md" — a path containing spaces, which has never resolved. The upstream file has since been renamed to appwrite_seam_contract.md independently. Both §7d links now point at views-appwrite b54928f rather than main. §7d claimed "referenced by URL, never by copy", which was true in letter and false in effect while an unpinned link could drift under it — and it did: the registry moved 1.3.0 to 1.4.0 on 2026-08-02, the same day. Recorded as Erratum E2 in the post-adoption record; contract_version stays 1.5 and the wire is untouched. The ban test is what stops this returning. Scoped to code — the package, the tests, AND scripts/ — because docs/ADRs and reports/ legitimately name the old contract when narrating what was decided under it. An ADR describing a 2026-07 decision correctly says what the thing was called in 2026-07; code speaks in the present tense to whoever reads it now. Review widened that scan from two roots to three. Two roots would have been register C-74's defect committed again, ninety minutes after registering it: a guard whose declared scope exceeds its actual scan, passing because the unscanned part happens to be clean. Mutation-tested at both widths. The ban also caught a parenthetical I had just written into appwrite_env.py's docstring naming the retired name to explain its retirement. Removed; that explanation belongs in the ADR, which now carries it. Co-authored-by: Claude Opus 5 (1M context) --- .../013_sampled_forecast_wire_contract.md | 31 +++++++++++-- tests/test_doc_accuracy.py | 44 +++++++++++++++++++ tests/test_env_declaration.py | 5 ++- tests/test_redaction_guard.py | 2 +- views_postprocessing/unfao/appwrite_env.py | 20 ++++++--- 5 files changed, 90 insertions(+), 12 deletions(-) diff --git a/docs/ADRs/013_sampled_forecast_wire_contract.md b/docs/ADRs/013_sampled_forecast_wire_contract.md index 7d2cf02..2ee920a 100644 --- a/docs/ADRs/013_sampled_forecast_wire_contract.md +++ b/docs/ADRs/013_sampled_forecast_wire_contract.md @@ -689,14 +689,23 @@ a "simplification." this contract governs the **data** crossing the Appwrite seam; the **identity/config** crossing the same seam (which key, which coordinates, sourced from where, named how) is governed by - **[the Appwrite Seam Contract — Identity, Secrets & Configuration Contract (VIEWS Appwrite seam)](https://github.com/views-platform/views-appwrite/blob/main/docs/ADRs/platform/the Appwrite Seam Contract_identity_secrets_configuration_contract.md)**, - homed in views-appwrite and referenced here **by URL, never by copy** — with its - coordinate registry - ([`coordinate_registry.toml`](https://github.com/views-platform/views-appwrite/blob/main/docs/ADRs/platform/coordinate_registry.toml)) + **[the Appwrite Seam Contract](https://github.com/views-platform/views-appwrite/blob/b54928f/docs/ADRs/platform/appwrite_seam_contract.md)**, + homed in views-appwrite and referenced here **by pinned URL, never by copy** — + with its coordinate registry + ([`coordinate_registry.toml`](https://github.com/views-platform/views-appwrite/blob/b54928f/docs/ADRs/platform/coordinate_registry.toml), v1.4.0) as the canonical source of the non-secret coordinates this repo's runtime resolves. This repo's declared environment (`unfao/appwrite_env.py`, fail-loud entry validation) follows that registry's names. + **Corrected 2026-08-02 (#184).** Both links were previously unpinned (`/blob/main/`), + and the first was **broken**: #158's rename of `PLATFORM-001` → *the Appwrite Seam + Contract* was applied as a blanket find-and-replace and rewrote the **filename inside + the URL**, producing a path containing spaces that could not resolve. The upstream file + has since been renamed to `appwrite_seam_contract.md` independently. Both now point at + a pinned commit, which is what "referenced by URL, never by copy" was always supposed + to mean — an unpinned `main` link drifts silently, and the registry moved 1.3.0 → 1.4.0 + on 2026-08-02. + --- ## §8 Non-goals (with recorded deferred intents) @@ -857,6 +866,20 @@ runs can therefore never touch the live bucket by accident. Dated events after adoption. Errata correct errors in this document; other entries record execution progress against it. +- **2026-08-02 — Erratum E2 (§7d links; `contract_version` stays 1.5):** §7d's first + link was **broken and had never resolved**. Issue #158 renamed the cross-repo contract + from `PLATFORM-001` to *the Appwrite Seam Contract* and the rename was applied as a + blanket find-and-replace, which rewrote the **filename inside the URL** — producing + `.../platform/the Appwrite Seam Contract_identity_secrets_configuration_contract.md`, + a path containing spaces. #158 was then closed while four `PLATFORM-001` citations + also survived in this repo's `.py`, one of them inside the message raised by + `appwrite_env.assert_env_declared`. Corrected in place (dated marker in §7d): the + upstream file is `appwrite_seam_contract.md`, and **both** §7d links are now pinned to + views-appwrite `b54928f` (registry v1.4.0) rather than tracking `main`. §7d claimed + *"referenced by URL, never by copy"* — true in letter and false in effect while an + unpinned link could drift under it, which it did: the registry moved 1.3.0 → 1.4.0 on + this date. `tests/test_doc_accuracy.py` now refuses the retired name in code. + (#184, epic #181.) - **2026-07-15 — §4.1a's recorded inconsistency CONFIRMED live.** During the Hop-B legacy-guard work (faoapi PR #200), a read-only audit of the live `unfao_bucket` found **six `orange_ensemble`-named forecast documents** stranded by the `name` diff --git a/tests/test_doc_accuracy.py b/tests/test_doc_accuracy.py index eb24a5e..2203627 100644 --- a/tests/test_doc_accuracy.py +++ b/tests/test_doc_accuracy.py @@ -149,3 +149,47 @@ def test_contract_package_does_not_import_the_partner(): f"contract/ imports the partner package: {offenders}. The machinery must be " "reusable by views-crafdapi / views-productionapi without taking FAO (C-69)." ) + + +# --- 3. retired cross-repo contract name (S3 / #184, finishing #158) -------------------- + +#: Retired 2026-07-31. The registry itself records the retirement: +#: ``former_contract_name = "PLATFORM-001" # retired 2026-07-31``. +_RETIRED_CONTRACT_NAME = "PLATFORM-001" + + +def test_the_retired_contract_name_is_gone_from_code(): + """A citation naming a contract that no longer exists sends the reader nowhere. + + #158 renamed ``PLATFORM-001`` to *the Appwrite Seam Contract* and was closed on + 2026-08-01 — but only the documentation half landed. Four references survived in + ``.py``, and one of them was **inside the message raised by + ``appwrite_env.assert_env_declared``**: the text an operator reads when a delivery + refuses to launch, pointing them at a retired name in a repo they do not own, on + the day their run failed. + + "Code" means all three roots that hold Python here — the package, the tests, and + ``scripts/`` (which builds the committed GAUL artifact and the §10 golden fixture). + Scanning fewer roots than the name claims is register **C-74**'s defect, found the + same day this test was written: a guard whose declared scope exceeds its actual + scan passes because the unscanned part happens to be clean. + + ``docs/ADRs/`` and ``reports/`` are deliberately excluded. They legitimately use + the old name when narrating what was decided under it — an ADR describing a 2026-07 + decision correctly says what the thing was called in 2026-07. Code has no such + excuse: it speaks in the present tense to whoever is reading it now. + """ + offenders = [] + roots = (_PKG, _REPO / "tests", _REPO / "scripts") + for source in sorted(f for root in roots for f in root.rglob("*.py")): + if source.resolve() == Path(__file__).resolve(): + continue # this file names it in order to ban it + for number, line in enumerate(source.read_text().splitlines(), 1): + if _RETIRED_CONTRACT_NAME in line: + offenders.append(f"{source.relative_to(_REPO)}:{number}") + assert not offenders, ( + f"{_RETIRED_CONTRACT_NAME!r} was retired on 2026-07-31 and must not appear in " + f"code: {offenders}. Use 'the Appwrite Seam Contract' and cite the registry by " + "pinned URL (views-appwrite/docs/ADRs/platform/coordinate_registry.toml). " + "Historical narration belongs in docs/ADRs or reports/, not here." + ) diff --git a/tests/test_env_declaration.py b/tests/test_env_declaration.py index 8ba81f6..17a23ba 100644 --- a/tests/test_env_declaration.py +++ b/tests/test_env_declaration.py @@ -183,8 +183,9 @@ def test_the_environment_refusal_logs_names_and_never_values(monkeypatch, caplog ) -def test_secret_env_names_follow_the_platform_naming_rule(): - # PLATFORM-001 D3: suffix _API_KEY/_PASSWORD/_TOKEN ⇒ secret. Exactly one +def test_secret_env_names_follow_the_seam_contract_naming_rule(): + # þing-01 D3, the Appwrite Seam Contract §3 ("Classification — declared, never + # inferred"): suffix _API_KEY/_PASSWORD/_TOKEN marks a secret. Exactly one # declared name is a secret; every other declared name is a coordinate. declared = set( appwrite_env.CONNECTION_ENV + appwrite_env.PROD_FORECASTS_ENV + appwrite_env.UNFAO_ENV diff --git a/tests/test_redaction_guard.py b/tests/test_redaction_guard.py index c3cfda7..5283f33 100644 --- a/tests/test_redaction_guard.py +++ b/tests/test_redaction_guard.py @@ -1,5 +1,5 @@ """Guards born from the þing-01 delivery-log/provenance redaction audit (#135, -orð_09 §3 commitment; PLATFORM-001 multi-carrier redaction clause). +orð_09 §3 commitment; the Appwrite Seam Contract's multi-carrier redaction clause). The audit (2026-07-28) found the delivery clean: no credential in any carrier — log lines carry filenames/run_ids/counts/regions, and the provenance description diff --git a/views_postprocessing/unfao/appwrite_env.py b/views_postprocessing/unfao/appwrite_env.py index b60a28d..d5eaf80 100644 --- a/views_postprocessing/unfao/appwrite_env.py +++ b/views_postprocessing/unfao/appwrite_env.py @@ -1,7 +1,16 @@ """The Appwrite-seam environment this package requires, DECLARED (þing-01 P1, #134). -Names follow the PLATFORM-001 coordinate registry (connection + target classes) -plus the one operator-issued secret slot. The LAUNCHER assembles the environment +Names follow **the Appwrite Seam Contract**'s coordinate registry (connection + +target classes) plus the one operator-issued secret slot. The contract is homed in +views-appwrite and referenced by pinned URL, never copied — copies were the +platform's original failure (þing-01 sáttmál S6): + + https://github.com/views-platform/views-appwrite/blob/b54928f/docs/ADRs/platform/coordinate_registry.toml + +That pin is registry **v1.4.0**. A pinned URL does not rot, but it does go stale — +naming the version is what makes the staleness legible to a reader here. + +The LAUNCHER assembles the environment (views-models M3: run.sh reads the owned registry; the secret stays the operator slot) — this package loads no dotenv and validates fail-loud instead (verdict D6). The old implicit borrow (`load_dotenv(ensemble_path_manager.dotenv)`) was the @@ -64,9 +73,10 @@ def assert_env_declared(names: tuple, *, store: str) -> None: # APPWRITE_DATASTORE_API_KEY secret slot. err_msg = ( f"{store}: the launcher did not assemble the required environment — " - f"missing {missing}. Coordinates come from the PLATFORM-001 registry " - "(views-models run.sh declares its sourcing); the secret is the " - "operator slot. This package no longer loads any dotenv (#134)." + f"missing {missing}. Coordinates come from the Appwrite Seam Contract's " + "coordinate registry, homed in views-appwrite (views-models run.sh " + "declares its sourcing); the secret is the operator slot. This package " + "no longer loads any dotenv (#134)." ) logger.error(err_msg) # ADR-008: logged persistently AND raised raise EnvironmentError(err_msg) From c9c107af03390c4c33435499a0376689acf452f0 Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 2 Aug 2026 02:53:32 +0200 Subject: [PATCH 06/23] =?UTF-8?q?fix(seam):=20#196=20=E2=80=94=20repin=20v?= =?UTF-8?q?iews-appwrite=20to=20main's=20tip;=20the=20first=20pin=20was=20?= =?UTF-8?q?an=20unmerged=20branch=20(#198)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit S3 (#184/#195) pinned the Appwrite Seam Contract at a commit resolved with `git -C ../views-appwrite rev-parse HEAD`. That checkout was sitting on the unmerged `feat/s1-single-writer-rule` branch, so the pin captured a commit that never reached `main` and has now been withdrawn (views-appwrite #30, #27). It declared registry v1.4.0; main carries the ratified v1.3.0. Verified before repinning, not taken on the issue's word: the withdrawn commit is reachable only from that feature branch, 47172af is origin/main's tip, its registry declares version = "1.3.0" (RATIFIED — þing-02), and both cited files exist at it. Four references repointed — appwrite_env.py's docstring and ADR-013 §7d's two links — plus the version labels, the §7d correction paragraph, and Erratum E2, all of which asserted a 1.3.0 -> 1.4.0 move that never landed. The withdrawn sha is deliberately not repeated anywhere, including in the prose narrating its withdrawal, so that grepping for it finds nothing. #196's acceptance criteria are met literally. The lesson is narrow and now recorded in both places a future contributor will look: resolve a cross-repo pin from the tip of the other repo's main, never from whatever its working copy has checked out. A pin is a claim about what the contract says; HEAD answers a different question — what someone was working on. Co-authored-by: Claude Opus 5 (1M context) --- .../013_sampled_forecast_wire_contract.md | 33 +++++++++++++------ views_postprocessing/unfao/appwrite_env.py | 13 ++++++-- 2 files changed, 33 insertions(+), 13 deletions(-) diff --git a/docs/ADRs/013_sampled_forecast_wire_contract.md b/docs/ADRs/013_sampled_forecast_wire_contract.md index 2ee920a..07b4b88 100644 --- a/docs/ADRs/013_sampled_forecast_wire_contract.md +++ b/docs/ADRs/013_sampled_forecast_wire_contract.md @@ -689,10 +689,10 @@ a "simplification." this contract governs the **data** crossing the Appwrite seam; the **identity/config** crossing the same seam (which key, which coordinates, sourced from where, named how) is governed by - **[the Appwrite Seam Contract](https://github.com/views-platform/views-appwrite/blob/b54928f/docs/ADRs/platform/appwrite_seam_contract.md)**, + **[the Appwrite Seam Contract](https://github.com/views-platform/views-appwrite/blob/47172af/docs/ADRs/platform/appwrite_seam_contract.md)**, homed in views-appwrite and referenced here **by pinned URL, never by copy** — with its coordinate registry - ([`coordinate_registry.toml`](https://github.com/views-platform/views-appwrite/blob/b54928f/docs/ADRs/platform/coordinate_registry.toml), v1.4.0) + ([`coordinate_registry.toml`](https://github.com/views-platform/views-appwrite/blob/47172af/docs/ADRs/platform/coordinate_registry.toml), v1.3.0) as the canonical source of the non-secret coordinates this repo's runtime resolves. This repo's declared environment (`unfao/appwrite_env.py`, fail-loud entry validation) follows that registry's names. @@ -703,8 +703,14 @@ a "simplification." the URL**, producing a path containing spaces that could not resolve. The upstream file has since been renamed to `appwrite_seam_contract.md` independently. Both now point at a pinned commit, which is what "referenced by URL, never by copy" was always supposed - to mean — an unpinned `main` link drifts silently, and the registry moved 1.3.0 → 1.4.0 - on 2026-08-02. + to mean — an unpinned `main` link drifts silently under the reader. + + **Re-pinned same day (#196).** The first pin taken here was resolved from a local + views-appwrite checkout's `HEAD` — which was sitting on an unmerged branch. That + commit declared registry v1.4.0, never reached `main`, and has been withdrawn. Both + links now pin `47172af`, the tip of `main`, carrying the **ratified v1.3.0**. The + lesson is narrow and worth keeping: **resolve a cross-repo pin from the tip of the + other repo's `main`, not from whatever its working copy has checked out.** --- @@ -874,12 +880,19 @@ record execution progress against it. a path containing spaces. #158 was then closed while four `PLATFORM-001` citations also survived in this repo's `.py`, one of them inside the message raised by `appwrite_env.assert_env_declared`. Corrected in place (dated marker in §7d): the - upstream file is `appwrite_seam_contract.md`, and **both** §7d links are now pinned to - views-appwrite `b54928f` (registry v1.4.0) rather than tracking `main`. §7d claimed - *"referenced by URL, never by copy"* — true in letter and false in effect while an - unpinned link could drift under it, which it did: the registry moved 1.3.0 → 1.4.0 on - this date. `tests/test_doc_accuracy.py` now refuses the retired name in code. - (#184, epic #181.) + upstream file is `appwrite_seam_contract.md`, and **both** §7d links are now pinned + rather than tracking `main`. §7d claimed *"referenced by URL, never by copy"* — true in + letter and false in effect while an unpinned link could drift under it. + `tests/test_doc_accuracy.py` now refuses the retired name in code. (#184, epic #181.) + + **Amended the same day (#196):** the first pin was taken from a local views-appwrite + checkout's `HEAD` while it sat on the unmerged `feat/s1-single-writer-rule` branch. That + commit declared registry v1.4.0 and has been **withdrawn unmerged** (views-appwrite #30, + #27); its sha is deliberately not repeated here, so that grepping for it finds nothing. + Both links now pin **`47172af`** — the tip of `main`, registry **v1.3.0**, ratified at + þing-02. Verified before re-pinning: the withdrawn commit is reachable only from that + branch, `47172af` is `origin/main`, and both cited files exist at it. **Resolve a + cross-repo pin from the other repo's `main`, never from its working copy's `HEAD`.** - **2026-07-15 — §4.1a's recorded inconsistency CONFIRMED live.** During the Hop-B legacy-guard work (faoapi PR #200), a read-only audit of the live `unfao_bucket` found **six `orange_ensemble`-named forecast documents** stranded by the `name` diff --git a/views_postprocessing/unfao/appwrite_env.py b/views_postprocessing/unfao/appwrite_env.py index d5eaf80..a9e1597 100644 --- a/views_postprocessing/unfao/appwrite_env.py +++ b/views_postprocessing/unfao/appwrite_env.py @@ -5,10 +5,17 @@ views-appwrite and referenced by pinned URL, never copied — copies were the platform's original failure (þing-01 sáttmál S6): - https://github.com/views-platform/views-appwrite/blob/b54928f/docs/ADRs/platform/coordinate_registry.toml + https://github.com/views-platform/views-appwrite/blob/47172af/docs/ADRs/platform/coordinate_registry.toml -That pin is registry **v1.4.0**. A pinned URL does not rot, but it does go stale — -naming the version is what makes the staleness legible to a reader here. +That pin is registry **v1.3.0** (ratified, þing-02). A pinned URL does not rot, but +it does go stale — naming the version is what makes the staleness legible here. + +**Pin from the tip of `main`, never from a sibling checkout's `HEAD`** (#196). The +first pin taken here was resolved with `git -C ../views-appwrite rev-parse HEAD` on a +checkout that happened to be sitting on an unmerged feature branch. That commit +declared v1.4.0, never reached `main`, and was withdrawn. A pin is a claim about what +the contract *says*; `HEAD` answers a different question — what someone was working +on. The LAUNCHER assembles the environment (views-models M3: run.sh reads the owned registry; the secret stays the operator From 5aff7f1c66e7b4b8fa82343233fa4ef869905c9f Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 2 Aug 2026 03:10:39 +0200 Subject: [PATCH 07/23] =?UTF-8?q?docs(accuracy):=20S11=20#197=20=E2=80=94?= =?UTF-8?q?=20living=20docs=20still=20described=20code=20#148=20deleted;?= =?UTF-8?q?=20the=20ban=20list=20was=20never=20updated=20(#199)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Epic #148's story #154 ("correct ADR-012 and the manager CIC to describe what the code is") was closed 2026-08-01 with ADR-012 done and the rest not. Ten live falsehoods survived across four living onboarding documents. The worst was views_postprocessing/unfao/managers/README.md, untouched by #154 and wrong in six places: it documented the Template Method as `_transform → _append_metadata`, pointed readers at tests/test_append_metadata.py, and claimed Appwrite settings have "no startup validation yet — tracked in #11" when þing-01 #134 added fail-loud validation and #11 is about timeouts. It sits inside the package, next to the code it misdescribed. tests/test_doc_accuracy.py scans all four files. Its ban list carried only the ADR-011 / C-39 symbols and nothing was added when #149 deleted a delivery path and #153 moved the machinery out of unfao/. The guard was not wrong; it was not updated. Extended 5 -> 19 entries, grouped by the PR that deleted each, with a (? --- README.md | 13 +- docs/CICs/UNFAOPostProcessorManager.md | 8 +- docs/architecture/role_and_seams.md | 15 +-- tests/test_doc_accuracy.py | 115 +++++++++++++++++- views_postprocessing/unfao/managers/README.md | 57 ++++++--- 5 files changed, 171 insertions(+), 37 deletions(-) diff --git a/README.md b/README.md index 6c0bd7b..ff12b06 100644 --- a/README.md +++ b/README.md @@ -72,11 +72,14 @@ In practice the manager is constructed and run by **views-models** | Stage | Method(s) | What happens | |-------|-----------|--------------| -| **Read** | `_read_historical_data`, `_read_forecast_data` | Historical actuals from views-datafactory (via the inherited loader); the forecast file from the Appwrite prediction store. The forecast file's identity is checked before use (C-25). | -| **Transform** | `_transform` → `_append_metadata` | Joins the 9 GAUL metadata columns onto each frame via `GaulLookupEnricher` (a parquet lookup). Prediction values are **not** transformed. | -| **Validate** | `_validate`, `_check_coverage` | Null-gate on the metadata columns; region coverage + GAUL-excluded-cell guards (C-34 / C-30). | -| **Clip** | `_clip_observed_history` | Drops fabricated zero-padded tail months from the historical actuals (C-26); the forecast is untouched. | -| **Save** | `_save` | Writes parquet and uploads to the UN FAO bucket with structured provenance (C-15). | +| **Read** | `_read_historical_frame`, `_read_forecast_data_contract` | Historical actuals from views-datafactory arrive **frame-native** (#126); the forecast run is resolved from the Appwrite store by its **run manifest**, with each shard's header verified on load (ADR-013 §4.3). | +| **Transform** | `_transform` | Resolution only. Prediction values are **not** transformed — no collapse, no reconciliation. | +| **Validate** | `_validate`, `_check_coverage` | Asserts the read resolved, then enforces the region coverage + GAUL-excluded-cell contract (C-34 / C-30). The metadata null-gate fires later, at artifact build (`contract/historical.assert_metadata_complete`). | +| **Save** | `_save` → `_save_contract` | Builds the ADR-013 wire — arrow shards, the §5 GAUL sidecar, the historical artifact — commits the run **manifest last**, and stamps each upload with structured provenance (C-15). | + +The pandas metadata-join and history-clip stages were retired with the legacy delivery path +in #149; their rules survive as called invariants under `delivery/`. See the +[manager README](views_postprocessing/unfao/managers/README.md) for what moved where. ### Output schema (geographic metadata columns) diff --git a/docs/CICs/UNFAOPostProcessorManager.md b/docs/CICs/UNFAOPostProcessorManager.md index e543a40..cf281c9 100644 --- a/docs/CICs/UNFAOPostProcessorManager.md +++ b/docs/CICs/UNFAOPostProcessorManager.md @@ -77,11 +77,11 @@ Assumptions that are not met **must cause failure**, not fallback behavior. **Kn - **Null values in required metadata columns:** Raises `ValueError` with null count and affected column name (C-01 resolved — validation active) - **Dataset initialization failure:** Raises `ValueError` in `_save()` if datasets are None - **Appwrite upload failure:** Propagates exception from `DatastoreModule` -- **Wrong forecast selected:** structurally impossible since #149. Selection is by **run manifest** — a commit marker whose contents are hash-verified — not by scanning the bucket for the newest `category="forecast"` upload. Declared identity is additionally checked **per shard header** against the launched ensemble inside `TargetLease.load()` (`contract/wire/source_selection.py:73-81`), so identity comes from the artifact's own content. The metadata-field check this bullet used to describe (`delivery/identity.py`) was retired in #150 and the legacy reader it served in #149; register C-25 is closed as *superseded by mechanism* +- **Wrong forecast selected:** structurally impossible since #149. Selection is by **run manifest** — a commit marker whose contents are hash-verified — not by scanning the bucket for the newest `category="forecast"` upload. Declared identity is additionally checked **per shard header** against the launched ensemble inside `TargetLease.load()` (`contract/wire/source_selection.py:73-81`), so identity comes from the artifact's own content. The metadata-field check this bullet used to describe (`delivery/identity.py`) was retired in #150 and the legacy reader it served in #149; register C-25 is closed as *superseded by mechanism* - **Launch config incomplete:** raises `LaunchConfigError` naming the missing key. A launcher that omits `wire_contract` or declares a `data_format` other than `feature_frame` is **refused**, never quietly routed into a fallback (ADR-003, register C-63) - **Region coverage mismatch:** Raises `CoverageError` in `_check_coverage()` (called from `_validate()`) if a pinned region's delivered cell count is wrong (S1/C-34) or a GAUL-uncovered excluded cell leaks into the delivery (S4/C-30) -- **Fabricated historical tail:** `_read_historical_frame()` drops months beyond the producer's `last_valid_month_id` at the read (`_clip_observed_history` was the pandas equivalent, retired with that path in #149) so unobserved zero-padding is not shipped as observed history (S2/C-26); **degrades open** (skips the clip with a WARNING) if the boundary cannot be resolved -- **Upload provenance:** every upload's `description` carries structured provenance (lookup version, region, expected/actual cell counts, unmapped count) via `_delivery_description()` (S5/C-15) +- **Fabricated historical tail:** `_read_historical_frame()` drops months beyond the producer's `last_valid_month_id` at the read (`_clip_observed_history` was the pandas equivalent, retired with that path in #149) so unobserved zero-padding is not shipped as observed history (S2/C-26); **degrades open** (skips the clip with a WARNING) if the boundary cannot be resolved +- **Upload provenance:** the historical artifact's `description` carries structured provenance (lookup version, region, expected/actual cell counts, unmapped count) built by `delivery/provenance.py` (`build_provenance` → `compact_description`) via the manager's `_historical_frame_description()` (S5/C-15). The **forecast** side carries no such description: its guarantee is the wire's verified chain — per-shard content hashes recorded in the §4.2 run manifest, header asserts on load, and manifest-last commit ordering. That is identity and integrity, not the C-15 provenance field set; the §4.2 manifest's keys are exactly `contract_version`, `run_id`, `targets`, `shards`, `expected_months`, `expected_cell_count`, `sidecar` — and it carries **no** `lookup_version`, `region` or `unmapped_count`. `_delivery_description()` was the pandas-path equivalent and was deleted with it in #149 The following **must never** fail silently: - Missing or None environment variables for Appwrite @@ -145,7 +145,7 @@ manager._save() - **Beige tests:** Missing ensemble name in config; None environment variables; empty forecast bucket; DataFrames with unexpected index structure - **Red tests:** Corrupted parquet downloads; network timeouts during upload; DataFrames where all cells map to None (all-ocean input) -Currently: the manager cannot be instantiated without `views-pipeline-core`, so its stage logic is covered by **source-scan and replica tests** — `tests/test_validation.py`, `tests/test_launch_config.py` (the refusals), `tests/test_gaul_lookup_access.py` (one lookup read, threaded). `tests/test_append_metadata.py` was deleted in #149 with the method it mirrored. A full end-to-end test against the live manager (C-03) still requires a production-like environment. +Currently: the manager cannot be instantiated without `views-pipeline-core`, so its stage logic is covered by **source-scan and replica tests** — `tests/test_validation.py`, `tests/test_launch_config.py` (the refusals), `tests/test_gaul_lookup_access.py` (one lookup read, threaded). `tests/test_append_metadata.py` was deleted in #149 with the method it mirrored. A full end-to-end test against the live manager (C-03) still requires a production-like environment. The input-integrity guards (S0–S6, epic #51) are representation-free invariants in `views_postprocessing/delivery/` that the manager **calls** (never inherits). Each has primitives unit tests — `tests/test_delivery_coverage.py` (S1/S4), `tests/test_delivery_observed_range.py` (S2), `tests/test_provenance.py` (S5), `tests/test_frame_extraction.py` (the seam), `tests/test_store_metadata.py` (store identity) — and `tests/test_input_integrity_e2e.py` drives the invariants on **primitives**, which is how the manager calls them. S3's forecast-identity rule moved to the wire layer (#150) and is covered by `tests/test_wire_source_selection.py`. The design contract (representation-free, called-not-inherited) is pinned by `tests/test_input_integrity_design_contract.py`. diff --git a/docs/architecture/role_and_seams.md b/docs/architecture/role_and_seams.md index 9587d0a..7813b2c 100644 --- a/docs/architecture/role_and_seams.md +++ b/docs/architecture/role_and_seams.md @@ -55,8 +55,9 @@ pattern: `_read → _transform → _validate → _save` in order. - this repo **fills in the steps** for the FAO path (the `_read*/_transform/_validate/_save` overrides in `unfao/managers/unfao.py`). -- pipeline-core also **provides the tools** the steps use: `ViewsDataLoader`, `PGMDataset`, +- pipeline-core also **provides the tools** the steps use: `ViewsDataLoader`, `DatastoreModule`, `AppwriteConfig`, the path managers. + (`PGMDataset` was on this list until #149 retired the pandas delivery path; this repo no longer references it — see Seam B.) So the runtime control flow is *inverted* ("don't call us, we'll call you"): views-models calls `manager.execute()`, which lives in **pipeline-core's base**, which calls back into @@ -110,12 +111,12 @@ The manager **calls** the invariants; it never makes them methods of itself. The always `extract (seam) → call invariant → raise`. This is why the guards are testable without the framework, and why they survive a representation change untouched. -**And they did survive one.** This seam was `unfao/extraction.py` (pandas) until #151. +**And they did survive one.** This seam was `unfao/extraction.py` (pandas) until #151. When the pandas delivery was retired, the invariants needed **no change at all** — only which module fed them. That is the design working exactly as this section claims, and it is -the evidence for the claim rather than a restatement of it. `delivery/identity.py` was also -retired (#150): the forecast-identity rule now lives in the wire layer, checked per shard -header against declared provenance — see §5 Seam C. +the evidence for the claim rather than a restatement of it. `delivery/identity.py` was also +retired (#150): the forecast-identity rule now lives in the wire layer, checked per shard +header against declared provenance — see §5 Seam C. ### Seam B — the inherited pandas base, and the C-40 gate @@ -123,7 +124,7 @@ Because this repo *is-a* pipeline-core postprocessor (section 3), three **concre pieces are inherited, not chosen: 1. the input loader (`ViewsDataLoader` → parquet → pandas), -2. the dataset container (`PGMDataset`, a pandas `DataFrame` with object-dtype cells), +2. the dataset container (`PGMDataset`, a pandas `DataFrame` with object-dtype cells), 3. the prediction-store parquet I/O. **This section described the state until 2026-07-27; it is no longer true and is kept @@ -147,7 +148,7 @@ samples per cell). This is where representation matters most: - views-frames stores a distribution natively as a contiguous `(N, S)` float32 array (sample axis explicit; a point is just `S=1`). -- pandas `PGMDataset` stores it as **object-dtype list-in-cell** — a separate numpy array +- pandas `PGMDataset` stores it as **object-dtype list-in-cell** — a separate numpy array boxed in each of N cells. Cost scales ~linearly with S (memory, an encode/decode tax at every parquet/API boundary, a silent resize on mismatched sample counts). diff --git a/tests/test_doc_accuracy.py b/tests/test_doc_accuracy.py index 2203627..de7500b 100644 --- a/tests/test_doc_accuracy.py +++ b/tests/test_doc_accuracy.py @@ -24,10 +24,36 @@ # --- 1. deleted-symbol scan ------------------------------------------------------------- -# Symbols that name code removed in ADR-011 / C-39. None should appear as *current* in a -# living onboarding doc. +# Symbols naming code this repo has deleted. None should appear as *current* in a living +# onboarding doc. +# +# **This list must grow with every deletion, and that is the lesson of S11 (#197).** It +# carried only the ADR-011 / C-39 entries for six weeks while epic #148 deleted a whole +# delivery path (#149) and moved the machinery out of `unfao/` (#153). The guard kept +# passing over four documents describing methods and files that no longer existed — +# including `unfao/managers/README.md`, which documented the Template Method as +# `_transform → _append_metadata` and pointed readers at a deleted test file, sitting +# inside the package next to the code it misdescribed. +# +# The guard was not wrong. It was not updated. A deletion PR that does not extend this +# regex has not finished. _BANNED = re.compile( - r"PriogridCountryMapper|mapping/README|use_disk_cache|cachetools|geopandas", + "|".join(( + # ADR-011 / C-39 — the runtime mapper and its shapefile toolchain (June 2026) + r"PriogridCountryMapper", r"mapping/README", r"use_disk_cache", r"cachetools", + r"geopandas", + # #149 — methods of the retired pandas delivery path (2026-07-31) + r"_append_metadata", r"_delivery_description", r"_clip_observed_history", + r"LEGACY_FORECAST_FILTERS", r"test_append_metadata", + # #150 / #151 / #153 — modules retired or moved out of `unfao/` by epic #148 + r"delivery/identity", r"unfao/extraction", r"unfao/frames", + r"unfao/historical", r"unfao/gaul_schema", r"unfao/wire", + # #149 — pipeline-core's pandas container, no longer referenced by this repo. + # Negative lookbehind: `FAO_PGMDataset` is views-faoapi's class and is live. + # It inherits IGNORECASE, so `fao_PGMDataset` is spared too — no such spelling + # exists, but the lookbehind is not case-exact and a reader should not assume it. + r"(? list[Path]: return [d for d in docs if d.exists()] -def test_living_docs_have_no_deleted_symbol_references(): +def _label(doc: Path) -> str: + """Repo-relative path where possible — three living docs are named ``README.md``, + so a bare basename cannot say which one failed.""" + try: + return doc.relative_to(_REPO).as_posix() + except ValueError: + return doc.name # a doc injected from tmp_path by the guard's own tests + + +def _deleted_symbol_offenders(docs: list[Path]) -> list[str]: + """Scan ``docs`` for banned symbols, honouring the line-scoped ``legacy-ok`` opt-out.""" offenders = [] - for doc in _living_docs(): + for doc in docs: for i, line in enumerate(doc.read_text().splitlines(), start=1): if "legacy-ok" in line: # explicit opt-out for an intentional historical mention continue if _BANNED.search(line): - offenders.append(f"{doc.relative_to(_REPO)}:{i}: {line.strip()}") + offenders.append(f"{_label(doc)}:{i}: {line.strip()}") + return offenders + + +def test_living_docs_have_no_deleted_symbol_references(): + offenders = _deleted_symbol_offenders(_living_docs()) assert not offenders, "deleted-symbol references in living docs:\n" + "\n".join(offenders) @@ -193,3 +234,65 @@ def test_the_retired_contract_name_is_gone_from_code(): "pinned URL (views-appwrite/docs/ADRs/platform/coordinate_registry.toml). " "Historical narration belongs in docs/ADRs or reports/, not here." ) + + +def test_the_ban_covers_the_post_148_deletions_and_spares_the_live_lookalike(): + """A ban list that was not updated when the code changed is why S11 (#197) existed. + + Two failure modes, both real, so both are pinned: + + 1. **Entries going missing** — someone trims the list and a document quietly starts + describing deleted code again. The loop below fails if any known entry stops + matching. Note what it *cannot* catch: a future deletion whose symbol nobody adds + is invisible to this test and to the guard alike, which is why the comment above + ``_BANNED`` says a deletion PR that does not extend it has not finished. No test + can substitute for that; only the habit can. + 2. **Too broad** — `FAO_PGMDataset` is a *views-faoapi* class, alive and correctly + cited in `contract/gaul_schema.py` and `contract/enrichment.py`. A careless + `PGMDataset` pattern would ban a true statement about another repo's code. + """ + for deleted in ( + "_append_metadata", "_delivery_description", "_clip_observed_history", + "LEGACY_FORECAST_FILTERS", "tests/test_append_metadata.py", + "delivery/identity.py", "unfao/extraction.py", "unfao/frames.py", + "unfao/historical.py", "unfao/gaul_schema.py", "unfao/wire/sink.py", + "PGMDataset", "PriogridCountryMapper", "geopandas", + ): + assert _BANNED.search(deleted), ( + f"{deleted!r} names code this repo deleted, but the ban list does not cover " + "it — a living doc could describe it as current and nothing would object" + ) + + assert not _BANNED.search("FAO_PGMDataset._METADATA_COLS"), ( + "the ban caught `FAO_PGMDataset`, which is views-faoapi's live class and the " + "consumer contract our sidecar is written against — banning a true statement " + "about another repo's code is how a guard earns its own deletion" + ) + + +def test_the_ban_actually_fires_on_a_living_doc(tmp_path): + """The scan, not just the regex — a pattern nothing applies is decoration.""" + doc = tmp_path / "README.md" + doc.write_text("The manager joins metadata in `_append_metadata`.\n") + assert _deleted_symbol_offenders([doc]), ( + "the scan reported nothing on a doc naming a deleted method" + ) + + +def test_the_legacy_ok_marker_still_works(tmp_path): + """The escape hatch the retirement records depend on — line-scoped, deliberately. + + Four documents now name deleted methods in order to say where the work went. That is + better than a gap, and it only stays possible while this marker keeps working. It is + line-scoped: a marker on the following line does not excuse the mention above it. + """ + excused = tmp_path / "ok.md" + excused.write_text("`_append_metadata` was retired in #149. \n") + assert not _deleted_symbol_offenders([excused]) + + wrong_line = tmp_path / "wrong.md" + wrong_line.write_text("`_append_metadata` was retired.\n\n") + assert _deleted_symbol_offenders([wrong_line]), ( + "a marker on the NEXT line excused the mention — the opt-out must stay line-scoped, " + "or a single marker silently covers a whole document" + ) diff --git a/views_postprocessing/unfao/managers/README.md b/views_postprocessing/unfao/managers/README.md index 1812df1..abbd524 100644 --- a/views_postprocessing/unfao/managers/README.md +++ b/views_postprocessing/unfao/managers/README.md @@ -13,9 +13,17 @@ and uploads the result to the FAO Appwrite store. `UNFAOPostProcessorManager` is a **concrete pipeline-core postprocessor** — it subclasses `PostprocessorManager` + `ForecastingModelManager` (Template Method) and fills the -`read → transform → validate → save` lifecycle. It is a thin orchestrator: the -representation-free input-integrity rules live in `views_postprocessing/delivery/` and are -**called** by the manager (via the `unfao/extraction.py` seam), never inherited into it. +`read → transform → validate → save` lifecycle. The representation-free input-integrity +rules live in `views_postprocessing/delivery/` and are **called** by the manager (via the +`contract/frame_extraction.py` seam), never inherited into it. + +It is **406 lines**, against a 450-line budget enforced by +`tests/test_doc_accuracy.py::test_the_manager_stays_within_its_line_budget`. It was 636 +before #149. ADR-012 deliberately stopped calling it "thin" and states a number instead — +a word nobody can check became a bound a test can. + +It is also **the only module in this repository that imports `views_pipeline_core`**, and a +test keeps it that way (register C-40). It does **not** transform prediction values (no collapse, no reconciliation — those are downstream). It joins metadata, guards integrity, and delivers. @@ -24,14 +32,31 @@ downstream). It joins metadata, guards integrity, and delivers. | Stage | Method(s) | What happens | |-------|-----------|--------------| -| **Read** | `_read_historical_data`, `_read_forecast_data` | Historical actuals via the inherited `ViewsDataLoader`; the forecast file from the Appwrite prediction store. The selected forecast file's **identity** (name/loa) is asserted before download (`delivery.identity`, C-25). | -| **Transform** | `_transform` → `_append_metadata` | Joins the 9 GAUL metadata columns via `GaulLookupEnricher` (a precomputed parquet lookup, ADR-011). | -| **Validate** | `_validate`, `_check_coverage` | Null-gate on the 9 metadata columns; region **coverage** + GAUL-**excluded-cell** guards (`delivery.coverage`, C-34 / C-30). | -| **Clip** | `_clip_observed_history` | Drops fabricated zero-padded tail months from the *historical* actuals (`delivery.observed_range`, C-26); the forecast is untouched. The boundary (`last_valid_month_id`) is read from the producer (datafactory) via `unfao/source_metadata.py`. | -| **Save** | `_save` | Writes timestamped parquet and uploads to the FAO bucket, stamping each upload with **structured provenance** (`delivery.provenance`, C-15). | +| **Read** | `_read_historical_frame`, `_read_forecast_data_contract` | Historical actuals arrive **frame-native** (#126) via the inherited loader; the forecast run is resolved from the Appwrite store by its **run manifest**, and each shard's header is verified on load (`contract/wire/source_selection.py`, ADR-013 §4.3). | +| **Transform** | `_transform` | Resolution only. Metadata is no longer joined onto a pandas frame here — the GAUL columns are attached where each artifact is built. | +| **Validate** | `_validate`, `_check_coverage` | Asserts the read **resolved** (frame + forecast run present), then enforces the region coverage contract (`delivery/coverage.py`, C-34 / C-30). | +| **Save** | `_save` → `_save_contract` | Builds the ADR-013 wire — arrow shards, the §5 GAUL sidecar, the historical artifact — and commits the run **manifest last**. Each upload is stamped with structured provenance (`delivery/provenance.py`, C-15). | + +**Where the null-gate went.** It is no longer in `_validate`. The 9 metadata columns are +gated at artifact-build time by `contract/historical.assert_metadata_complete`, and the +forecast's guarantees are the wire's own verified chain: content hashes, header asserts, +per-target coverage inside each lease's `load()`, the §6 no-collapse gate, and gid parity +at `_save`. + +The 9 metadata columns are the single-source contract in `contract/gaul_schema.py` +(`COLUMNS` → `METADATA_COLS`, register C-70). + +**Retired in #149** (epic #148). Named here deliberately — older issues and PRs still +reference them, and a reader who greps for one deserves an answer rather than a gap: + +| retired | what it was | where the work went | +|---|---|---| +| `_append_metadata` | the pandas GAUL metadata join | attached per artifact at build time | +| `_clip_observed_history` | dropped zero-padded tail months | the rule survives as `delivery/observed_range.py` | +| `_delivery_description` | the store-document text | `delivery/provenance.py` | -The 9 metadata columns are the single-source contract in `unfao/gaul_schema.py` -(`METADATA_COLS`). +The pandas delivery path they belonged to is gone. There is one path, and a launcher that +has not declared it is refused by name (`contract/launch_config.py`, register C-63). ## Running it @@ -45,14 +70,16 @@ manager.execute() # read → transform → validate → save In production this is invoked by **views-models** (`postprocessors/un_fao/main.py`), not directly. The manager cannot be instantiated without `views-pipeline-core` + Appwrite env, -so its stage logic is covered by replica tests (`tests/test_validation.py`, -`tests/test_append_metadata.py`) and the input-integrity e2e suite -(`tests/test_input_integrity_e2e.py`) — see the CIC §10. +so its stage logic is covered by source-scan and seam tests rather than by instantiating +it — `tests/test_validation.py`, `tests/test_input_integrity_e2e.py`, +`tests/test_hop_b_sink_e2e.py` and the ADR-013 conformance suite. See the CIC §10. ## Configuration -Appwrite settings are read from the environment via `os.getenv` (no startup validation yet — -tracked in #11). The full variable set is documented in the repo +Appwrite settings are read from the environment and **validated fail-loud before any store +is constructed** — `unfao/appwrite_env.py` declares the required names and refuses a partial +environment, naming every missing variable (þing-01 D6, #134). This package loads no dotenv; +the launcher assembles the environment. The full variable set is documented in the repo [README](../../../README.md#configuration); note `APPWRITE_PROD_FORECASTS_COLLECTION_ID` is flagged there as needs-verify (the previously-documented value was found absent in live Appwrite). From a067d61f928473dddd4388ca18a1b0b8b7552bfe Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 2 Aug 2026 03:46:38 +0200 Subject: [PATCH 08/23] =?UTF-8?q?test(validation):=20S4=20#185=20=E2=80=94?= =?UTF-8?q?=20retire=20the=20=5Fvalidate=20replica;=20test=20the=20gate=20?= =?UTF-8?q?where=20it=20fires=20(#200)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit tests/test_validation.py defined its own validate_dataframe() and ran 43 parametrised cases against it. Forty-three tests, zero production code — a closed loop that could not have failed if the real gate were deleted outright. Its header claimed "the logic tested matches unfao.py:_validate() exactly". False since #149, which stopped _validate null-gating entirely; its docstring now reads "Neither payload is null-gated here." It also carried REQUIRED_METADATA_COLS as a nine-element literal — the hand copy of the contract that C-70 was resolved to make single-source. Replaced by 14 tests of contract/historical.assert_metadata_complete, the code that actually gates a delivery, parametrised over the IMPORTED METADATA_COLS so a column added to gaul_schema.COLUMNS is gated automatically. Plus source-scan pins that the gate has not drifted back into _validate and that the manager still calls it. Mutation-tested: narrowing the gate to a single column fails 9 of the 14. The old suite passed that same mutation untouched. The behaviour C-01 fixed is preserved in full; it is simply asserted against the function that performs it. C-03 closes. Residual 1 (the replica) is gone. Residual 2 (the enrich->validate end-to-end path) is RELOCATED to #18 by number, per the convention that a relocation is not complete until the destination exists and is cited. Every leg is covered — enrichment, artifact build, reader parity, invariants on primitives, the wire e2e, and now the null-gate. What remains is the manager ORCHESTRATING them, which needs pipeline-core and a production-like Appwrite environment that þing-02 D2 forbids. That gap already had two trackers; a third here was noise. Three `assert False` campaign probes retired because the findings they assert are discharged: 3.2's "validation uses replica not real code", 4.3's "filter_cols change not detected", and 4.3's "enrichment test checks only 5 of 9" — the last verified before removal, since test_enrichment.py asserts list(lookup.columns) == METADATA_COLS, exact equality on all nine, then iterates the full set twice more. C-36's own resolution predicted this: "the pure-assert False probes don't test the live condition, so a fixed finding won't auto-flip." A discharged probe must be retired by hand or it permanently asserts a defect that no longer exists. Each removal leaves a dated note; the modules stay, because the campaign record is the point. Review caught the story closing C-03 while the CIC still cited it as open, next to a now-false "replica tests" — the exact drift class, in the contract document. Corrected to name #18 and þing-02 D2. Co-authored-by: Claude Opus 5 (1M context) --- docs/CICs/UNFAOPostProcessorManager.md | 2 +- reports/technical_risk_register.md | 55 +++-- tests/test_falsification_campaign_3_2.py | 23 +- tests/test_falsification_campaign_4_3.py | 42 ++-- tests/test_validation.py | 268 ++++++++++++----------- 5 files changed, 198 insertions(+), 192 deletions(-) diff --git a/docs/CICs/UNFAOPostProcessorManager.md b/docs/CICs/UNFAOPostProcessorManager.md index cf281c9..2629c93 100644 --- a/docs/CICs/UNFAOPostProcessorManager.md +++ b/docs/CICs/UNFAOPostProcessorManager.md @@ -145,7 +145,7 @@ manager._save() - **Beige tests:** Missing ensemble name in config; None environment variables; empty forecast bucket; DataFrames with unexpected index structure - **Red tests:** Corrupted parquet downloads; network timeouts during upload; DataFrames where all cells map to None (all-ocean input) -Currently: the manager cannot be instantiated without `views-pipeline-core`, so its stage logic is covered by **source-scan and replica tests** — `tests/test_validation.py`, `tests/test_launch_config.py` (the refusals), `tests/test_gaul_lookup_access.py` (one lookup read, threaded). `tests/test_append_metadata.py` was deleted in #149 with the method it mirrored. A full end-to-end test against the live manager (C-03) still requires a production-like environment. +Currently: the manager cannot be instantiated without `views-pipeline-core`, so its stage logic is covered by **source-scan and seam tests** — `tests/test_validation.py` (the metadata null-gate, tested against `contract/historical.assert_metadata_complete` where it fires, plus pins that it has not drifted back into `_validate`), `tests/test_launch_config.py` (the refusals), `tests/test_gaul_lookup_access.py` (one lookup read, threaded). There is no longer a replica of any manager method: `tests/test_validation.py` held one until S4 (#185) and it had diverged from `_validate` since #149, while `tests/test_append_metadata.py` was deleted in #149 with the method it mirrored. A full end-to-end test against the **live manager** still requires a production-like environment and is tracked as **#18**; þing-02 **D2** forbids integration tests against the production Appwrite project, no non-production one existing. The input-integrity guards (S0–S6, epic #51) are representation-free invariants in `views_postprocessing/delivery/` that the manager **calls** (never inherits). Each has primitives unit tests — `tests/test_delivery_coverage.py` (S1/S4), `tests/test_delivery_observed_range.py` (S2), `tests/test_provenance.py` (S5), `tests/test_frame_extraction.py` (the seam), `tests/test_store_metadata.py` (store identity) — and `tests/test_input_integrity_e2e.py` drives the invariants on **primitives**, which is how the manager calls them. S3's forecast-identity rule moved to the wire layer (#150) and is covered by `tests/test_wire_source_selection.py`. The design contract (representation-free, called-not-inherited) is pinned by `tests/test_input_integrity_design_contract.py`. diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 8e5a23e..b0c1c7c 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -6,8 +6,8 @@ | Owner | Dylan Pinheiro / PRIO MD&D Team | | Last Updated | 2026-08-02 | | Total Concerns | 74 | -| Open Concerns | 22 | -| Resolved Concerns | 52 | +| Open Concerns | 21 | +| Resolved Concerns | 53 | --- @@ -113,24 +113,6 @@ that indexes only deleted code is noise. ## Open Concerns -### C-03: Test coverage gaps in manager validation and the enrich→validate path - -| Field | Value | -|-------|-------| -| ID | C-03 | -| Tier | 3 | -| Source | `repo-assimilation` (2026-06-02), `test-review` (2026-06-02) | -| Trigger | When modifying the manager's `_validate()` or the enricher, verify that the test suite covers the changed behavior — end-to-end coverage across the enrich→validate path is still absent | -| Location | `tests/test_validation.py`, `views_postprocessing/unfao/managers/unfao.py` | - -Initial state was zero test coverage. A 73-test suite was written (2026-06-02) covering the (now-deleted) mapper's core guarantees and the validation logic (missing columns, null rejection, error messages). Remaining gaps after the mapper removal: (1) the validation tests replicate `_validate()` logic in a standalone function because `views-pipeline-core` is unavailable in test environments — if the real `_validate()` diverges, tests pass while production fails; (2) no end-to-end test enriches through `GaulLookupEnricher` then validates through the manager. - -Tier recalibrated from 2 to 3 during review-rr (2026-06-02): the gap is maintainability (test-code divergence, missing integration path), not structural fragility. - -**Update 2026-06-24:** narrowed with the mapper deletion (C-39, PR #42). The mapper-coverage dimension is gone with the mapper (`tests/test_mapping.py` deleted; the determinism/cache/shapefile/`ThreadPoolExecutor` gaps no longer exist). Two manager-side gaps remain: the standalone `_validate()` replica and the missing enrich→validate end-to-end test. - ---- - ### C-07: Undeclared direct runtime dependencies in pyproject.toml | Field | Value | @@ -711,6 +693,37 @@ See also C-40 (the inheritance/representation coupling this migration unwinds), --- ## Resolved Concerns + +### C-03: Test coverage gaps in manager validation and the enrich→validate path — RESOLVED + +| Field | Value | +|-------|-------| +| ID | C-03 | +| Resolved | 2026-08-02 | +| Resolution | **Closed by S4 (#185): one of its two residuals is gone, the other is relocated to a tracker that already exists.** + +**Residual 1 — the `_validate()` replica — REMOVED.** `tests/test_validation.py` defined its own `validate_dataframe` and then tested *that*: forty-odd parametrised cases exercising a function declared twelve lines above them and **no production code at all**. A closed loop that could not fail while the delivery broke. Its header claimed *"the logic tested matches unfao.py:_validate() exactly"* — false since **#149**, which stopped `_validate` null-gating entirely (its docstring now says *"Neither payload is null-gated here"*). It also carried `REQUIRED_METADATA_COLS` as a nine-element literal, a hand copy of the contract **C-70** was resolved to make single-source. + +Replaced by tests of `contract/historical.assert_metadata_complete` — the code that actually gates a delivery — parametrised over the **imported** `METADATA_COLS`, plus source-scan pins that the gate stays at build time and is still invoked. Verified 2026-08-02: `pytest -q tests/test_validation.py` → **14 passed**. Mutation-tested: narrowing the gate to a single column fails **9 of 14**; the old suite passed that mutation untouched, because it was not testing the gate. + +**Residual 2 — the enrich→validate end-to-end test — RELOCATED to #18**, per the Register Conventions' relocation rule (a relocation is not complete until the destination exists and is cited by number). Every *leg* is now covered — enrichment (`test_enrichment.py`, 16), artifact build (`test_historical_builder.py`, 7), reader parity (`test_historical_parity.py`, 3), the invariants on primitives (`test_input_integrity_e2e.py`, 8), the wire end to end (`test_hop_b_sink_e2e.py`, 6), the null-gate (`test_validation.py`, 14). What remains uncovered is **the manager orchestrating them**, which needs views-pipeline-core and a production-like Appwrite environment — and þing-02 **D2** forbids integration tests against the production project, no non-production one existing. + +That gap has **two standing trackers already**, which is why keeping a third here is noise rather than signal: issue **#18** (open since 2026-06-04) and `tests/test_falsification_campaign_3_5.py`, an `xfail(strict)` probe that **flips to XPASS the moment someone writes the test** — a self-surfacing tracker, which is more than this entry was doing. | +| Tier | 3 | +| Source | `repo-assimilation` (2026-06-02), `test-review` (2026-06-02) | +| Trigger | When modifying the manager's `_validate()` or the enricher, verify that the test suite covers the changed behavior — end-to-end coverage across the enrich→validate path is still absent | +| Location | `tests/test_validation.py`, `views_postprocessing/unfao/managers/unfao.py` | + +Initial state was zero test coverage. A 73-test suite was written (2026-06-02) covering the (now-deleted) mapper's core guarantees and the validation logic (missing columns, null rejection, error messages). Remaining gaps after the mapper removal: (1) the validation tests replicate `_validate()` logic in a standalone function because `views-pipeline-core` is unavailable in test environments — if the real `_validate()` diverges, tests pass while production fails; (2) no end-to-end test enriches through `GaulLookupEnricher` then validates through the manager. + +Tier recalibrated from 2 to 3 during review-rr (2026-06-02): the gap is maintainability (test-code divergence, missing integration path), not structural fragility. + +**Update 2026-06-24:** narrowed with the mapper deletion (C-39, PR #42). The mapper-coverage dimension is gone with the mapper (`tests/test_mapping.py` deleted; the determinism/cache/shapefile/`ThreadPoolExecutor` gaps no longer exist). Two manager-side gaps remain: the standalone `_validate()` replica and the missing enrich→validate end-to-end test. + +--- + +--- + ### C-43: ADR-011 enrichment swap shipped without its output-equivalence proof — and the proof is now unrecoverable — RESOLVED | Field | Value | @@ -760,7 +773,7 @@ This is a textbook instance of **C-42**'s registered hazard (acting on a mis-sta **Residual (why this entry stays open):** the verification was a one-off session result, not a standing guarantee. Nothing in CI re-runs it, so a future lookup rebuild against a wrong or stale datafactory would ship silently exactly as before. **C-43 closes when `tests/test_gaul_lookup_fidelity.py` is committed and green** — the entry should then cite the test, not the session. Tracked as **Cluster K**; the same test discharges C-59 and C-61. -See also C-03 (the sibling enrich→validate test-coverage gap), C-22 (no post-delivery correction/recall process — **now acute: the consequence path is live**), C-39 / C-31 / C-23 (the resolved mapper-deletion cluster this emerged from), C-30 (coverage — discharged by the same run that left this standing), C-32 / C-34 (RESOLVED — the go-global scale risks that fired cleanly), D-08 (the swap-to-lookup-first decision whose verification debt this is), #131 (run-0 delivery-integrity verification). +See also C-03 (the sibling enrich→validate test-coverage gap — RESOLVED 2026-08-02; its manager-orchestration residual relocated to #18), C-22 (no post-delivery correction/recall process — **now acute: the consequence path is live**), C-39 / C-31 / C-23 (the resolved mapper-deletion cluster this emerged from), C-30 (coverage — discharged by the same run that left this standing), C-32 / C-34 (RESOLVED — the go-global scale risks that fired cleanly), D-08 (the swap-to-lookup-first decision whose verification debt this is), #131 (run-0 delivery-integrity verification). --- diff --git a/tests/test_falsification_campaign_3_2.py b/tests/test_falsification_campaign_3_2.py index 0014fa7..b13f85a 100644 --- a/tests/test_falsification_campaign_3_2.py +++ b/tests/test_falsification_campaign_3_2.py @@ -4,6 +4,16 @@ Generated by falsification campaign on 2026-06-02 READ-ONLY CAMPAIGN: These stubs document test coverage gaps. + +DISCHARGED 2026-08-02 (S4 / #185): `test_falsify_3_2_validation_uses_replica_not_real_code` +is removed — the replica it described is gone. `tests/test_validation.py` now tests +`contract/historical.assert_metadata_complete` directly, parametrised over the imported +`gaul_schema.METADATA_COLS`. Mutation-proven: narrowing the gate to one column fails 9 of +its 14 tests, where the replica-based suite passed that mutation untouched. + +A probe written as `assert False` can never flip to XPASS, so the "surfaced for promotion" +mechanism below cannot reach it — a discharged probe has to be retired by hand, and leaving +one in place asserts a defect that no longer exists. """ import pytest @@ -25,16 +35,3 @@ def test_falsify_3_2_wrong_type_columns_not_tested(): """ assert False, "No test for wrong-type column values (e.g., string in numeric column)" - -def test_falsify_3_2_validation_uses_replica_not_real_code(): - """ - test_validation.py uses a standalone validate_dataframe() function that - replicates the logic from UNFAOPostProcessorManager._validate(). If the - real _validate() changes (e.g., adds a new column to the required list), - the replica won't be updated and tests will pass while production fails. - - This is a structural test-code coupling gap, not a missing test case. - - Severity: Soft falsification - """ - assert False, "Validation tests use replicated logic, not the real _validate() method — divergence risk" diff --git a/tests/test_falsification_campaign_4_3.py b/tests/test_falsification_campaign_4_3.py index 7902984..13638a7 100644 --- a/tests/test_falsification_campaign_4_3.py +++ b/tests/test_falsification_campaign_4_3.py @@ -2,6 +2,23 @@ Falsification test stubs for Campaign Claim 4.3: "Schema change would fail a test" Generated by falsification campaign on 2026-06-02 + +DISCHARGED 2026-08-02 (S4 / #185): `test_falsify_4_3_filter_cols_change_not_detected_by_validation_tests` +is removed. The hardcoded `REQUIRED_METADATA_COLS` replica it described is gone; +`tests/test_validation.py` parametrises over the imported `gaul_schema.METADATA_COLS`, so +adding a column to the declared contract extends the gate's coverage automatically — the +property register C-70 was resolved to give us. + +`test_falsify_4_3_enrichment_test_checks_only_5_of_9_filter_cols` is removed for the same +reason, verified in the same pass. It described `test_enrichment` asserting 5 of the 9 +contract columns. That file now imports `METADATA_COLS` and asserts +`list(lookup.columns) == METADATA_COLS` — exact equality on all nine — then iterates the +full set again for presence and for nulls. The four columns the probe said had "no schema +drift protection" are covered three times over. + +**Claim 4.3 is therefore fully discharged and this module holds no live probe.** It is kept +rather than deleted because the campaign record is the point: it says what was checked on +2026-06-02, what was wrong, and when each finding was closed. """ import pytest @@ -12,28 +29,3 @@ pytestmark = pytest.mark.xfail(reason="falsification probe — asserts a known-open finding by design (see reports/technical_risk_register.md, C-36); xfail(strict) keeps the suite green-when-healthy and surfaces any probe that starts passing", strict=True) -def test_falsify_4_3_enrichment_test_checks_only_5_of_9_filter_cols(): - """ - test_enrichment_adds_expected_columns checks 5 columns: - country_iso_a3, admin1_gaul1_code, admin1_gaul1_name, - admin2_gaul2_code, admin2_gaul2_name. - - But filter_cols in unfao.py has 9 columns (also: pg_xcoord, pg_ycoord, - admin1_gaul0_code, admin1_gaul0_name). Removing the missing 4 from - either side would not fail any test. - - Severity: Soft falsification - """ - assert False, "Enrichment schema test checks 5 of 9 filter_cols — 4 columns have no schema drift protection" - - -def test_falsify_4_3_filter_cols_change_not_detected_by_validation_tests(): - """ - The validation tests use a hardcoded REQUIRED_METADATA_COLS list that - replicates unfao.py's _necessary_metadata_cols. If someone adds or - removes a column from unfao.py's list, the test replica is not - automatically updated — the tests continue to pass against the old list. - - Severity: Soft falsification - """ - assert False, "Validation test's column list is a replica — adding/removing a column in unfao.py is not caught" diff --git a/tests/test_validation.py b/tests/test_validation.py index df791a8..93f0243 100644 --- a/tests/test_validation.py +++ b/tests/test_validation.py @@ -1,147 +1,151 @@ -"""Tests for the metadata validation logic in UNFAOPostProcessorManager._validate(). +"""The metadata null-gate, tested where it actually lives (S4 / #185, register C-03). -Covers the C-01 fix (null validation re-enabled) and CIC failure modes. -See docs/CICs/UNFAOPostProcessorManager.md for the contract under test. +**What this file used to be, and why it had to change.** It opened with: -Note: views-pipeline-core may not be installed in test environments, so we -replicate the validation logic here rather than importing the manager class. -The logic tested matches unfao.py:_validate() exactly. + Note: views-pipeline-core may not be installed in test environments, so we + replicate the validation logic here rather than importing the manager class. + The logic tested matches unfao.py:_validate() exactly. + +Both halves stopped being true. + +*The replica.* The file defined its own ``validate_dataframe`` and then tested that. +Forty-odd parametrised cases exercised a function declared twelve lines above them and +**no production code at all** — a closed loop that could not fail while the delivery +broke, and could not tell a reader anything about the system. + +*The fidelity claim.* ``_validate`` stopped null-gating in #149. Its own docstring now +says so: *"Neither payload is null-gated here."* It asserts the read **resolved** and +calls ``_check_coverage``. The null gate moved to ``contract/historical`` and fires at +artifact-build time. So the file asserted equivalence with a method it no longer +resembled, and had done since 2026-07-31. + +*The column list.* ``REQUIRED_METADATA_COLS`` was a nine-element literal — a hand copy +of the contract that register **C-70** was resolved to make single-source. It is now +imported. + +**What this file is now.** Tests of ``historical.assert_metadata_complete`` — the code +that actually gates a delivery — plus source-scan pins that the gate stays at build time +rather than drifting back into ``_validate``. The original justification for the replica +no longer applies either: ``contract/`` is dependency-light and imports without +views-pipeline-core (``tests/test_clone_readiness.py`` proves it in a subprocess). + +The behaviour C-01 fixed — a null in a required GAUL column must stop the delivery +rather than ship — is preserved in full. It is simply asserted against the function that +performs it. """ -import pytest -import pandas as pd +from __future__ import annotations + +from pathlib import Path + import numpy as np -import logging +import pyarrow as pa +import pytest + +from views_postprocessing.contract import historical +from views_postprocessing.contract.gaul_schema import CODE_COLS, COORD_COLS, METADATA_COLS + +_MANAGER_SOURCE = ( + Path(__file__).resolve().parent.parent + / "views_postprocessing" / "unfao" / "managers" / "unfao.py" +) +_HISTORICAL_SOURCE = ( + Path(__file__).resolve().parent.parent + / "views_postprocessing" / "contract" / "historical.py" +) + +def _artifact(n_rows: int = 5) -> pa.Table: + """A minimal historical artifact shaped like ``build_historical_table``'s output. + + Columns and dtypes come from ``gaul_schema`` (C-70), so this helper cannot drift + from the declared contract — a rename there breaks the import, not a later assert. + """ + columns = { + "month_id": pa.array(np.full(n_rows, 500, dtype=np.int64)), + "priogrid_id": pa.array(np.arange(100_001, 100_001 + n_rows, dtype=np.int64)), + "sb_best": pa.array(np.zeros(n_rows, dtype=np.float32)), + } + for col in METADATA_COLS: + if col in CODE_COLS or col in COORD_COLS: + columns[col] = pa.array(np.arange(n_rows, dtype=np.float64)) + else: + columns[col] = pa.array([f"{col}-value"] * n_rows, type=pa.string()) + return pa.table(columns) + + +def _with_null(col: str, row: int = 0, n_rows: int = 5) -> pa.Table: + """The artifact with one metadata value knocked out — the C-01 defect, reproduced.""" + table = _artifact(n_rows) + values = table.column(col).to_pylist() + values[row] = None + idx = table.column_names.index(col) + return table.set_column(idx, col, pa.array(values, type=table.column(col).type)) -logger = logging.getLogger(__name__) +# ── the gate ──────────────────────────────────────────────────────────────── -REQUIRED_METADATA_COLS = [ - "pg_xcoord", "pg_ycoord", "country_iso_a3", - "admin1_gaul1_code", "admin1_gaul1_name", - "admin1_gaul0_code", "admin1_gaul0_name", - "admin2_gaul2_code", "admin2_gaul2_name", -] +def test_a_complete_artifact_passes(): + historical.assert_metadata_complete(_artifact()) # must not raise -def make_valid_dataframe(n_rows=5): - """Create a DataFrame that passes all validation checks.""" - return pd.DataFrame({ - "pg_xcoord": np.random.uniform(0, 50, n_rows), - "pg_ycoord": np.random.uniform(-10, 10, n_rows), - "country_iso_a3": ["AAA"] * n_rows, - "admin1_gaul1_code": [101] * n_rows, - "admin1_gaul1_name": ["A-South"] * n_rows, - "admin1_gaul0_code": [1] * n_rows, - "admin1_gaul0_name": ["CountryA"] * n_rows, - "admin2_gaul2_code": [1021] * n_rows, - "admin2_gaul2_name": ["A-South-W"] * n_rows, - "pred_value": np.random.uniform(0, 1, n_rows), - }) +@pytest.mark.parametrize("col", METADATA_COLS) +def test_a_null_in_any_required_metadata_column_stops_the_delivery(col): + """C-01: incomplete geographic metadata must never reach the UN FAO. + + Parametrised over the **imported** contract, not a copy — add a column to + ``gaul_schema.COLUMNS`` and it is gated here automatically, which is the property + C-70 was resolved to give us. + """ + with pytest.raises(historical.HistoricalArtifactError): + historical.assert_metadata_complete(_with_null(col)) + + +def test_the_refusal_names_the_column_and_counts_the_nulls(): + """A refusal an operator cannot act on is barely better than no refusal.""" + with pytest.raises(historical.HistoricalArtifactError) as excinfo: + historical.assert_metadata_complete(_with_null("country_iso_a3", n_rows=10)) + message = str(excinfo.value) + assert "country_iso_a3" in message, "the refusal must name the offending column" + assert "1/10" in message, "the refusal must say how much of the artifact is affected" + + +def test_every_declared_metadata_column_is_gated(): + """The gate iterates the declared contract; nothing is gated by coincidence. + + Reads the source on purpose — the parametrised test above proves each column *is* + caught; this proves it is caught **because** the column is declared, rather than + because someone happened to list it twice. + """ + assert "for col in METADATA_COLS:" in _HISTORICAL_SOURCE.read_text(), ( + "assert_metadata_complete must iterate the declared METADATA_COLS. A hand-written " + "column list here is register C-70 returning: one contract, two spellings." + ) + +# ── where the gate lives, and where it must not drift back to ─────────────── -def validate_dataframe(df, label="dataframe"): - """Replicates the validation logic from UNFAOPostProcessorManager._validate(). +def test_the_manager_does_not_null_gate_in_validate(): + """#149 moved the gate to build time. A silent move back would double-gate the + historical leg while leaving the forecast leg — which is arrow, not pandas — with a + gate written for a representation it no longer uses. - This must stay in sync with views_postprocessing/unfao/managers/unfao.py. + Source-scan because the manager needs views-pipeline-core and Appwrite env to + instantiate; this is the repo's standing pattern for manager-side facts. """ - for col in REQUIRED_METADATA_COLS: - if col not in df.columns: - err_msg = f"{label} is missing required metadata column: {col}. Found columns: {df.columns.tolist()}" - logger.error(err_msg) - raise ValueError(err_msg) - null_count = df[col].isnull().sum() - if null_count > 0: - err_msg = f"{label} has {null_count} null values in required metadata column: {col} ({null_count}/{len(df)} rows)." - logger.error(err_msg) - raise ValueError(err_msg) - - -# --------------------------------------------------------------------------- -# GREEN: Valid data passes validation -# --------------------------------------------------------------------------- - -class TestValidDataPasses: - def test_valid_dataframes_pass(self): - validate_dataframe(make_valid_dataframe(), "Historical dataframe") - validate_dataframe(make_valid_dataframe(), "Forecast dataframe") - - def test_extra_columns_are_fine(self): - df = make_valid_dataframe() - df["extra_col"] = "foo" - validate_dataframe(df) - - -# --------------------------------------------------------------------------- -# RED: Missing columns raise ValueError (CIC §6) -# --------------------------------------------------------------------------- - -class TestMissingColumns: - @pytest.mark.parametrize("col", REQUIRED_METADATA_COLS) - def test_missing_historical_column_raises(self, col): - df = make_valid_dataframe().drop(columns=[col]) - with pytest.raises(ValueError, match="missing required metadata column"): - validate_dataframe(df, "Historical dataframe") - - @pytest.mark.parametrize("col", REQUIRED_METADATA_COLS) - def test_missing_forecast_column_raises(self, col): - df = make_valid_dataframe().drop(columns=[col]) - with pytest.raises(ValueError, match="missing required metadata column"): - validate_dataframe(df, "Forecast dataframe") - - -# --------------------------------------------------------------------------- -# RED: Null values raise ValueError (C-01 fix) -# --------------------------------------------------------------------------- - -class TestNullValuesRejected: - @pytest.mark.parametrize("col", REQUIRED_METADATA_COLS) - def test_null_in_historical_column_raises(self, col): - df = make_valid_dataframe() - df.loc[0, col] = None - with pytest.raises(ValueError, match="null values"): - validate_dataframe(df, "Historical dataframe") - - @pytest.mark.parametrize("col", REQUIRED_METADATA_COLS) - def test_null_in_forecast_column_raises(self, col): - df = make_valid_dataframe() - df.loc[0, col] = None - with pytest.raises(ValueError, match="null values"): - validate_dataframe(df, "Forecast dataframe") - - def test_all_null_column_raises(self): - df = make_valid_dataframe() - df["country_iso_a3"] = None - with pytest.raises(ValueError, match="null values"): - validate_dataframe(df) - - def test_nan_treated_as_null(self): - df = make_valid_dataframe() - df.loc[0, "pg_xcoord"] = float("nan") - with pytest.raises(ValueError, match="null values"): - validate_dataframe(df) - - -# --------------------------------------------------------------------------- -# RED: Error messages include useful context (ADR-008) -# --------------------------------------------------------------------------- - -class TestErrorMessages: - def test_missing_col_error_names_the_column(self): - df = make_valid_dataframe().drop(columns=["country_iso_a3"]) - with pytest.raises(ValueError, match="country_iso_a3"): - validate_dataframe(df) - - def test_null_error_includes_count(self): - df = make_valid_dataframe(n_rows=10) - df.loc[0:2, "pg_xcoord"] = None # 3 nulls - with pytest.raises(ValueError, match="3"): - validate_dataframe(df) - - def test_errors_are_logged(self, caplog): - df = make_valid_dataframe().drop(columns=["country_iso_a3"]) - with caplog.at_level(logging.ERROR): - with pytest.raises(ValueError): - validate_dataframe(df) - assert any("country_iso_a3" in r.message for r in caplog.records) + source = _MANAGER_SOURCE.read_text() + validate_body = source.split("def _validate(")[1].split("def _check_coverage(")[0] + for token in ("isnull", "is_null", "null_count", "METADATA_COLS"): + assert token not in validate_body, ( + f"_validate references {token!r} again. The metadata null-gate belongs at " + "artifact build (historical.assert_metadata_complete), where the artifact " + "exists; _validate asserts that the read resolved. See #149." + ) + + +def test_the_manager_still_calls_the_gate_on_the_built_artifact(): + """The other half: having moved, the gate must actually be invoked somewhere.""" + assert "historical.assert_metadata_complete(" in _MANAGER_SOURCE.read_text(), ( + "nothing calls the metadata null-gate — the historical artifact would ship with " + "missing geography and no error signal (register C-01, C-43)." + ) From 0b0f370671787c5c5aee83da73f9a67faebdac16 Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 2 Aug 2026 04:37:02 +0200 Subject: [PATCH 09/23] =?UTF-8?q?S5=20(#186)=20=E2=80=94=20the=20lookup=20?= =?UTF-8?q?declares=20its=20own=20version=20instead=20of=20the=20consumer?= =?UTF-8?q?=20inferring=20it=20(#201)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(lookup): S5 #186 — the lookup declares its own version instead of the consumer inferring it gaul_lookup.version() reconstructed the delivery's traceability stamp by traversing three levels of views-datafactory's ingestion-ledger shape (source_provenance -> land_gaul_region -> content_digest) inside a bare `except (ValueError, AttributeError): pass`, returning the string "unknown" when any level was absent or reshaped. A rename upstream would therefore have made every delivery untraceable, silently, in the one field C-15 exists to answer AFTER a suspect delivery. Nothing would have noticed: a provenance record was still written; it just stopped meaning anything. That is the declare-don't-infer violation ADR-003 forbids, at the consumer. The builder now composes the stamp — it is the only thing holding both the region and the producer's ledger — and writes it as one flat `lookup_version` key. The consumer reads that key and RAISES when it is absent, logging first (ADR-008). The json import and every exception handler are gone from the module. An untraceable build fails at BUILD, not at delivery. _provenance is best-effort by design; a missing ledger used to surface as a placeholder in production, and now surfaces where a human is present to fix it. Review caught the fix reintroducing its own defect class one level up. The first draft read `land_gaul_region or gaul_admin_area_majority`, and since _provenance does not take a region, land_gaul_region is present for EVERY build — so `--region all` would have been stamped `all@f74d3b2b`, the land_gaul region definition's digest, on a global artifact. Authoritative-looking and wrong, silently. Replaced by a declared `stamp_dataset(region)`: area-majority for `all`, `_region` otherwise, and a region whose entry is absent is refused rather than substituted. Verified: all -> all@272cdb01, land_gaul -> land_gaul@f74d3b2b, africa_me_legacy -> refused. The committed artifact is rebuilt. Proven metadata-only: 64,742 rows, column order and dtypes unchanged, a content hash over every column's values IDENTICAL, and the stamp resolves to land_gaul@f74d3b2b — the same string the old traversal produced, so no delivered provenance changes meaning. tests/test_gaul_lookup_fidelity.py (now 26) still compares it value-for-value against the seven datafactory parquets. ADR-013 is untouched: contract_version 1.5, fixtures byte-identical, and the sidecar cannot inherit lookup metadata because build_sidecar constructs a fresh table. Also from review: the ledger is append-only and the LAST entry per dataset wins, which was harmless while provenance was decorative and is load-bearing now that the stamp raises — stated in _provenance rather than left in the loop. Digest length is declared so producer and consumer agree on what a valid stamp is. And the GaulLookupEnricher CIC still documented the "unknown" degrade this removes. Co-Authored-By: Claude Opus 5 (1M context) * chore(register): S5 #186 — resolve C-60, and record that the fix reproduced its own defect The entry closes citing the tests, with the sidecar-stamping residual deferred behind a named trigger (the next ADR-013 version bump) rather than a vague later. Kept visible in the resolution: the first draft of the fix reintroduced C-60's own defect class — a silent fallback that would have stamped `--region all` with the land_gaul region's digest. The reflex that produced the concern reappeared while fixing the concern, and that is worth more to a future reader than a clean narrative. Cluster K now has one entry left: C-46. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) --- docs/CICs/GaulLookupEnricher.md | 13 ++- reports/technical_risk_register.md | 58 ++++++----- scripts/build_gaul_lookup.py | 85 ++++++++++++++-- tests/test_gaul_lookup_access.py | 83 +++++++++++++++- tests/test_gaul_lookup_fidelity.py | 93 +++++++++++++++++- views_postprocessing/contract/gaul_lookup.py | 68 ++++++++----- views_postprocessing/data/gaul_lookup.parquet | Bin 902066 -> 902187 bytes 7 files changed, 337 insertions(+), 63 deletions(-) diff --git a/docs/CICs/GaulLookupEnricher.md b/docs/CICs/GaulLookupEnricher.md index a9d7ef0..4d0938e 100644 --- a/docs/CICs/GaulLookupEnricher.md +++ b/docs/CICs/GaulLookupEnricher.md @@ -62,9 +62,16 @@ views-datafactory area-majority join), so this class does only a table join. - Output: the input frame (or, with `only_metadata=True`, just `pg_id_col` + `time_id_col`) left-merged with the 9 metadata columns. - Public attribute: `lookup_version` — a short, stampable id read from the - lookup's embedded provenance at construction (`@`, or - `"unknown"` if the lookup carries none). The manager stamps it on each - delivery so a delivery is traceable to the exact lookup build. + lookup's **declared** `lookup_version` metadata key at construction + (`@<8-char source digest>`). Delegates to + `contract.gaul_lookup.version`. The manager stamps it on each delivery so a + delivery is traceable to the exact lookup build. + **It does not degrade.** An artifact carrying no declared key raises + `gaul_lookup.LookupVersionError` (logged at ERROR first, per ADR-008) rather + than returning a placeholder. Until S5 (#186) this returned the string + `"unknown"` whenever views-datafactory's ingestion-ledger shape moved under + it — silently, in the one field register C-15 exists to answer *after* a + suspect delivery. See register **C-60**. - Side effects: logs the lookup size + version at construction (INFO); logs a WARNING with the count and sample of unmatched cell ids when any occur; logs ignored mapper-only kwargs at DEBUG. No file writes, no network. diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index b0c1c7c..457a3cc 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -6,8 +6,8 @@ | Owner | Dylan Pinheiro / PRIO MD&D Team | | Last Updated | 2026-08-02 | | Total Concerns | 74 | -| Open Concerns | 21 | -| Resolved Concerns | 53 | +| Open Concerns | 20 | +| Resolved Concerns | 54 | --- @@ -71,7 +71,7 @@ covered a single open entry (see Historical clusters below). **The lesson, and it generalises past this cluster.** C-43's residual said the forward-check *"was a one-off session result, not a standing guarantee"* — and the fix was to attach it to something the interpreter runs. The entries then reproduced the identical error one level up: they stated their closing conditions in prose and attached them to nothing. S2 therefore added a closing-condition check to `test_register_integrity.py`. **A guarantee needs a check, and that applies to the register's own guarantees too.** -**Remaining in this cluster:** **C-60** (the flat declared `lookup_version` key — S5 / #186) and **C-46** (the hardcoded datafactory path — S7 / #188). Both unblocked, both in epic #181. +**C-60 closed 2026-08-02 (S5 / #186)** — the flat declared key landed and the committed artifact was rebuilt, metadata-only. **Remaining in this cluster: C-46 alone** (the hardcoded datafactory path — S7 / #188), which is the last thing keeping the cluster open. ### Cluster L: The won migration was never cleaned up **Root cause:** the frame-native contract path replaced the pandas path and **won** — run-0 delivered global-land on 2026-07-27 and FAO has been served from it since. The replaced path was deliberately kept behind a config fork "until run 0 proves the contract path live" (C-40) and was then never removed. Everything below is residue of that one omission, not independent defects. @@ -454,26 +454,6 @@ Cross-refs: C-57 (registry drift — the most likely way a coordinate goes wrong --- -### C-60: The lookup provenance stamp reaches into the producer's ledger schema and degrades to `"unknown"` on a bare except - -| Field | Value | -|-------|-------| -| ID | C-60 | -| Tier | 3 | -| Source | `expert-code-review` (2026-07-31) — Ousterhout lens (information leakage / silent degradation) | -| Trigger | When views-datafactory renames or restructures its ingestion-ledger entries (`dataset` key, `content_digest` field, or the `land_gaul_region` entry name) — verify `lookup_version` still resolves to a real value rather than the string `"unknown"`; nothing fails if it does not | -| Location | `views_postprocessing/unfao/enrichment.py:61-79` (`_read_version`); written at `scripts/build_gaul_lookup.py:89-107` (`_provenance`) and `:163-164`; consumed as the C-15 provenance field via the manager's delivery description | - -`_read_version` reconstructs the stamp by traversing three levels of the producer's schema — parquet metadata → `source_provenance` JSON → `land_gaul_region` → `content_digest` — and wraps the traversal in `except (ValueError, AttributeError): pass`, returning `"unknown"` when anything along the path is absent. This is a **declare-don't-infer violation at the consumer**: the value that ties a delivered artifact to the exact lookup build (C-15's traceability provenance) can silently become a placeholder, and no gate notices. - -No wrong data results — this is a traceability failure, not a correctness one → **Tier 3**. But it defeats the specific question C-15 exists to answer *after* a suspect delivery ("which lookup produced this?"), and C-22 has no correction procedure that could compensate. - -**Mitigation:** have `build_gaul_lookup.py` write a **flat, declared `lookup_version` key** into the parquet metadata, and have `_read_version` read that one key and **raise** if absent. The consumer stops knowing the producer's nested ledger schema, and the stamp stops being able to vanish quietly. Separately worth stamping `lookup_version` into the sidecar's own parquet metadata so a delivered artifact is self-describing without the store document. - -Cross-refs: C-15 (the provenance this field serves), C-22 (the recall process that would need it), C-57 (the same class — a cross-repo fact this repo reads without a way to detect drift), **Cluster K**. - ---- - ### C-62: The pinned pipeline-core release still installs geopandas and torch into a repo that architecturally excised them | Field | Value | @@ -694,6 +674,38 @@ See also C-40 (the inheritance/representation coupling this migration unwinds), ## Resolved Concerns +### C-60: The lookup provenance stamp reaches into the producer's ledger schema and degrades to `"unknown"` on a bare except — RESOLVED + +| Field | Value | +|-------|-------| +| ID | C-60 | +| Resolved | 2026-08-02 | +| Resolution | **Closed by S5 (#186).** The builder composes the stamp and writes one flat `lookup_version` key; `contract/gaul_lookup.version` reads that key and **raises** `LookupVersionError` (logged first, ADR-008) when it is absent. The three-level traversal of views-datafactory's ledger shape, the bare `except … pass` and the `"unknown"` branch are all gone — pinned by `tests/test_gaul_lookup_access.py`, which asserts by **AST** rather than by word-matching that the module imports no `json` and holds zero exception handlers. An untraceable *build* now fails at build time, where a human is present, instead of surfacing as a placeholder in production. + +**The committed artifact was rebuilt, metadata-only.** 64,742 rows, column order and dtypes unchanged, a content hash over every column's values identical, and the stamp resolves to `land_gaul@f74d3b2b` — the same string the old traversal produced, so no delivered provenance changes meaning. ADR-013 untouched: `contract_version` 1.5, `tests/fixtures/` byte-identical (`build_sidecar` constructs a fresh table, so lookup metadata cannot reach the wire — verified empirically, sidecar schema metadata is `[]`). + +**⚠ The fix reintroduced this entry's own defect class, caught in review before merge.** The first draft read `provenance.get("land_gaul_region") or provenance.get("gaul_admin_area_majority")`. Since `_provenance` takes no region, `land_gaul_region` is present for *every* build — so `--region all` would have been stamped `all@f74d3b2b`, the land_gaul region definition's digest, on a global artifact: authoritative-looking, wrong, silent. Replaced by a declared `stamp_dataset(region)` with no fallback; a region whose ledger entry is absent is **refused**, not substituted. Verified: `all@272cdb01`, `land_gaul@f74d3b2b`, `africa_me_legacy` → raises. Worth keeping visible — the reflex that produced C-60 reappeared while fixing C-60. + +**Residual, deliberately deferred:** stamping `lookup_version` into the **sidecar's own** parquet metadata, so a delivered artifact is self-describing without the store document. Not done here because it changes wire bytes and therefore `contract_version` — an ADR-013 amendment with three repos to notify. Named trigger: **the next ADR-013 version bump**, whatever prompts it. + +**Also recorded from review:** the ingestion ledger is append-only and the **last entry per dataset wins** (14 `gaul_admin_area_majority` entries, 2 `land_gaul_region`). Harmless while provenance was decorative; load-bearing now that the stamp raises, so it is stated in `_provenance` rather than left implicit — the same shape **C-73** was opened for on the forecast path, intended here and now said. `docs/CICs/GaulLookupEnricher.md` still documented the `"unknown"` degrade and was corrected. | +| Tier | 3 | +| Source | `expert-code-review` (2026-07-31) — Ousterhout lens (information leakage / silent degradation) | +| Trigger | When views-datafactory renames or restructures its ingestion-ledger entries (`dataset` key, `content_digest` field, or the `land_gaul_region` entry name) — verify `lookup_version` still resolves to a real value rather than the string `"unknown"`; nothing fails if it does not | +| Location | `views_postprocessing/unfao/enrichment.py:61-79` (`_read_version`); written at `scripts/build_gaul_lookup.py:89-107` (`_provenance`) and `:163-164`; consumed as the C-15 provenance field via the manager's delivery description | + +`_read_version` reconstructs the stamp by traversing three levels of the producer's schema — parquet metadata → `source_provenance` JSON → `land_gaul_region` → `content_digest` — and wraps the traversal in `except (ValueError, AttributeError): pass`, returning `"unknown"` when anything along the path is absent. This is a **declare-don't-infer violation at the consumer**: the value that ties a delivered artifact to the exact lookup build (C-15's traceability provenance) can silently become a placeholder, and no gate notices. + +No wrong data results — this is a traceability failure, not a correctness one → **Tier 3**. But it defeats the specific question C-15 exists to answer *after* a suspect delivery ("which lookup produced this?"), and C-22 has no correction procedure that could compensate. + +**Mitigation:** have `build_gaul_lookup.py` write a **flat, declared `lookup_version` key** into the parquet metadata, and have `_read_version` read that one key and **raise** if absent. The consumer stops knowing the producer's nested ledger schema, and the stamp stops being able to vanish quietly. Separately worth stamping `lookup_version` into the sidecar's own parquet metadata so a delivered artifact is self-describing without the store document. + +Cross-refs: C-15 (the provenance this field serves), C-22 (the recall process that would need it), C-57 (the same class — a cross-repo fact this repo reads without a way to detect drift), **Cluster K**. + +--- + +--- + ### C-03: Test coverage gaps in manager validation and the enrich→validate path — RESOLVED | Field | Value | diff --git a/scripts/build_gaul_lookup.py b/scripts/build_gaul_lookup.py index 7b60b44..b488cdb 100644 --- a/scripts/build_gaul_lookup.py +++ b/scripts/build_gaul_lookup.py @@ -90,8 +90,79 @@ def _region_gids(datafactory: Path, region: str) -> set[int] | None: return set(json.loads(path.read_text())) -def _provenance(datafactory: Path) -> dict: - """Pull the land_gaul ledger entry for traceability (best-effort).""" +#: The ledger dataset naming the whole GAUL area-majority join, used to stamp an +#: unregionalised (``--region all``) build. +AREA_MAJORITY_DATASET = "gaul_admin_area_majority" + +#: Length of the digest carried in a stamp. Declared so producer and consumer agree. +DIGEST_CHARS = 8 + + +def stamp_dataset(region: str) -> str: + """DECLARED: which ledger dataset's digest stamps this region's build. + + One rule, no fallback. ``--region all`` is stamped by the area-majority join that + produced every value; a regional build is stamped by that region's own definition + entry, because the region determines which rows exist and a change to it changes + the artifact. + + **Why not "whichever entry happens to be present" (register C-60, review of #186).** + The first draft read ``land_gaul_region or gaul_admin_area_majority``. Since + ``_provenance`` does not take a region, ``land_gaul_region`` is present for *every* + build — so ``--region all`` would have been stamped ``all@f74d3b2b``, the land_gaul + region definition's digest, on a global artifact. Authoritative-looking and wrong, + silently: the exact defect class C-60 exists to close, reintroduced one level up. + A region with no ledger entry is now refused, not substituted. + """ + return AREA_MAJORITY_DATASET if region == "all" else f"{region}_region" + + +def _lookup_version(region: str, provenance: dict) -> str: + """Compose the DECLARED build stamp: ``@``. + + This is the value written as a flat ``lookup_version`` key and read back verbatim + by ``contract.gaul_lookup.version`` — register **C-60**. Composing it here is the + point: the builder is the only thing holding both the region and the producer's + ledger, so the consumer stops needing to know views-datafactory's schema in order + to answer "which lookup produced this delivery?". + + **Refuses an untraceable build.** ``_provenance`` is best-effort by design — a + missing or reshaped ledger yields ``{}``. Previously that produced the string + ``"unknown"`` at *delivery* time, silently, in the one field C-15 exists to answer + after a suspect delivery. Failing here moves the error to the moment a human is + present to fix it: the build. + """ + dataset = stamp_dataset(region) + digest = str((provenance.get(dataset) or {}).get("content_digest", "")) + if len(digest) < DIGEST_CHARS: + raise LookupBuildError( + f"cannot compose a lookup_version for region {region!r}: the datafactory " + f"ingestion ledger has no usable content_digest for {dataset!r} " + f"(provenance/gaul_admin/ingestion_ledger.jsonl; need at least " + f"{DIGEST_CHARS} characters, got {len(digest)}). An artifact without a " + "declared version cannot be traced back to the build that made it, which " + "is what register C-15 needs after a suspect delivery. Rebuild against a " + "datafactory checkout whose ledger declares that dataset — this build is " + "NOT stamped from some other region's entry." + ) + return f"{region}@{digest[:DIGEST_CHARS]}" + + +def _provenance(datafactory: Path, *, datasets: tuple[str, ...]) -> dict: + """Pull the named ledger entries for traceability (best-effort). + + Kept richer than the stamp — it carries timestamps and the upstream GAUL digest, + which are worth having. What changed in C-60 is that the STAMP is no longer + derived from it by the consumer; see ``_lookup_version``. + + **The ledger is append-only and the LAST entry per dataset wins.** The real file + carries 14 ``gaul_admin_area_majority`` entries and 2 ``land_gaul_region`` ones; + this loop overwrites, so the most recent ingestion is what stamps the build. That + was harmless while provenance was decorative. Since C-60 the stamp *raises* on its + absence, so the selection rule is load-bearing and is stated here rather than left + to be inferred from the loop. (Recency-over-a-broad-match is the shape register + **C-73** was opened for on the forecast path; here it is intended, and now said.) + """ ledger = (datafactory / "provenance" / "gaul_admin" / "ingestion_ledger.jsonl") out = {} @@ -101,8 +172,7 @@ def _provenance(datafactory: Path) -> dict: e = json.loads(line) except json.JSONDecodeError: continue - if e.get("dataset") in ("land_gaul_region", - "gaul_admin_area_majority"): + if e.get("dataset") in datasets: out[e["dataset"]] = { k: e[k] for k in ("content_digest", "source_gaul_digest", "timestamp") @@ -188,8 +258,11 @@ def build(datafactory: Path, region: str, out: Path) -> pd.DataFrame: meta[b"region"] = region.encode() meta[b"n_cells"] = str(len(df)).encode() meta[b"n_dropped_incomplete"] = str(dropped).encode() - meta[b"source_provenance"] = json.dumps( - _provenance(datafactory)).encode() + prov = _provenance(datafactory, datasets=(AREA_MAJORITY_DATASET, stamp_dataset(region))) + # The DECLARED stamp: one flat key, composed here, read verbatim by the consumer + # (C-60). Key order in parquet metadata carries no meaning; this is a dict. + meta[b"lookup_version"] = _lookup_version(region, prov).encode() + meta[b"source_provenance"] = json.dumps(prov).encode() table = table.replace_schema_metadata(meta) pq.write_table(table, out) diff --git a/tests/test_gaul_lookup_access.py b/tests/test_gaul_lookup_access.py index e6be468..d0de6c6 100644 --- a/tests/test_gaul_lookup_access.py +++ b/tests/test_gaul_lookup_access.py @@ -11,9 +11,15 @@ read per run, and the read *injected* into both consumers rather than fetched by each. """ +import ast +import importlib +import inspect +import logging from pathlib import Path import pyarrow as pa +import pyarrow.parquet as pq +import pytest from views_postprocessing.contract import gaul_lookup @@ -34,13 +40,80 @@ def test_load_returns_the_arrow_table(): assert "priogrid_gid" in table.column_names -def test_version_reports_the_build_stamp(): +def test_version_reports_the_declared_build_stamp(): + """C-60: the stamp is read from one declared key, not reconstructed. + + This used to assert ``stamp != "unknown"`` — a real check while ``version()`` + could *return* that placeholder. It no longer can: the branch is gone and the + function raises instead. What is worth pinning now is the shape a delivery + traces by. + """ stamp = gaul_lookup.version() - assert stamp != "unknown", ( - "the lookup's provenance stamp did not resolve — a delivery would ship " - "untraceable provenance (register C-60)" + assert "@" in stamp, f"expected `@`, got {stamp!r}" + region, digest = stamp.split("@", 1) + assert region and len(digest) == 8, stamp + + +def test_version_raises_rather_than_degrading_when_the_key_is_absent(tmp_path): + """The load-bearing half of C-60's fix, without which it is unproven. + + An artifact built before the declared key existed — or by a future builder that + forgets it — must stop the delivery, not hand it a placeholder that reads like a + version. ``"unknown"`` in a provenance record is worse than a crash: it is + written, stored and shipped, and nothing downstream can tell it from a real stamp. + """ + unstamped = tmp_path / "no_version.parquet" + pq.write_table(pa.table({"priogrid_gid": pa.array([1, 2, 3])}), unstamped) + with pytest.raises(gaul_lookup.LookupVersionError, match="lookup_version"): + gaul_lookup.version(unstamped) + + +def test_version_logs_before_it_raises(tmp_path, caplog): + """ADR-008:48/51, matching the shape S1 (#182) gave the entry validators.""" + unstamped = tmp_path / "no_version.parquet" + pq.write_table(pa.table({"priogrid_gid": pa.array([1])}), unstamped) + with caplog.at_level(logging.ERROR): + with pytest.raises(gaul_lookup.LookupVersionError): + gaul_lookup.version(unstamped) + errors = [r for r in caplog.records if r.levelno >= logging.ERROR] + assert len(errors) == 1 and "lookup_version" in errors[0].getMessage() + + +def test_version_no_longer_knows_the_producers_ledger_schema(): + """The declare-don't-infer half: the consumer stopped traversing upstream shape. + + ``version()`` reached three levels into views-datafactory's ingestion ledger — + ``source_provenance`` → ``land_gaul_region`` → ``content_digest`` — inside a bare + ``except … pass``. A rename upstream made every delivery untraceable with no + signal. The builder composes the stamp now; this reads one key. + + **Tested by mechanism, not by word.** An earlier draft scanned the file for those + key names and failed on the docstring above, which names them in order to explain + the fix. Prose that records why a thing changed is not the thing. So this asserts + the two capabilities the traversal required and the degrade depended on: JSON + decoding of the producer's blob, and an exception handler that swallows. Neither + can be present without the defect being reachable, and neither is triggered by a + sentence. + """ + module = importlib.import_module("views_postprocessing.contract.gaul_lookup") + tree = ast.parse(inspect.getsource(module)) + + imports = { + alias.name + for node in ast.walk(tree) + if isinstance(node, ast.Import) + for alias in node.names + } + assert "json" not in imports, ( + "gaul_lookup imports json again — decoding the producer's provenance blob is " + "how the consumer came to know views-datafactory's schema (register C-60)." + ) + + handlers = [n for n in ast.walk(tree) if isinstance(n, ast.ExceptHandler)] + assert not handlers, ( + f"gaul_lookup has {len(handlers)} exception handler(s). The stamp degraded to " + '"unknown" through a bare `except … pass`; this module now declares or raises.' ) - assert "@" in stamp # `@` def test_the_delivery_reads_the_lookup_exactly_once(monkeypatch): diff --git a/tests/test_gaul_lookup_fidelity.py b/tests/test_gaul_lookup_fidelity.py index 78fe669..a451188 100644 --- a/tests/test_gaul_lookup_fidelity.py +++ b/tests/test_gaul_lookup_fidelity.py @@ -315,15 +315,28 @@ def _synthetic_source(n: int = 6): ) -def _build_with(monkeypatch, source, tmp_path): +#: A resolvable ledger entry, so a synthetic build can reach the invariant block. +#: Keyed on the area-majority dataset because ``_build_with`` builds ``region="all"``, +#: whose DECLARED stamp source is that entry — see ``builder.stamp_dataset``. +_SYNTHETIC_PROVENANCE = {"gaul_admin_area_majority": {"content_digest": "0123456789abcdef"}} + + +def _build_with(monkeypatch, source, tmp_path, provenance=None): """Run the builder against ``source``, bypassing the datafactory entirely. - ``region="all"`` short-circuits ``_region_gids`` (returns None), and - ``_provenance`` is best-effort, so no checkout is required. + ``region="all"`` short-circuits ``_region_gids`` (returns None). Provenance is + stubbed for the same reason the source is: since C-60 the builder **refuses** a + build whose ledger yields no digest, so a synthetic run needs a synthetic ledger + to reach the invariants these tests are about. That refusal is not bypassed — + it is proven directly by ``test_builder_refuses_a_build_it_cannot_stamp``. """ import scripts.build_gaul_lookup as builder monkeypatch.setattr(builder, "_load_source", lambda _df: source) + monkeypatch.setattr( + builder, "_provenance", + lambda _df, **_kw: _SYNTHETIC_PROVENANCE if provenance is None else provenance, + ) return builder.build(Path("/nonexistent"), "all", tmp_path / "lookup.parquet") @@ -384,3 +397,77 @@ def test_coord_dtypes_are_wire_stable(lookup): ) for col in COORD_COLS: assert lookup.schema.field(col).type == "double", f"{col} must be float64" + + +# ── the declared build stamp (S5 / #186, register C-60) ───────────────────── + +def test_builder_refuses_a_build_it_cannot_stamp(monkeypatch, tmp_path): + """C-60: an untraceable artifact must fail at BUILD, not degrade at delivery. + + Before this, an unresolvable ledger produced the string ``"unknown"`` in the + delivery's provenance — silently, in the one field C-15 exists to answer after a + suspect delivery. The record was still written; it just stopped meaning anything. + Moving the failure to the build puts it where a human is present to fix it. + """ + import scripts.build_gaul_lookup as builder + + with pytest.raises(builder.LookupBuildError, match="lookup_version"): + _build_with(monkeypatch, _synthetic_source(), tmp_path, provenance={}) + + +def test_the_builder_writes_the_declared_version_key(monkeypatch, tmp_path): + """The producer composes the stamp; nothing downstream reconstructs it.""" + import pyarrow.parquet as pq + + out = tmp_path / "lookup.parquet" + _build_with(monkeypatch, _synthetic_source(), tmp_path) + meta = {k.decode(): v.decode() for k, v in (pq.read_metadata(out).metadata or {}).items()} + assert meta["lookup_version"] == "all@01234567", ( + "the builder must write a flat, declared lookup_version — @" + ) + + +def test_the_committed_artifact_declares_its_version(): + """The artifact this repo actually ships, not a synthetic one.""" + from views_postprocessing.contract import gaul_lookup + + stamp = gaul_lookup.version() + assert stamp.startswith("land_gaul@"), stamp + assert "@" in stamp and len(stamp.split("@")[1]) == 8, ( + f"expected @<8-char digest>, got {stamp!r}" + ) + + +def test_the_stamp_source_is_declared_per_region_never_borrowed(): + """C-60, review of #186: the first draft borrowed whatever entry was present. + + ``_provenance`` does not take a region, so ``land_gaul_region`` is in the blob for + *every* build. A ``land_gaul_region or gaul_admin_area_majority`` fallback would + therefore have stamped ``--region all`` as ``all@`` — an + authoritative-looking claim about a global artifact, sourced from one region's + definition. That is the defect class C-60 exists to close, one level up. + """ + import scripts.build_gaul_lookup as builder + + assert builder.stamp_dataset("all") == builder.AREA_MAJORITY_DATASET + assert builder.stamp_dataset("land_gaul") == "land_gaul_region" + assert builder.stamp_dataset("africa_me_legacy") == "africa_me_legacy_region" + + # A region whose declared entry is absent is REFUSED, not silently substituted — + # even though another dataset's digest is sitting right there in the blob. + blob = {"land_gaul_region": {"content_digest": "f74d3b2bdeadbeef"}} + assert builder._lookup_version("land_gaul", blob) == "land_gaul@f74d3b2b" + with pytest.raises(builder.LookupBuildError, match="africa_me_legacy_region"): + builder._lookup_version("africa_me_legacy", blob) + + +def test_a_short_digest_is_refused_rather_than_truncated_silently(): + """Producer and consumer must agree on what a valid stamp is. + + ``[:8]`` alone would emit ``land_gaul@abcd`` from a 4-char digest — accepted by + the builder, rejected by ``test_the_committed_artifact_declares_its_version``. + """ + import scripts.build_gaul_lookup as builder + + with pytest.raises(builder.LookupBuildError, match="at least 8 characters"): + builder._lookup_version("land_gaul", {"land_gaul_region": {"content_digest": "abcd"}}) diff --git a/views_postprocessing/contract/gaul_lookup.py b/views_postprocessing/contract/gaul_lookup.py index 098d21f..6907f31 100644 --- a/views_postprocessing/contract/gaul_lookup.py +++ b/views_postprocessing/contract/gaul_lookup.py @@ -20,41 +20,63 @@ from __future__ import annotations -import json +import logging from pathlib import Path import pyarrow as pa import pyarrow.parquet as pq +logger = logging.getLogger(__name__) + +#: The single declared metadata key carrying the artifact's build stamp. +#: Written by ``scripts/build_gaul_lookup.py``; read verbatim here (C-60). +VERSION_KEY = "lookup_version" + + +class LookupVersionError(RuntimeError): + """The lookup artifact does not declare which build produced it.""" + #: The committed ADR-011 lookup. Declared public — three modules depend on it, and a #: leading underscore said the opposite (C-68). LOOKUP_PATH: Path = Path(__file__).resolve().parent.parent / "data" / "gaul_lookup.parquet" def version(path: Path | None = None) -> str: - """A short, stampable build id for the lookup, for delivery provenance (C-15). - - Format: ``@`` (e.g. ``land_gaul@f74d3b2b``) so a - delivery can be traced to the exact lookup build. - - **Known limitation, tracked as register C-60:** this reaches three levels into - views-datafactory's ingestion-ledger shape (``source_provenance`` → - ``land_gaul_region`` → ``content_digest``) and returns ``"unknown"`` if any level - is absent or reshaped — so the traceability stamp can silently degrade. Moved - verbatim from ``GaulLookupEnricher._read_version`` in #152 **without behaviour - change**; fixing it means having the builder write a flat declared key and - raising here when it is missing, which is C-60's own scope. + """The lookup's DECLARED build stamp, for delivery provenance (C-15). + + Reads one flat key — ``lookup_version`` — written by the builder, which is the + only thing holding both the region and the producer's ledger. Format: + ``@`` (e.g. ``land_gaul@f74d3b2b``), so a delivery + can be traced to the exact lookup build that made it. + + Raises: + LookupVersionError: if the artifact declares no version. It does not + degrade to a placeholder — see below. + + **Why it raises (register C-60).** This function used to reconstruct the stamp + by traversing three levels of views-datafactory's ingestion-ledger shape + (``source_provenance`` → ``land_gaul_region`` → ``content_digest``) inside a + bare ``except … pass``, returning the string ``"unknown"`` when any level was + absent or reshaped. A rename upstream would therefore have made every delivery + untraceable, silently, in the one field C-15 exists to answer *after* a suspect + delivery — and no gate would have noticed, because a provenance record was + still written; it just stopped meaning anything. + + The consumer no longer knows the producer's schema. The builder declares; this + reads. ADR-003. """ - meta = pq.read_metadata(path or LOOKUP_PATH).metadata or {} - meta = {k.decode(): v.decode() for k, v in meta.items()} - region = meta.get("region", "?") - digest = "?" - try: - prov = json.loads(meta.get("source_provenance", "{}")) - digest = (prov.get("land_gaul_region", {}).get("content_digest", "?"))[:8] - except (ValueError, AttributeError): - pass - return "unknown" if region == "?" and digest == "?" else f"{region}@{digest}" + raw = pq.read_metadata(path or LOOKUP_PATH).metadata or {} + declared = raw.get(VERSION_KEY.encode()) + if not declared: + err_msg = ( + f"the GAUL lookup at {path or LOOKUP_PATH} declares no {VERSION_KEY!r} — " + "a delivery built from it could not be traced back to the build that " + "produced it (register C-15). Rebuild the artifact with " + "scripts/build_gaul_lookup.py, which writes the key (C-60)." + ) + logger.error(err_msg) # ADR-008: logged persistently AND raised + raise LookupVersionError(err_msg) + return declared.decode() def load(path: Path | None = None) -> pa.Table: diff --git a/views_postprocessing/data/gaul_lookup.parquet b/views_postprocessing/data/gaul_lookup.parquet index 9c28ccbad39524532515327ecf1dabddfdc03c3b..307ac497a30e790223fd22d0301d07ce1f509a7c 100644 GIT binary patch delta 2612 zcma)8&5zsE7N^jmg>v}@jYH*37)vNV~^9ZXOdtaXP|%pOLp$L_k5qC8ee>xYK82o6DP4#6=#hGKl~S9E zb*fRimG0N;w3>0iIm%Fx9-a53I;UaR!=sq?kOpz12W(PI;k=6)swf8EUHDekin-h* zPB(y-kzsl0JAzVhmCB~07Pj56vgt+UrX>_M+fgNHTaB_7Z1pyhTHp%hywfV@EupdP z2# zs{t*p2KC8G;7#;tePZ~-SeLhAU8+@9BXG$^w^PUK~J=yCe@~H(M)W?>}J9B+pSLM73==0PyJQ75!sQQw_A480V<`eS5?&{@I(Fl5i5zs5TOkaf8?^=0C7o-ugN|%js$AW;(O6)}y0TQz5Mn;I| zD{D~U1Xq|@_&`;tI)U>>>PJly&i%y z&M=Hhw`_5(d{vKI8S@GME8FVD}^&xp&g)&2i0nd@gxlOa7t~7f9jlyldcnRDIWe0)* zShP~jp4#bds1Q@J#$LHePM`XX?lb38a{3-;=n6_6s>7}wlEsC{RV$n9jRTTfh@C>E z%#1y9_E?!+*(0MzBETP9LT?au+m%E8WX6-e&gA0XfZ{`jVVTrMvs&%oBf=7r@rsvop-TL0(|Ld`j*=F|h=U zko|X`Ja^oUlpS_;pFF|7y+K|(H#M62zp;O20dpE2Iq(1@2Za24FCIERci{s23?$jL zeex)}@E%JJ$tm{W0XfIs|BO7wv;)$|)xGTI0Xfeu9RlA^2V~oC)|1udKPKx4PJf0X zmyOyja|m7PZ}`|p`^4JKz+yG>0A{~qL3t1`%R9W++0fr?t1<`v@^yM+8lM_6Ydp$Y!6cfl^`9J?pH)U`SX)^+_6R*g7%g5N-;gY0FcQs+7{I6)L4dw8CEhB;5l@&tCuRCcB9p$Ln3eA-j%$HoH#1 zcGhc6@@c!;{Zq%KRtptaRV{))=tw(thDxHm)5E8RM}{-Qqr+pvsiqb#mdztt&T(Fg5t> z`3OF}x=JZ6z>CiUr3sp<(=4TRny&M`RO%7FkDHl&{!&2>X^MK3E*C&tlJUeAisg|G zPk>)a)BVel$W@yPAiy`Mj{@_uS8Y`FUd_;poi#mKGb)N}l*NvrZ*AM28(m*vpWGzp zPpOU_7_nkgm9m~DnT{IM8tMw0{gS*gP*)OhwWLPXlJS-*(K&dVwH3cEd%cF`6>DZJ zD_B`EF!kz|6-zXinrK(swpv9O-kOhC+UWP=JYj$4Qu6pIJp%0Rd ziG|FSpsrVWNy7}I0_Amsx&l90GeODDr(82DZdsUtN~&#EiY~IL!u3~i8S3+8Oq1-W z${}GRr(20>u9fTc>7s-73_93X*uYiW(32Zsin#3@Y*O1Rx7KvA3mB;*{J29Q{PM^z ztQJ=mOSGpJsDw1YAwaAF5n>KEwE~{Sqw4tvV65E)$ZV9%N~31P#CBB|x^^JX$=P-j z(gU@w25UOj-`$YiXj@Jzjuq<(7T^pOLC+24JAj8|!_rp~8(qL-n{Dop*{5nA6dPFa^+0hk;Q*Pg zi7uvmZj=92ReN#9rCRlxkyfBquAMTKDq?Kw_;^+|I zIxS;t-kM%+LuCOHg3DocK!)AFOU}-ER!(eGC1dPk7z6uwhn!{3F1f_6+$PVm_6|9D z(gB=xt9d0^iMf4C5F2X1ez{AYV}I_F*V(UoB*DJ8LuT0dT|(I95jn%&*dsGjS*DGM z&)4i*Wc`S&YalsQX8(-H+tZFJz)=tOh=2#o?&}_7OCz$4nzQ>#$JoCk(m&_W0B?~M zSD<8y3*pYmIkREqHMdq_pO44|)IE-B7s2sKiCQVV{S8}S?~cf6Cfp^b*iU!JQTFo= zd6E4wf}lUTL(V)CjKK>390TmN+oUty(!lM75#erUi zyI*WnbD4dBjjP_M=8FJQMX2V=yz=Yu5Djo+Kwb>S=xKvY0L?KuE^HFkffcvI{^FGb zJLYa%O+f@=uaDP|AL7Ve5bVpXXxPH@<9xxbJ!Q<)@-5RhQ=@k#Jh^Qx=epltjkv zk(XI!L{417{k((&2^B(P>f^?I_V+#VQqdoy0wraP1Cqj9640ShjF3N)9NFfvQ(D9g zi6ZSx{3*!XAB9ElkxNmIQUb}V@cN)goxaNMe?z31_s$ Date: Sun, 2 Aug 2026 05:53:01 +0200 Subject: [PATCH 10/23] =?UTF-8?q?test(seam):=20S6=20#187=20=E2=80=94=20det?= =?UTF-8?q?ect=20coordinate-registry=20drift;=20existence=20is=20not=20rea?= =?UTF-8?q?chability=20(#202)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Appwrite Seam Contract's registry is the authority for the 13 environment names this package declares, and it is deliberately referenced, never copied — one owner, no duplicated values. That was always right. What C-57 named was the gap it left: nothing mechanical could tell you the two had diverged. appwrite_env now declares the edition it was verified against, SEAM_CONTRACT_VERSION and SEAM_CONTRACT_COMMIT. A version string and a sha are not coordinate values; what is recorded is WHICH EDITION WAS READ, and that is precisely what makes drift detectable. Bumping them is not bookkeeping — it asserts someone re-checked. Four checks, each mutation-proven with bytecode disabled: names + declared class rename a coordinate here -> 4 failures registry [meta] version pretend v1.2.0 -> 1 failure commit reachable from main pin the withdrawn b54928f -> 1 failure no value copied into code add a constant/dict/kwarg -> 1 failure each The reachability check exists because existence was not enough. #196: S3 pinned a commit resolved with `rev-parse HEAD` on a checkout sitting on an unmerged branch. The commit existed, both cited files existed at it, and every check anyone had written passed — but it had never reached main, declared a version never ratified, and was withdrawn. A pin is a claim about what the contract SAYS; only reachability supports that claim. The value-copy check took two wrong narrowings before the right one. A substring scan flagged three false positives — file_metadata is a FUNCTION NAME here, and the other two appear only in refusal labels and docstrings. Narrowing to assignments and defaults then caught neither a dict value nor a keyword argument, and the keyword argument is the shape this repo would actually produce: AppwriteConfig(bucket_id=...) is how every store is configured. The right axis was exact equality on string constants, not statement shape — it catches all three real shapes while the false positives fall out on their own. When a guard cries wolf, check whether the matching is wrong before assuming the scope is. Review also caught the expectation map deriving class from an _API_KEY suffix — the exact inference the registry forbids, inside the test enforcing the registry. Now written out, with a completeness check so an unclassified name fails. tests/conftest.py holds the sibling-repo resolution S7 (#188, C-46) will adopt for views-datafactory: declared env var, conventional sibling path, never an absolute path to one machine, and a skip that names what to set so the test is runnable rather than merely skippable. Co-authored-by: Claude Opus 5 (1M context) --- reports/technical_risk_register.md | 79 ++++--- tests/conftest.py | 109 ++++++++++ tests/test_env_declaration.py | 238 +++++++++++++++++++++ views_postprocessing/unfao/appwrite_env.py | 17 +- 4 files changed, 415 insertions(+), 28 deletions(-) create mode 100644 tests/conftest.py diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 457a3cc..54459bb 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -6,8 +6,8 @@ | Owner | Dylan Pinheiro / PRIO MD&D Team | | Last Updated | 2026-08-02 | | Total Concerns | 74 | -| Open Concerns | 20 | -| Resolved Concerns | 54 | +| Open Concerns | 19 | +| Resolved Concerns | 55 | --- @@ -404,30 +404,6 @@ See also C-36 (the resolved strict-xfail conversion this extends), C-44 (the dat --- -### C-57: PLATFORM-001 coordinate registry is referenced by URL, so nothing detects drift between it and this repo's declared environment - -| Field | Value | -|-------|-------| -| ID | C-57 | -| Tier | 3 | -| Source | `manual` (2026-07-31) — review-rr blind-spot analysis, following the þing-01 verdict (`orð_dómr.md`, ratified as amended 2026-07-28) | -| Trigger | When views-appwrite amends `coordinate_registry.toml` — renames a coordinate, retires the legacy secret slot in favour of `APPWRITE_{READ,WRITE,PROVISION}_API_KEY`, or adds a target — verify `views_postprocessing/unfao/appwrite_env.py` still matches. Nothing mechanical will tell you: the registry is deliberately **referenced, never copied**, and the two live in different repositories | -| Location | `views_postprocessing/unfao/appwrite_env.py` (`CONNECTION_ENV`, `PROD_FORECASTS_ENV`, `UNFAO_ENV`); views-appwrite `docs/ADRs/platform/coordinate_registry.toml` (the authority); `tests/test_env_declaration.py` (guards this repo's half only); `docs/ADRs/013_sampled_forecast_wire_contract.md` §7(d) (the URL reference) | - -The þing-01 assembly (D1) settled that the PLATFORM-001 contract is **homed in views-appwrite and referenced by URL, never by copy** — a deliberate and correct choice: copies were the platform's original disease (sáttmál S6, the copy-chain this repo's own `load_dotenv` borrow was the runtime edge of, killed in #134/PR #137). But referencing-not-copying moves the failure mode rather than removing it: **the registry can now change without this repo noticing.** - -This repo's half is well guarded. `tests/test_env_declaration.py` pins that every `APPWRITE_*` name the manager reads is declared, that all three store paths validate before constructing an `AppwriteConfig`, that empty-string counts as missing, and that exactly one declared name is a secret by the D3 suffix rule. **What no test can see is the other side of the reference** — whether `coordinate_registry.toml` still spells the coordinates the way `appwrite_env.py` does. Divergence surfaces at runtime as a fail-loud `EnvironmentError` from `assert_env_declared` (good — that is D6 working), but only on a delivery run, and only after the launcher has already been reconfigured. - -Two named changes are already anticipated and will fire this trigger: the **retirement of the legacy `APPWRITE_DATASTORE_API_KEY`** in favour of the three-tier read/write/provision slots (D4), and any target-coordinate addition for the second store (issue #97). Tier 3 — coordination and cost-of-change across a repo boundary; the failure is loud, not silent, and D6's entry validation is the backstop that keeps it that way. - -**Deliberately out of scope here:** the þing-01 redaction clause is mechanically enforced (`tests/test_redaction_guard.py` — the delivery modules stay credential-blind and the provenance description is a closed keyset), and D2's ruling that **integration tests against the production Appwrite project are FORBIDDEN** (no non-production project exists) is a standing prohibition, not a drift risk. - -**⚠ CORRECTED 2026-08-02.** The word *already* above overclaimed: **C-74** showed that guard scans one of its five declared roots, four having pointed at paths that #153 moved. The keyset half is enforced; the credential-blindness half is enforced over `delivery/` only until C-74 lands. - -Cross-refs: C-74 (the guard this paragraph vouched for), C-33 (store identity still hardcoded per store — the same env surface, different concern), C-58 (what happens when a coordinate is wrong rather than missing), C-44 (the pipeline-core version coupling that would carry a registry change), issues #134/#135/#138 (this repo's discharged þing-01 obligations), #104 (README env block placeholders). - ---- - ### C-58: A wrong Appwrite coordinate auto-provisions a new empty target instead of raising — both client lineages, on every write | Field | Value | @@ -674,6 +650,57 @@ See also C-40 (the inheritance/representation coupling this migration unwinds), ## Resolved Concerns +### C-57: PLATFORM-001 coordinate registry is referenced by URL, so nothing detects drift between it and this repo's declared environment — RESOLVED + +| Field | Value | +|-------|-------| +| ID | C-57 | +| Resolved | 2026-08-02 | +| Resolution | **Closed by S6 (#187).** The registry stays **referenced, never copied** — that was always the right call. What C-57 named was the gap it left: nothing mechanical could tell you the two had diverged. There is now a detector, built while the answer was known-good rather than after a failed delivery. + +`appwrite_env` declares the edition it was verified against — `SEAM_CONTRACT_VERSION = "1.3.0"` and `SEAM_CONTRACT_COMMIT = "47172af"`. **A version string and a sha are not coordinate values**; what is recorded is *which edition was read*, which is precisely what makes drift detectable. Four checks in `tests/test_env_declaration.py`, each mutation-proven: + +| check | mutation | result | +|---|---|---| +| every declared name exists, with the class the **registry declares** (never inferred from a prefix) | rename `APPWRITE_UNFAO_BUCKET_ID` here | 4 failures | +| the registry's `[meta] version` matches the pin | pretend v1.2.0 | 1 failure, naming the version to re-verify against | +| the pinned commit is **reachable from views-appwrite's `main`** | pin the withdrawn `b54928f` | 1 failure | +| no coordinate value is baked into a constant or default | add `UNFAO_BUCKET = "unfao_bucket"` | 1 failure | + +**The reachability check exists because existence was not enough.** #196: S3 pinned a commit resolved with `rev-parse HEAD` on a checkout sitting on an unmerged branch. The commit existed, both cited files existed at it, every check anyone had written passed — and it had never reached `main`, declared a version never ratified, and was withdrawn. A pin is a claim about what the contract *says*; only reachability supports that claim. + +**The value-copy check took two wrong narrowings before the right one, and both are worth recording.** A substring text scan flagged three "leaks": `file_metadata` (a **function name** in `contract/store_metadata.py`), `production_forecasts` and `unfao_bucket` (only in refusal labels and docstrings naming which store a function serves). None was a copy, and a guard that fails on `def file_metadata(record)` gets deleted — after which the real rule is unguarded. + +Narrowing to *assignments and default arguments* then failed the other way: it caught neither a dict value nor a keyword argument, and the keyword argument is the shape this repo would actually produce — `AppwriteConfig(bucket_id=os.getenv(...))` is how every store is configured, and swapping one `os.getenv` for a literal there is the violation. Verified: that draft caught **zero** of the two. + +The right axis was **exact equality on string constants**, not statement shape. It catches dict values, keyword arguments and constants alike, while all three original false positives fall out on their own: a function name is not a `Constant`; `"unfao_bucket datastore"` is not equal to `"unfao_bucket"`; docstrings are excluded outright. The lesson is narrow and reusable: **when a guard cries wolf, check whether the matching is wrong before assuming the scope is.** + +**Gated, and honestly so.** The checks need a views-appwrite checkout and skip without one, naming `VIEWS_APPWRITE` and the conventional sibling path so a contributor can run them rather than merely watch them skip. The would-catch-a-rename proof runs in CI with no checkout at all. Resolution helper shared with **C-46** (S7) in `tests/conftest.py` — the second incident, which is this repo's named trigger for extracting. + +**Residual:** the gated half does not run in CI, which needs a views-appwrite checkout in the workflow — a CI-cost and cross-repo-coupling decision, not a code fix. Same shape as **C-46**'s residual and worth deciding once for both (S7 / #188). + +A second, smaller instance of the same shape: these checks parse TOML with `tomllib`, stdlib from Python 3.11, and `pyproject` declares `>=3.11`. CI runs 3.11 and executes them. The maintainer's box runs **3.10**, below the declared floor, so they skip there — the local suite is quietly weaker than a green `pytest -q` suggests. Not a repo defect and not worth its own entry; recorded because "a gate that does not run" is exactly what C-46 is open for, and the CI decision should cover both. | +| Tier | 3 | +| Source | `manual` (2026-07-31) — review-rr blind-spot analysis, following the þing-01 verdict (`orð_dómr.md`, ratified as amended 2026-07-28) | +| Trigger | When views-appwrite amends `coordinate_registry.toml` — renames a coordinate, retires the legacy secret slot in favour of `APPWRITE_{READ,WRITE,PROVISION}_API_KEY`, or adds a target — verify `views_postprocessing/unfao/appwrite_env.py` still matches. Nothing mechanical will tell you: the registry is deliberately **referenced, never copied**, and the two live in different repositories | +| Location | `views_postprocessing/unfao/appwrite_env.py` (`CONNECTION_ENV`, `PROD_FORECASTS_ENV`, `UNFAO_ENV`); views-appwrite `docs/ADRs/platform/coordinate_registry.toml` (the authority); `tests/test_env_declaration.py` (guards this repo's half only); `docs/ADRs/013_sampled_forecast_wire_contract.md` §7(d) (the URL reference) | + +The þing-01 assembly (D1) settled that the PLATFORM-001 contract is **homed in views-appwrite and referenced by URL, never by copy** — a deliberate and correct choice: copies were the platform's original disease (sáttmál S6, the copy-chain this repo's own `load_dotenv` borrow was the runtime edge of, killed in #134/PR #137). But referencing-not-copying moves the failure mode rather than removing it: **the registry can now change without this repo noticing.** + +This repo's half is well guarded. `tests/test_env_declaration.py` pins that every `APPWRITE_*` name the manager reads is declared, that all three store paths validate before constructing an `AppwriteConfig`, that empty-string counts as missing, and that exactly one declared name is a secret by the D3 suffix rule. **What no test can see is the other side of the reference** — whether `coordinate_registry.toml` still spells the coordinates the way `appwrite_env.py` does. Divergence surfaces at runtime as a fail-loud `EnvironmentError` from `assert_env_declared` (good — that is D6 working), but only on a delivery run, and only after the launcher has already been reconfigured. + +Two named changes are already anticipated and will fire this trigger: the **retirement of the legacy `APPWRITE_DATASTORE_API_KEY`** in favour of the three-tier read/write/provision slots (D4), and any target-coordinate addition for the second store (issue #97). Tier 3 — coordination and cost-of-change across a repo boundary; the failure is loud, not silent, and D6's entry validation is the backstop that keeps it that way. + +**Deliberately out of scope here:** the þing-01 redaction clause is mechanically enforced (`tests/test_redaction_guard.py` — the delivery modules stay credential-blind and the provenance description is a closed keyset), and D2's ruling that **integration tests against the production Appwrite project are FORBIDDEN** (no non-production project exists) is a standing prohibition, not a drift risk. + +**⚠ CORRECTED 2026-08-02.** The word *already* above overclaimed: **C-74** showed that guard scans one of its five declared roots, four having pointed at paths that #153 moved. The keyset half is enforced; the credential-blindness half is enforced over `delivery/` only until C-74 lands. + +Cross-refs: C-74 (the guard this paragraph vouched for), C-33 (store identity still hardcoded per store — the same env surface, different concern), C-58 (what happens when a coordinate is wrong rather than missing), C-44 (the pipeline-core version coupling that would carry a registry change), issues #134/#135/#138 (this repo's discharged þing-01 obligations), #104 (README env block placeholders). + +--- + +--- + ### C-60: The lookup provenance stamp reaches into the producer's ledger schema and degrades to `"unknown"` on a bare except — RESOLVED | Field | Value | diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 0000000..dfee96c --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,109 @@ +"""Shared test fixtures — currently just one: how to find a sibling repository. + +**Why this exists (S6 / #187, S7 / #188; register C-46, C-57).** Three places needed a +views-platform sibling checkout and each found it differently: + +- ``scripts/build_gaul_lookup.py`` resolved ``$VIEWS_DATAFACTORY``, then the sibling + directory, with a ``--datafactory`` override and a fail-loud message naming both — + the shape that has been working in production for weeks; +- ``tests/test_gaul_lookup_fidelity.py`` had its own ``skipif``; +- ``tests/test_datafactory_deploy_readiness.py`` hardcoded an **absolute path to one + developer's laptop**, so the only cross-repo release gate in the repo has never run + anywhere else — register **C-46**. + +This is the second incident, which is this repo's named trigger for extracting. The +shape is not guessed: it is the one the build script already runs. + +**Deliberately not a package import.** The siblings are separate repositories, not +dependencies — this repo declares three (`views-pipeline-core`, `views-frames`, +`pyarrow`) and none of them is a sibling checkout. What is being located is a +*working copy on disk*, for tests that compare this repo against another repo's +current state. If that ever becomes a runtime need, it is a different problem with a +different answer (a published artifact, not a path). +""" + +from __future__ import annotations + +import os +import subprocess +from pathlib import Path + +import pytest + +_REPO = Path(__file__).resolve().parent.parent + +#: repo name -> the environment variable that overrides its location. +#: Declared, never derived from the name: ``views-datafactory`` → ``VIEWS_DATAFACTORY`` +#: happens to be mechanical, but a future sibling need not follow the pattern and +#: guessing it would be the inference ADR-003 forbids. +SIBLING_ENV = { + "views-datafactory": "VIEWS_DATAFACTORY", + "views-appwrite": "VIEWS_APPWRITE", +} + + +def sibling_repo(name: str) -> Path | None: + """The checkout of ``name``, or ``None`` if it is not resolvable. + + Order: the declared environment variable, then the conventional sibling directory + **relative to this repository** — never an absolute path to a particular machine. + + Returns ``None`` rather than raising so callers can skip; a missing sibling is a + normal condition in CI, where only this repo is checked out. + """ + if name not in SIBLING_ENV: + raise KeyError( + f"no environment variable declared for sibling {name!r}; add it to " + f"SIBLING_ENV rather than guessing one from the name" + ) + override = os.environ.get(SIBLING_ENV[name]) + candidate = Path(override) if override else _REPO.parent / name + return candidate if candidate.exists() else None + + +def require_sibling(name: str) -> Path: + """``sibling_repo`` or skip, with a message naming what to set. + + A skip that says only "checkout not present" tells a contributor nothing. This one + tells them the variable and the conventional location, so the test is runnable + rather than merely skippable. + """ + path = sibling_repo(name) + if path is None: + pytest.skip( + f"{name} checkout not found — set {SIBLING_ENV[name]}=/path/to/{name}, " + f"or place it alongside this repo at {(_REPO.parent / name)}" + ) + return path + + +def git_output(repo: Path, *args: str) -> str: + """Run a read-only git command in ``repo``; empty string on any failure.""" + result = subprocess.run( + ["git", "-C", str(repo), *args], + capture_output=True, text=True, check=False, timeout=30, + ) + return result.stdout.strip() + + +def commit_is_on_main(repo: Path, commit: str) -> bool: + """Is ``commit`` an ancestor of ``origin/main`` (falling back to ``main``)? + + **Existence is not reachability, and that distinction cost a PR** (#196). S3 pinned + the Appwrite Seam Contract at a commit resolved with ``rev-parse HEAD`` on a + checkout that happened to be sitting on an unmerged feature branch. The commit + existed, both cited files existed at it, and every check anyone had written passed + — but it had never reached ``main`` and was withdrawn days later. + + A pin is a claim about what the contract *says*. Only reachability from ``main`` + supports that claim. + """ + for ref in ("origin/main", "main"): + if not git_output(repo, "rev-parse", "--verify", "--quiet", ref): + continue + result = subprocess.run( + ["git", "-C", str(repo), "merge-base", "--is-ancestor", commit, ref], + capture_output=True, check=False, timeout=30, + ) + return result.returncode == 0 + return False diff --git a/tests/test_env_declaration.py b/tests/test_env_declaration.py index 17a23ba..85bae19 100644 --- a/tests/test_env_declaration.py +++ b/tests/test_env_declaration.py @@ -14,15 +14,19 @@ pinned by source scan instead — the repo's standing pattern. """ +import ast import logging import re from pathlib import Path import pytest +from tests.conftest import commit_is_on_main, git_output, require_sibling, sibling_repo from views_postprocessing.contract import launch_config from views_postprocessing.unfao import appwrite_env +_PKG = Path(__file__).resolve().parent.parent / "views_postprocessing" + _MANAGER_SOURCE = ( Path(__file__).resolve().parent.parent @@ -192,3 +196,237 @@ def test_secret_env_names_follow_the_seam_contract_naming_rule(): ) secrets = {n for n in declared if n.endswith(("_API_KEY", "_PASSWORD", "_TOKEN"))} assert secrets == {"APPWRITE_DATASTORE_API_KEY"} + + +# ── drift against the Appwrite Seam Contract's registry (S6 / #187, C-57) ──── +# +# The registry is the authority for every name below and lives in views-appwrite. It +# is deliberately **referenced, never copied** — one owner, no duplicated values, no +# drift-by-fork — and that is the right call. What it leaves is C-57's gap: nothing +# mechanical tells you when the two have diverged. +# +# Checked 2026-08-02: there is no live drift. This builds the detector while the +# answer is known-good, which is the cheap moment; the alternative is discovering it +# during a failed delivery. +# +# **Never assert on a coordinate VALUE.** The registry holds non-secret identifiers, +# and copying one into a test is the same violation as copying it into code. These +# compare names, declared classes, and the edition — nothing else. + +_REGISTRY_RELPATH = Path("docs") / "ADRs" / "platform" / "coordinate_registry.toml" + +#: How this module treats each declared name, vs. the registry's own `class` field. +#: +#: **Written out, not derived.** An earlier draft built this by suffix — anything ending +#: `_API_KEY` is a secret, everything else in CONNECTION_ENV is a connection. That is the +#: precise inference the registry's header forbids ("class is DECLARED here, never +#: inferred from a name's prefix"), reproduced inside the test written to enforce it. +#: It happened to be correct, which is what makes the habit worth breaking rather than +#: excusing. Adding a name without classifying it now fails below. +_EXPECTED_CLASS = { + "APPWRITE_ENDPOINT": "connection", + "APPWRITE_DATASTORE_PROJECT_ID": "connection", + "APPWRITE_DATASTORE_API_KEY": "secret", + "APPWRITE_PROD_FORECASTS_BUCKET_ID": "target", + "APPWRITE_PROD_FORECASTS_BUCKET_NAME": "target", + "APPWRITE_PROD_FORECASTS_COLLECTION_ID": "target", + "APPWRITE_PROD_FORECASTS_COLLECTION_NAME": "target", + "APPWRITE_UNFAO_BUCKET_ID": "target", + "APPWRITE_UNFAO_BUCKET_NAME": "target", + "APPWRITE_UNFAO_COLLECTION_ID": "target", + "APPWRITE_UNFAO_COLLECTION_NAME": "target", + "APPWRITE_METADATA_DATABASE_ID": "target", + "APPWRITE_METADATA_DATABASE_NAME": "target", +} + + +def test_every_declared_name_is_classified_here(): + """The map above must cover the module exactly — no silent gaps, no strays. + + Without this, adding a name to one of the ENV tuples would simply not be checked + against the registry, and the drift test would keep passing while covering less. + That is register **C-74**'s shape: a guard quietly narrower than it claims. + """ + declared = set( + appwrite_env.CONNECTION_ENV + appwrite_env.PROD_FORECASTS_ENV + appwrite_env.UNFAO_ENV + ) + assert set(_EXPECTED_CLASS) == declared, ( + f"unclassified: {sorted(declared - set(_EXPECTED_CLASS))}; " + f"stale: {sorted(set(_EXPECTED_CLASS) - declared)}" + ) + + +def _load_registry(repo: Path) -> dict: + tomllib = pytest.importorskip( + "tomllib", + reason="tomllib is stdlib from Python 3.11; pyproject declares >=3.11, so a " + "conforming environment has it. CI runs 3.11.", + ) + return tomllib.loads((repo / _REGISTRY_RELPATH).read_text()) + + +def _declared_classes(registry: dict) -> dict[str, str]: + """name -> the class the registry DECLARES for it (never inferred from the name).""" + return { + name: body.get("class") + for section in ("connection", "target", "secret") + for name, body in registry.get(section, {}).items() + } + + +def test_every_declared_name_exists_in_the_registry_with_the_class_we_treat_it_as(): + """C-57: a rename or reclassification upstream must not be silent here.""" + repo = require_sibling("views-appwrite") + declared = _declared_classes(_load_registry(repo)) + + missing = sorted(n for n in _EXPECTED_CLASS if n not in declared) + assert not missing, ( + f"names this package requires are absent from the Appwrite Seam Contract's " + f"registry: {missing}. Either the registry retired them or this module invented " + "them; the registry is the authority." + ) + misclassified = { + n: (expected, declared[n]) + for n, expected in _EXPECTED_CLASS.items() + if declared[n] != expected + } + assert not misclassified, ( + f"class mismatch (expected, registry) {misclassified}. Class is DECLARED by the " + "registry, never inferred from a name's prefix — a coordinate treated as a " + "secret (or the reverse) is a redaction bug waiting to happen." + ) + + +def test_the_pinned_contract_edition_still_matches_the_registry(): + """The check that catches everything the other one cannot — including additions. + + Names and classes catch a rename. The edition catches **any** other change: a new + target this repo ought to adopt, a retired secret slot, a reworded rule. It fails + loudly and tells you what to do rather than what broke. + """ + repo = require_sibling("views-appwrite") + actual = _load_registry(repo)["meta"]["version"] + assert actual == appwrite_env.SEAM_CONTRACT_VERSION, ( + f"the Appwrite Seam Contract's registry moved to v{actual}; this repo declares " + f"v{appwrite_env.SEAM_CONTRACT_VERSION}. Re-verify appwrite_env's declaration " + f"against v{actual}, then bump SEAM_CONTRACT_VERSION and SEAM_CONTRACT_COMMIT " + "together. Do not bump one alone — the pair is the claim." + ) + + +def test_the_pinned_commit_is_reachable_from_the_contract_repos_main(): + """Existence is not reachability, and that distinction cost a merged PR (#196). + + S3 pinned a commit resolved with ``rev-parse HEAD`` on a views-appwrite checkout + that happened to be sitting on an unmerged branch. The commit existed. Both cited + files existed at it. Every check anyone had written passed. It had never reached + ``main``, declared a version that was never ratified, and was withdrawn. + """ + repo = require_sibling("views-appwrite") + commit = appwrite_env.SEAM_CONTRACT_COMMIT + if not git_output(repo, "cat-file", "-t", commit): + pytest.skip( + f"{commit} is not in the local views-appwrite checkout — run `git fetch` " + "there; a stale clone cannot answer whether the pin reached main" + ) + assert commit_is_on_main(repo, commit), ( + f"the pinned commit {commit!r} is not an ancestor of views-appwrite's main. A " + "pin taken from a working copy's HEAD can land on an unmerged branch — that is " + "#196, verbatim. Re-pin from `git rev-parse --short origin/main`." + ) + + +def test_the_drift_check_would_catch_a_rename(tmp_path): + """A gated test that cannot fail is decoration — so prove this one bites in CI. + + Runs with **no** views-appwrite checkout: a synthetic registry with one name + renamed and one reclassified, fed to the same comparison the gated tests use. + """ + registry = { + "meta": {"version": appwrite_env.SEAM_CONTRACT_VERSION}, + "connection": {"APPWRITE_ENDPOINT": {"class": "connection"}}, + "target": {"APPWRITE_UNFAO_BUCKET_ID": {"class": "secret"}}, # reclassified + "secret": {"APPWRITE_DATASTORE_API_KEY": {"class": "secret"}}, + } + declared = _declared_classes(registry) + + assert "APPWRITE_DATASTORE_PROJECT_ID" not in declared, "fixture should omit it" + missing = sorted(n for n in _EXPECTED_CLASS if n not in declared) + assert missing, "the detector reported no missing names against a registry that omits most" + + mismatched = [ + n for n, expected in _EXPECTED_CLASS.items() + if n in declared and declared[n] != expected + ] + assert "APPWRITE_UNFAO_BUCKET_ID" in mismatched, ( + "a target reclassified as a secret went unnoticed — that is the case where " + "getting it wrong leaks or hides a value" + ) + + +def _docstring_nodes(tree: ast.AST) -> set[int]: + """ids of the string Constants that are docstrings — prose, not values.""" + out = set() + for node in ast.walk(tree): + if isinstance(node, (ast.Module, ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)): + first = node.body[0] if node.body else None + if ( + isinstance(first, ast.Expr) + and isinstance(first.value, ast.Constant) + and isinstance(first.value.value, str) + ): + out.add(id(first.value)) + return out + + +def test_no_coordinate_value_is_copied_into_this_repo(): + """The registry's own rule: *"never bake a value into code, an example, or a + dataclass default."* Consumers READ and VALIDATE; the launcher supplies values. + + **Two wrong narrowings before this one, both instructive.** + + A substring text scan over every ``.py`` flagged three "leaks": ``file_metadata`` + (a **function name** in ``contract/store_metadata.py``), ``production_forecasts`` + and ``unfao_bucket`` (only in refusal labels and docstrings naming which store a + function serves). None was a copy, and a guard that fails on + ``def file_metadata(record)`` gets deleted — after which the real rule is unguarded. + + Narrowing to *assignments and default arguments* then went too far in the other + direction: it caught neither a dict value nor a keyword argument, and the keyword + argument is the shape this repo would actually produce — + ``AppwriteConfig(bucket_id=...)`` is how every store is configured, and swapping one + ``os.getenv`` for a literal there is the violation. + + The right axis was **exact equality on string constants**, not statement shape. It + catches dict values and kwargs, while all three original false positives fall out on + their own: a function name is not a ``Constant``; ``"unfao_bucket datastore"`` is not + equal to ``"unfao_bucket"``; docstrings are excluded outright. + """ + repo = sibling_repo("views-appwrite") + if repo is None: + pytest.skip("views-appwrite checkout not found — set VIEWS_APPWRITE") + registry = _load_registry(repo) + values = { + body["value"] + for section in ("connection", "target") + for body in registry.get(section, {}).values() + if isinstance(body.get("value"), str) and len(body["value"]) > 6 + } + + copied = [] + for source in sorted(_PKG.rglob("*.py")): + tree = ast.parse(source.read_text()) + docstrings = _docstring_nodes(tree) + for node in ast.walk(tree): + if ( + isinstance(node, ast.Constant) + and isinstance(node.value, str) + and node.value in values + and id(node) not in docstrings + ): + copied.append(f"{source.relative_to(_PKG)}:{node.lineno} = {node.value!r}") + assert not copied, ( + f"coordinate value(s) from the registry are copied into code: {copied}. The " + "registry is referenced, never copied — values reach this package through the " + "environment the launcher assembles, validated by assert_env_declared." + ) diff --git a/views_postprocessing/unfao/appwrite_env.py b/views_postprocessing/unfao/appwrite_env.py index a9e1597..934f372 100644 --- a/views_postprocessing/unfao/appwrite_env.py +++ b/views_postprocessing/unfao/appwrite_env.py @@ -7,8 +7,10 @@ https://github.com/views-platform/views-appwrite/blob/47172af/docs/ADRs/platform/coordinate_registry.toml -That pin is registry **v1.3.0** (ratified, þing-02). A pinned URL does not rot, but -it does go stale — naming the version is what makes the staleness legible here. +That pin is registry **v1.3.0** (ratified, þing-02) — declared below as +``SEAM_CONTRACT_VERSION`` / ``SEAM_CONTRACT_COMMIT`` so the pin is a value a test can +check rather than a fact buried in prose. A pinned URL does not rot, but it does go +stale, and nothing in this repository could previously tell you it had (register C-57). **Pin from the tip of `main`, never from a sibling checkout's `HEAD`** (#196). The first pin taken here was resolved with `git -C ../views-appwrite rev-parse HEAD` on a @@ -34,6 +36,17 @@ logger = logging.getLogger(__name__) +#: The Appwrite Seam Contract edition these names were verified against, and the commit +#: this repo cites. **A version string and a sha are not coordinate values** — the +#: registry forbids copying its values, and nothing here copies one. What is recorded is +#: *which edition was read*, which is exactly what makes drift detectable. +#: +#: Bumping these is not bookkeeping: it asserts that someone re-checked this module's +#: declaration against that edition of the registry. ``tests/test_env_declaration.py`` +#: enforces the pair against a local views-appwrite checkout when one is present. +SEAM_CONTRACT_VERSION = "1.3.0" +SEAM_CONTRACT_COMMIT = "47172af" + CONNECTION_ENV = ( "APPWRITE_ENDPOINT", "APPWRITE_DATASTORE_PROJECT_ID", From af5a894af932db5ab6f7bbeba1697b2feae19d86 Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 2 Aug 2026 06:21:29 +0200 Subject: [PATCH 11/23] =?UTF-8?q?test(siblings):=20S7=20#188=20=E2=80=94?= =?UTF-8?q?=20one=20declared=20way=20to=20find=20views-datafactory;=20Clus?= =?UTF-8?q?ter=20K=20closes=20(#203)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four sites resolved the producer's checkout four different ways, and one of them was an absolute path to a single developer's home directory. The repo's only cross-repo release gate had therefore never run anywhere but that laptop — which is what register C-46 was open for, and why "it is skipped in CI" understated it. All four now resolve the same way: $VIEWS_DATAFACTORY, else the conventional directory beside this repo, never an absolute path to a particular machine. Every skip names the variable AND the path, so a contributor can run the test rather than watch it skip. A fourth site turned up while doing the work — tests/test_delivery_coverage.py's drift tripwire had its own parents[2] walk. The issue listed three. Leaving it would have made a closure titled "one declared way" false. The builder keeps its own resolver, deliberately and now argued in writing. A script must not import from tests/ — that is the dependency direction backwards — and the contracts genuinely differ: the script returns a Path even when the checkout is absent so main can raise naming both the flag and the variable, while the test helper returns None because a missing sibling is a normal skip. WET before DRY. What is guarded is not the duplication but the property that matters: that the two AGREE. A builder writing from one checkout while the tests verify against another would report success on a lookup compared to a producer it was not built from. Review sharpened that guard. It first skipped when no checkout was present — which is exactly when the fallback path is exercised, so the one case it existed for was the case it declined to check, and it declined in CI where it matters most. The fallback paths are now compared unconditionally; mutation-proven by pointing the builder at a different directory. Review also caught this change contradicting itself in the register: Cluster K's resolution-scope line still called C-60 and C-46 "partial" eight lines above the note closing them, and C-46's body still asserted a [backlog] state that ended on resolution. Both fixed. Cluster K closes: five entries, all full. Its own note records what the strategy did not anticipate — that a cluster about "the artifact is trusted but unverified" had a verification apparatus that was itself partly unverifiable. The CI question is recorded as a residual with a recommendation and a named trigger rather than implemented, because the issue put it out of scope: do not couple this repo's per-PR CI to another repo's default branch; if wanted, a weekly scheduled check that opens an issue on divergence. Co-authored-by: Claude Opus 5 (1M context) --- reports/technical_risk_register.md | 67 +++++++++++++++------- scripts/build_gaul_lookup.py | 16 ++++++ tests/test_datafactory_deploy_readiness.py | 21 +++++-- tests/test_delivery_coverage.py | 13 +++-- tests/test_gaul_lookup_fidelity.py | 48 ++++++++++++++-- 5 files changed, 130 insertions(+), 35 deletions(-) diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 54459bb..bef03a1 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -6,8 +6,8 @@ | Owner | Dylan Pinheiro / PRIO MD&D Team | | Last Updated | 2026-08-02 | | Total Concerns | 74 | -| Open Concerns | 19 | -| Resolved Concerns | 55 | +| Open Concerns | 18 | +| Resolved Concerns | 56 | --- @@ -63,7 +63,7 @@ covered a single open entry (see Historical clusters below). **Entries:** C-59, C-61 (build-time guarantees unenforced), C-60 (provenance stamp can silently degrade), C-46 (the one cross-repo check is CI-skipped on a hardcoded path), C-43 (the value-correctness debt this cluster's fix discharges) **Highest tier:** 3 (C-59, C-60, C-61 — C-59 recalibrated 2→3 on 2026-07-31 mutation evidence) **Fix strategy:** one test file — `tests/test_gaul_lookup_fidelity.py` — split into an always-on half (gid uniqueness, region-set equality, coordinate formula against a committed ground-truth sample, no nulls, no `-1` codes) and a `skipif`-gated half comparing all 7 GAUL columns against the datafactory sibling. Plus two one-line hardenings in `scripts/build_gaul_lookup.py`: assert index uniqueness, and convert the bare `assert`s to explicit raises. -**Resolution scope:** Full for C-59/C-61/C-43's residual guard; partial for C-60 (needs the flat declared `lookup_version` key) and C-46 (needs the hardcoded path removed). +**Resolution scope:** Full for all five. C-59/C-61/C-43 discharged by the one test file this strategy predicted; C-60 by the flat declared `lookup_version` key (S5 / #186); C-46 by one declared way of resolving the producer's checkout (S7 / #188). **✅ MOSTLY CLOSED 2026-08-02 (S2 / #183).** The prediction held: **one test file discharged three entries.** `tests/test_gaul_lookup_fidelity.py` (18 tests) closed **C-43**, **C-59** and **C-61** together, exactly as the fix strategy above said it would — the always-on half against the committed artifact, the `skipif` half against views-datafactory, plus the builder's bare `assert`s converted to `LookupBuildError` raises. @@ -71,7 +71,11 @@ covered a single open entry (see Historical clusters below). **The lesson, and it generalises past this cluster.** C-43's residual said the forward-check *"was a one-off session result, not a standing guarantee"* — and the fix was to attach it to something the interpreter runs. The entries then reproduced the identical error one level up: they stated their closing conditions in prose and attached them to nothing. S2 therefore added a closing-condition check to `test_register_integrity.py`. **A guarantee needs a check, and that applies to the register's own guarantees too.** -**C-60 closed 2026-08-02 (S5 / #186)** — the flat declared key landed and the committed artifact was rebuilt, metadata-only. **Remaining in this cluster: C-46 alone** (the hardcoded datafactory path — S7 / #188), which is the last thing keeping the cluster open. +**✅ CLOSED 2026-08-02 (epic #181).** All five entries resolved: C-59 and C-61 (build-time guarantees, S2), C-43 (the value-correctness debt, S2), C-60 (the declared `lookup_version`, S5) and C-46 (the machine-specific path, S7). + +**The fix strategy predicted the shape correctly** — *"one test file, split into an always-on half and a `skipif`-gated half, plus two one-line hardenings in the builder"* — and that is what discharged three of the five. What it did not anticipate is that the cluster's own tooling was part of the problem: the gated half it prescribed could not run for anyone but the maintainer (C-46), and the provenance stamp it relied on could silently become the string `"unknown"` (C-60). A cluster about *"the artifact is trusted but unverified"* had a verification apparatus that was itself partly unverifiable. + +**What remains is not this repo's:** the artifact is now checked value-for-value against views-datafactory's parquets, but whether the **producer's** area-majority join is correct at high latitudes is views-datafactory#387. C-43's scope split holds — transcription fidelity is proven, assignment correctness is not ours to prove. ### Cluster L: The won migration was never cleaned up **Root cause:** the frame-native contract path replaced the pandas path and **won** — run-0 delivered global-land on 2026-07-27 and FAO has been served from it since. The replaced path was deliberately kept behind a config fork "until run 0 proves the contract path live" (C-40) and was then never removed. Everything below is residue of that one omission, not independent defects. @@ -386,24 +390,6 @@ This entry's trigger holds the bump on **two** conditions. Their status has dive --- -### C-46: `test_datafactory_deploy_readiness` is hardcoded to a local path — CI-skipped, and currently failing on the one machine that runs it `[backlog]` - -| Field | Value | -|-------|-------| -| ID | C-46 | -| Tier | 4 | -| Source | `repo-assimilation` (2026-06-27) | -| Trigger | When treating `test_datafactory_deploy_readiness` as a release gate (it never runs in CI), or when a contributor's local `pytest` fails on it — re-promote / re-pin the strict-xfail now that views-datafactory has advanced to `1.5.0`-dev past its `v1.4.0` tag | -| Location | `tests/test_datafactory_deploy_readiness.py` (`_DF = Path("/home/simon/.../views-datafactory")`, `skipif(not _DF.exists())`) | - -The cross-repo deploy-readiness gates introduced under C-36 are guarded by `skipif` on a **hardcoded local datafactory checkout path**, so they are **skipped in CI** and only ever execute on one developer's machine. There, `test_version_bumped_past_latest_tag` is currently **failing**: it is an `xfail(strict)` that flipped to XPASS because datafactory moved to `1.5.0`-dev past its `v1.4.0` tag — exactly the auto-flip C-36's resolution anticipated, but because of the hardcoded path the flip surfaces as a **local red** rather than a CI signal, and breaks local `pytest` runs (the suite is run with this test deselected). No correctness/reliability impact on the delivery → **Tier 4** (test hygiene). C-36 (resolved) converted these gates to strict-xfail but did not capture the local-path / CI-skip dimension. - -See also C-36 (the resolved strict-xfail conversion this extends), C-44 (the datafactory version-state coupling). - -**Tagged `[backlog]` during review-rr (2026-07-31):** Tier 4, single-machine scope, mechanical fix. Kept in the register for completeness rather than active risk management — see the Register Conventions note on the `[backlog]` tag. - ---- - ### C-58: A wrong Appwrite coordinate auto-provisions a new empty target instead of raising — both client lineages, on every write | Field | Value | @@ -650,6 +636,43 @@ See also C-40 (the inheritance/representation coupling this migration unwinds), ## Resolved Concerns +### C-46: `test_datafactory_deploy_readiness` is hardcoded to a local path — CI-skipped, and currently failing on the one machine that runs it — RESOLVED + +| Field | Value | +|-------|-------| +| ID | C-46 | +| Resolved | 2026-08-02 | +| Resolution | **Closed by S7 (#188).** Four sites resolved views-datafactory four different ways; one of them was `Path("/home/simon/Documents/scripts/views_platform/views-datafactory")`, so the repo's only cross-repo release gate had **never run anywhere but one laptop**. All four now go through one declared resolution — `$VIEWS_DATAFACTORY`, else the conventional directory beside this repo, never an absolute path to a particular machine. + +| site | was | +|---|---| +| `tests/test_datafactory_deploy_readiness.py` | an absolute path to one developer's home directory | +| `tests/test_gaul_lookup_fidelity.py` | its own `_REPO.parent / "views-datafactory"` | +| `tests/test_delivery_coverage.py` | its own `parents[2] / …` (a fourth site, found while doing the work — the issue listed three) | +| `scripts/build_gaul_lookup.py` | the shape that was already right, kept | + +Verified 2026-08-02: `grep -rn "/home/" tests/ scripts/ views_postprocessing/ --include=*.py` → **0**. With the sibling present the deploy gate runs (1 passed, 3 xfailed — its deliberately-tuned `xfail`s untouched); without it, 13 clean skips and no errors. Every skip names `VIEWS_DATAFACTORY` **and** the conventional path, so a contributor can run the test rather than watch it skip. + +**The builder keeps its own resolver, deliberately.** A script must not import from `tests/` — that is the dependency direction backwards — and the contracts genuinely differ: the script returns a `Path` even when the checkout is absent so `main` can raise naming both the flag and the variable, while the test helper returns `None` because a missing sibling is a normal skip. WET before DRY: two copies that are understood beat one abstraction that is guessed. What is guarded instead is the property that actually matters — that they **agree** — because a builder writing from one checkout while the tests verify against another would report success on a lookup compared to a producer it was not built from. + +**Residual, and the decision it needs.** These checks still do not run in CI. There are three such gated groups now — this deploy gate, the fidelity suite's producer-comparison half, and **C-57**'s registry-drift half — and the question should be answered once for all three rather than three times. + +**Recommendation, for whoever takes it:** do **not** add sibling checkouts to the per-PR workflow. It couples this repo's CI to another repo's default branch, so an unrelated upstream commit turns this repo red — precisely the flapping `TestReleaseGate` already documents and was re-pinned for once. The always-on halves already guard the committed artifact; the gated halves answer *"is the producer's current state still consistent with ours?"*, which is a **scheduled** cross-repo question, not a merge gate. If it is wanted, the vehicle is a weekly workflow that opens an issue on divergence. **Named trigger:** the next time an upstream change reaches FAO through this repo without anyone noticing first. | +| Tier | 4 | +| Source | `repo-assimilation` (2026-06-27) | +| Trigger | When treating `test_datafactory_deploy_readiness` as a release gate (it never runs in CI), or when a contributor's local `pytest` fails on it — re-promote / re-pin the strict-xfail now that views-datafactory has advanced to `1.5.0`-dev past its `v1.4.0` tag | +| Location | `tests/test_datafactory_deploy_readiness.py` (`_DF = Path("/home/simon/.../views-datafactory")`, `skipif(not _DF.exists())`) | + +The cross-repo deploy-readiness gates introduced under C-36 are guarded by `skipif` on a **hardcoded local datafactory checkout path**, so they are **skipped in CI** and only ever execute on one developer's machine. There, `test_version_bumped_past_latest_tag` is currently **failing**: it is an `xfail(strict)` that flipped to XPASS because datafactory moved to `1.5.0`-dev past its `v1.4.0` tag — exactly the auto-flip C-36's resolution anticipated, but because of the hardcoded path the flip surfaces as a **local red** rather than a CI signal, and breaks local `pytest` runs (the suite is run with this test deselected). No correctness/reliability impact on the delivery → **Tier 4** (test hygiene). C-36 (resolved) converted these gates to strict-xfail but did not capture the local-path / CI-skip dimension. + +See also C-36 (the resolved strict-xfail conversion this extends), C-44 (the datafactory version-state coupling). + +**Was tagged `[backlog]` at review-rr (2026-07-31)** — Tier 4, single-machine scope, mechanical fix — and kept for completeness rather than active risk management. The tag was dropped when S7 resolved it. Worth noting for the convention itself: a `[backlog]` entry is deprioritised, not dormant, and this one turned out to be blocking three gated cross-repo checks from running anywhere but one machine. + +--- + +--- + ### C-57: PLATFORM-001 coordinate registry is referenced by URL, so nothing detects drift between it and this repo's declared environment — RESOLVED | Field | Value | diff --git a/scripts/build_gaul_lookup.py b/scripts/build_gaul_lookup.py index b488cdb..46eb21d 100644 --- a/scripts/build_gaul_lookup.py +++ b/scripts/build_gaul_lookup.py @@ -52,6 +52,22 @@ def _resolve_datafactory() -> Path: Order: $VIEWS_DATAFACTORY, then the sibling repo next to this one (views_platform/views-datafactory). Overridable with --datafactory. + + **Deliberately duplicated with** ``tests/conftest.sibling_repo`` (S7 / #188, + register C-46). Not an oversight and not laziness: + + - a script must not import from ``tests/`` — that is the dependency direction + backwards, and it would make the build depend on the test tree; + - the contracts differ. This returns a ``Path`` **even when the checkout is + absent**, so ``main`` can raise its own message naming both the flag and the + variable. The test helper returns ``None``, because a missing sibling is a + normal skip, not an error. + + Two copies that are understood beat one abstraction that is guessed. What is + guarded instead is the thing that actually matters — that they **agree** — + pinned by ``tests/test_gaul_lookup_fidelity.py``. If they ever resolve to + different checkouts, a rebuilt artifact would be verified against a producer it + was not built from. """ env = os.environ.get("VIEWS_DATAFACTORY") if env: diff --git a/tests/test_datafactory_deploy_readiness.py b/tests/test_datafactory_deploy_readiness.py index b1792b4..f8ffa8d 100644 --- a/tests/test_datafactory_deploy_readiness.py +++ b/tests/test_datafactory_deploy_readiness.py @@ -7,18 +7,29 @@ until the datafactory deploy candidate is actually releasable and the served artifact matches the branch. -Point _DF at the local datafactory checkout to run. +Resolution is the repo's one declared way of finding a sibling checkout +(``tests/conftest.sibling_repo``): ``$VIEWS_DATAFACTORY``, else the conventional +directory beside this repo. Never an absolute path to a particular machine — that +was C-46, and it kept this gate from ever running anywhere but one laptop. """ import json import subprocess -from pathlib import Path import pytest -_DF = Path("/home/simon/Documents/scripts/views_platform/views-datafactory") -_pytestmark = pytest.mark.skipif(not _DF.exists(), reason="datafactory checkout not present") -pytestmark = _pytestmark +from tests.conftest import sibling_repo + +_DF = sibling_repo("views-datafactory") +pytestmark = pytest.mark.skipif( + _DF is None, + reason=( + "views-datafactory checkout not found — set VIEWS_DATAFACTORY=/path/to/" + "views-datafactory, or place it alongside this repo. Until S7 (#188) this " + "module hardcoded an absolute path to one developer's machine, so the only " + "cross-repo release gate in the repo could not run anywhere else (C-46)." + ), +) def _git(*args: str) -> str: diff --git a/tests/test_delivery_coverage.py b/tests/test_delivery_coverage.py index cebab92..716e194 100644 --- a/tests/test_delivery_coverage.py +++ b/tests/test_delivery_coverage.py @@ -4,10 +4,11 @@ """ import json -from pathlib import Path import pytest +from tests.conftest import sibling_repo + from views_postprocessing.delivery.coverage import ( EXPECTED_CELLS, CoverageError, @@ -93,12 +94,16 @@ def test_no_excluded_cells_is_noop_when_region_unpinned(): # Cross-check the frozen manifest against the live producer when its checkout is present # (CI has no sibling → skip). This is the drift tripwire C-30 asks for. -_DF = Path(__file__).resolve().parents[2] / "views-datafactory" / "src" / "datafactory_query" +_DATAFACTORY = sibling_repo("views-datafactory") +_DF = None if _DATAFACTORY is None else _DATAFACTORY / "src" / "datafactory_query" @pytest.mark.skipif( - not (_DF / "land_pgids.json").exists(), - reason="views-datafactory sibling checkout not present", + _DF is None or not (_DF / "land_pgids.json").exists(), + reason=( + "views-datafactory checkout not found — set VIEWS_DATAFACTORY=/path/to/" + "views-datafactory, or place it alongside this repo" + ), ) def test_manifest_matches_datafactory_land_minus_land_gaul(): land = {int(x) for x in json.loads((_DF / "land_pgids.json").read_text())} diff --git a/tests/test_gaul_lookup_fidelity.py b/tests/test_gaul_lookup_fidelity.py index a451188..870672d 100644 --- a/tests/test_gaul_lookup_fidelity.py +++ b/tests/test_gaul_lookup_fidelity.py @@ -33,12 +33,14 @@ from __future__ import annotations import json +import os from pathlib import Path import numpy as np import pyarrow.parquet as pq import pytest +from tests.conftest import sibling_repo from views_postprocessing.delivery import coverage from views_postprocessing.contract.gaul_schema import ( CODE_COLS, @@ -57,13 +59,17 @@ # The lookup is built for this region; its count is pinned in delivery/coverage.py. _REGION = "land_gaul" -_DATAFACTORY = _REPO.parent / "views-datafactory" -_HAS_DATAFACTORY = (_DATAFACTORY / "data" / "raw" / "gaul_admin").is_dir() +_DATAFACTORY = sibling_repo("views-datafactory") +_HAS_DATAFACTORY = _DATAFACTORY is not None and ( + _DATAFACTORY / "data" / "raw" / "gaul_admin" +).is_dir() _needs_datafactory = pytest.mark.skipif( not _HAS_DATAFACTORY, reason=( - "views-datafactory sibling checkout not present — the producer-comparison half " - "cannot run here. The always-on tests in this module still guard the artifact." + "views-datafactory checkout not found — set VIEWS_DATAFACTORY=/path/to/" + "views-datafactory, or place it alongside this repo. Only the " + "producer-comparison half is skipped; the always-on tests still guard the " + "committed artifact." ), ) @@ -471,3 +477,37 @@ def test_a_short_digest_is_refused_rather_than_truncated_silently(): with pytest.raises(builder.LookupBuildError, match="at least 8 characters"): builder._lookup_version("land_gaul", {"land_gaul_region": {"content_digest": "abcd"}}) + + +def test_the_builder_and_the_tests_resolve_the_same_datafactory(): + """The one thing worth guarding about the deliberate duplication (S7 / #188). + + ``scripts/build_gaul_lookup._resolve_datafactory`` and + ``tests/conftest.sibling_repo`` are separate on purpose — a script must not import + from ``tests/``, and their contracts differ (a Path even when absent, versus None). + Register **C-46** was never about there being two functions; it was about one of + them being an absolute path to a single machine. + + What must not drift is the **answer**. If the builder wrote an artifact from one + checkout while these tests verified it against another, the fidelity suite would be + comparing a lookup to a producer it was not built from — and reporting success. + """ + import scripts.build_gaul_lookup as builder + + # The FALLBACK is the only place they can diverge — both read $VIEWS_DATAFACTORY + # first — and it is exercised precisely when no checkout exists. So compare the + # computed paths unconditionally: gating this on a checkout being present would + # skip the one case the test is for, and skip it in CI, where it matters most. + if "VIEWS_DATAFACTORY" not in os.environ: + assert builder._resolve_datafactory() == _REPO.parent / "views-datafactory", ( + "the builder's fallback and the tests' fallback resolve different " + "directories; with no environment override they would disagree silently" + ) + + resolved = sibling_repo("views-datafactory") + if resolved is None: + return # nothing further to compare; the fallback agreement is asserted above + assert builder._resolve_datafactory().resolve() == resolved.resolve(), ( + "the builder and the tests resolve different views-datafactory checkouts; a " + "rebuilt artifact would be verified against a producer it was not built from" + ) From fb496485cb79770a3dd4c95af898eec1108eaf38 Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 2 Aug 2026 06:34:41 +0200 Subject: [PATCH 12/23] =?UTF-8?q?test(redaction):=20S10=20#192=20=E2=80=94?= =?UTF-8?q?=20repair=20the=20=C3=BEing-01=20guard;=20a=20declared=20root?= =?UTF-8?q?=20that=20vanishes=20now=20fails=20(#204)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit tests/test_redaction_guard.py pins the #135 delivery-log redaction audit over five module trees. Epic #148's #153 moved the machinery from unfao/ to contract/ and four of the five roots were never re-pointed. The guard kept passing while scanning 6 files instead of 17. Re-pointing fixes today. The assertion that every declared root EXISTS fixes the class, and that is the half worth having: rglob on a nonexistent directory yields an empty iterator rather than raising, so a missing root and a clean root were indistinguishable to the scan. A package move silently emptied it while the suite stayed green. A third check bounds the count, so narrowing surfaces as a number rather than as nothing. Mutation-proven by restoring the pre-S10 roots: two failures, one naming all four missing paths, one reporting "covers only 6 files". The guard is also now proven to bite on the defect it was written for — a module reading os.environ['APPWRITE_DATASTORE_API_KEY'] — which had never been demonstrated. An unproven guard is what two silent days buys you. Explicit list, not derivation, decided and recorded. test_clone_readiness.py enumerates an overlapping set for a different question; folding them together would couple two guards whose sets are free to diverge, since a module can be partner-neutral without being credential-blind. What makes the list safe is the existence assertion. No leak occurred. The relocated modules were checked at registration and again here: zero hits for os.environ, getenv, load_dotenv, API_KEY, credentials. Two entries that had vouched for this guard are restored rather than quietly reverted — C-57's "already mechanically enforced" and Cluster I's fix strategy — because a claim that was false for two days is worth more as a record. Also caught by the register's own integrity guard while writing this: the extraction regex for C-74 used a `(?=^### )` boundary, and since C-74 was the last entry in Open Concerns it swallowed the `## Disagreements` heading, filing three D-entries under Open. Four integrity tests failed immediately. Fixed with a section-aware boundary — the guard S2 built catching a defect in the tooling that maintains the register it guards. Co-authored-by: Claude Opus 5 (1M context) --- reports/technical_risk_register.md | 82 ++++++++++++++------------ tests/test_redaction_guard.py | 94 +++++++++++++++++++++++++++--- 2 files changed, 133 insertions(+), 43 deletions(-) diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index bef03a1..0f3db06 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -6,8 +6,8 @@ | Owner | Dylan Pinheiro / PRIO MD&D Team | | Last Updated | 2026-08-02 | | Total Concerns | 74 | -| Open Concerns | 18 | -| Resolved Concerns | 56 | +| Open Concerns | 17 | +| Resolved Concerns | 57 | --- @@ -47,7 +47,7 @@ covered a single open entry (see Historical clusters below). **Root cause:** the register, ADR prose, and issue bodies are hand-maintained mirrors of cross-repo state that moves under them. **Entries:** C-44, C-46, C-47, C-57, C-74 (a guard whose declared scan roots silently stopped existing — the cluster's disease inside the cluster's own prescription) — plus this register's own findings at review-rr 2026-07-31 (header miscount, two RESOLVED entries misfiled under Open, eight stale `unfao.py` line ranges after the manager grew 273→636 lines, two unnamespaced foreign-register IDs). Historical precedent: the entire C-48–C-55 ADR-013 audit series, and C-42/C-47. **Highest tier:** 3 -**Fix strategy:** this repo already solved this disease once — the ADR-013 audit series ended with **40 permanent guard tests** (`tests/test_falsify_adr013_*.py`), and the same pattern now guards the þing-01 invariants (`tests/test_env_declaration.py`, `tests/test_redaction_guard.py` — the latter **only over the roots that still exist**, see C-74: a guard is only as good as the assertion that its inputs are real). There is **no equivalent for the register**. A small `tests/test_register_integrity.py` — header counts match section counts; no RESOLVED body under `## Open Concerns`; every `C-\d+`/`D-\d+` reference resolves or is namespaced to a foreign register — would make this class self-detecting. +**Fix strategy:** this repo already solved this disease once — the ADR-013 audit series ended with **40 permanent guard tests** (`tests/test_falsify_adr013_*.py`), and the same pattern now guards the þing-01 invariants (`tests/test_env_declaration.py`, `tests/test_redaction_guard.py` — the latter briefly **only over the roots that still existed**, see C-74, resolved: a guard is only as good as the assertion that its inputs are real, and it now carries that assertion). There is **no equivalent for the register**. A small `tests/test_register_integrity.py` — header counts match section counts; no RESOLVED body under `## Open Concerns`; every `C-\d+`/`D-\d+` reference resolves or is namespaced to a foreign register — would make this class self-detecting. **Resolution scope:** Full for the mechanical half. ### Cluster J: Delivery aftercare has no mechanism @@ -536,38 +536,6 @@ Cross-refs: **C-25** (whose resolution this corrects), **C-40** (the inherited p --- -### C-74: The þing-01 redaction guard scans four paths that stopped existing — it reports success while covering one root of five - -| Field | Value | -|-------|-------| -| ID | C-74 | -| Tier | 3 — **not a leak today**: the relocated modules were checked directly and are still credential-blind (zero hits for `os.environ`, `getenv`, `load_dotenv`, `API_KEY`, `credentials`). What is gone is the thing that would notice them ceasing to be. A security-adjacent control that cannot fail is a maintainability defect until the day it is a correctness one. | -| Source | `review-diff` (2026-08-02) — S1/#182 review; found while reading the redaction discipline the new ADR-008 test cites as "the wider rule" | -| Trigger | When any module under `contract/wire`, `contract/historical.py`, `contract/track_a_source.py` or `contract/frame_extraction.py` gains environment access — the #135 guard will not report it. Also fires on **the next package move**: `rglob` on a vanished root yields silence, not an error, so any future relocation narrows the scan again with no signal | -| Location | `tests/test_redaction_guard.py:25-31` (`_CREDENTIAL_BLIND`), `:34-39` (`_python_sources`) | - -The þing-01 delivery-log redaction audit (#135, orð_09 §3) certified five module trees as credential-blind and pinned that finding as a permanent guard. Epic #148's S5 (#153) then moved the machinery out from under `unfao/` into `contract/`. **Four of the five roots were never re-pointed:** - -| declared root | exists | files scanned | -|---|---|---| -| `unfao/wire` | no | 0 | -| `delivery` | yes | 6 | -| `unfao/historical.py` | no | 0 | -| `unfao/track_a_source.py` | no | 0 | -| `unfao/frame_extraction.py` | no | 0 | - -**The failure is silent by construction.** `_python_sources` branches `if path.is_file(): … else: path.rglob("*.py")`, and `rglob` on a **nonexistent** directory yields an empty iterator rather than raising. A missing root and a clean root are therefore indistinguishable to the test, which passes either way. The guard reports an audited fact as pinned while pinning roughly a third of it. - -**Why this is Cluster I and not merely a stale path.** The register's own text asserts the opposite. **C-57** says, in its "deliberately out of scope" paragraph, that *"the þing-01 redaction clause is **already mechanically enforced** (`tests/test_redaction_guard.py` — the delivery modules stay credential-blind…)"* — and **Cluster I's fix strategy cites this very file** as an example of the guard pattern working. Both were written in good faith and both are now overclaims. That is the cluster's disease reproducing inside the cluster's own prescription. - -It is also the **sixth** instance of the stale-claim class found in two days, after C-63 and C-47 (filed Open with the defect fixed), C-43/C-59/C-61 (filed Open with their stated closing conditions met), #158 (closed with the code half undone) and `tests/test_validation.py` (claiming fidelity to a method it no longer resembles — **C-03**). The common shape is not carelessness: it is that a *move* leaves prose and paths behind, and nothing in this repo asserts that a declared path exists. - -**Mitigation:** re-point the four roots at `contract/`, and add a root-existence assertion so a future relocation fails loudly instead of silently narrowing. The second half is the load-bearing one — re-pointing fixes today, asserting existence fixes the class. Tracked as a story under **epic #181**. - -Cross-refs: **C-57** (whose "already mechanically enforced" claim this falsifies — corrected in place), **C-03** (the same class in `test_validation.py`), **C-46** (a different guard that also does not run, by a different mechanism), **C-63**, **C-47**, **Cluster I**, #135, #153, #182. - ---- - ## Disagreements ### D-12: Post-Run-0 infrastructure & naming intents — repo rename, internal-store transport, compute co-location @@ -636,6 +604,48 @@ See also C-40 (the inheritance/representation coupling this migration unwinds), ## Resolved Concerns +### C-74: The þing-01 redaction guard scans four paths that stopped existing — it reports success while covering one root of five — RESOLVED + +| Field | Value | +|-------|-------| +| ID | C-74 | +| Resolved | 2026-08-02 | +| Resolution | **Closed by S10 (#192).** The four roots that #153 orphaned now point at `contract/`, and the scan covers **17 files where it covered 6**. Verified 2026-08-02. + +**The half that matters is the second one.** Re-pointing fixes today; `test_every_declared_credential_blind_root_exists` fixes the class. `rglob` on a nonexistent directory yields an empty iterator rather than raising, so a missing root and a clean root were indistinguishable — a package move silently emptied the scan while the suite stayed green. A declared path that does not exist is now an error, not silence. A third check bounds the count, so narrowing shows up as a number rather than as nothing. + +Mutation-proven by restoring the pre-S10 roots: **two failures**, one naming all four missing paths, one reporting `covers only 6 files`. The guard was also proven to bite on the defect it was written for — a synthetic module reading `os.environ['APPWRITE_DATASTORE_API_KEY']` — which had never been demonstrated, and an unproven guard is what two silent days buys you. + +**List, not derivation — decided and recorded.** `tests/test_clone_readiness.py` enumerates an overlapping module set for a *different* question ("does the machinery import without the partner?"). Folding them together would couple two guards whose sets are free to diverge: a module can be partner-neutral without being credential-blind. Two lists that each say what they mean beat one that means neither. What makes the explicit list safe is the existence assertion — without it, a list is exactly the fragile thing it looked like here. + +**No leak occurred.** The relocated modules were checked directly at registration and again here: zero hits for `os.environ`, `getenv`, `load_dotenv`, `API_KEY`, `credentials`. The audited fact stayed true; what had gone was the thing that would notice it stopping. | +| Tier | 3 — **not a leak today**: the relocated modules were checked directly and are still credential-blind (zero hits for `os.environ`, `getenv`, `load_dotenv`, `API_KEY`, `credentials`). What is gone is the thing that would notice them ceasing to be. A security-adjacent control that cannot fail is a maintainability defect until the day it is a correctness one. | +| Source | `review-diff` (2026-08-02) — S1/#182 review; found while reading the redaction discipline the new ADR-008 test cites as "the wider rule" | +| Trigger | When any module under `contract/wire`, `contract/historical.py`, `contract/track_a_source.py` or `contract/frame_extraction.py` gains environment access — the #135 guard will not report it. Also fires on **the next package move**: `rglob` on a vanished root yields silence, not an error, so any future relocation narrows the scan again with no signal | +| Location | `tests/test_redaction_guard.py:25-31` (`_CREDENTIAL_BLIND`), `:34-39` (`_python_sources`) | + +The þing-01 delivery-log redaction audit (#135, orð_09 §3) certified five module trees as credential-blind and pinned that finding as a permanent guard. Epic #148's S5 (#153) then moved the machinery out from under `unfao/` into `contract/`. **Four of the five roots were never re-pointed:** + +| declared root | exists | files scanned | +|---|---|---| +| `unfao/wire` | no | 0 | +| `delivery` | yes | 6 | +| `unfao/historical.py` | no | 0 | +| `unfao/track_a_source.py` | no | 0 | +| `unfao/frame_extraction.py` | no | 0 | + +**The failure is silent by construction.** `_python_sources` branches `if path.is_file(): … else: path.rglob("*.py")`, and `rglob` on a **nonexistent** directory yields an empty iterator rather than raising. A missing root and a clean root are therefore indistinguishable to the test, which passes either way. The guard reports an audited fact as pinned while pinning roughly a third of it. + +**Why this is Cluster I and not merely a stale path.** The register's own text asserts the opposite. **C-57** says, in its "deliberately out of scope" paragraph, that *"the þing-01 redaction clause is **already mechanically enforced** (`tests/test_redaction_guard.py` — the delivery modules stay credential-blind…)"* — and **Cluster I's fix strategy cites this very file** as an example of the guard pattern working. Both were written in good faith and both are now overclaims. That is the cluster's disease reproducing inside the cluster's own prescription. + +It is also the **sixth** instance of the stale-claim class found in two days, after C-63 and C-47 (filed Open with the defect fixed), C-43/C-59/C-61 (filed Open with their stated closing conditions met), #158 (closed with the code half undone) and `tests/test_validation.py` (claiming fidelity to a method it no longer resembles — **C-03**). The common shape is not carelessness: it is that a *move* leaves prose and paths behind, and nothing in this repo asserts that a declared path exists. + +**Mitigation:** re-point the four roots at `contract/`, and add a root-existence assertion so a future relocation fails loudly instead of silently narrowing. The second half is the load-bearing one — re-pointing fixes today, asserting existence fixes the class. Tracked as a story under **epic #181**. + +Cross-refs: **C-57** (whose "already mechanically enforced" claim this falsifies — corrected in place), **C-03** (the same class in `test_validation.py`), **C-46** (a different guard that also does not run, by a different mechanism), **C-63**, **C-47**, **Cluster I**, #135, #153, #182. + +--- + ### C-46: `test_datafactory_deploy_readiness` is hardcoded to a local path — CI-skipped, and currently failing on the one machine that runs it — RESOLVED | Field | Value | @@ -716,7 +726,7 @@ Two named changes are already anticipated and will fire this trigger: the **reti **Deliberately out of scope here:** the þing-01 redaction clause is mechanically enforced (`tests/test_redaction_guard.py` — the delivery modules stay credential-blind and the provenance description is a closed keyset), and D2's ruling that **integration tests against the production Appwrite project are FORBIDDEN** (no non-production project exists) is a standing prohibition, not a drift risk. -**⚠ CORRECTED 2026-08-02.** The word *already* above overclaimed: **C-74** showed that guard scans one of its five declared roots, four having pointed at paths that #153 moved. The keyset half is enforced; the credential-blindness half is enforced over `delivery/` only until C-74 lands. +**⚠ CORRECTED 2026-08-02, then restored the same day.** The word *already* above overclaimed at the time: **C-74** showed that guard scanning one of its five declared roots, four having pointed at paths #153 moved. **C-74 closed later that day (S10 / #192)** — the roots are re-pointed, the scan covers 17 files, and a declared root that does not exist now fails rather than emptying the scan silently. The sentence above is true again, and the episode is left visible because a claim that was false for two days is worth more as a record than as a correction quietly reverted. Cross-refs: C-74 (the guard this paragraph vouched for), C-33 (store identity still hardcoded per store — the same env surface, different concern), C-58 (what happens when a coordinate is wrong rather than missing), C-44 (the pipeline-core version coupling that would carry a registry change), issues #134/#135/#138 (this repo's discharged þing-01 obligations), #104 (README env block placeholders). diff --git a/tests/test_redaction_guard.py b/tests/test_redaction_guard.py index 5283f33..62e2b26 100644 --- a/tests/test_redaction_guard.py +++ b/tests/test_redaction_guard.py @@ -20,12 +20,27 @@ # the environment and never see a key (the manager's DIP port hands them data # only). A hit here means a credential VALUE could newly reach a delivery log # line or artifact — re-run the #135 audit before allowing it. +# +# **These paths moved and this list did not** (register C-74, S10 / #192). Epic +# #148's #153 relocated the machinery from ``unfao/`` to ``contract/``; four of the +# five roots below kept pointing at ``unfao/`` and the guard went on passing while +# scanning **6 files instead of 16**. It was not wrong; it was not updated. +# +# Written out as an explicit list rather than derived from the package tree — +# deliberately. ``tests/test_clone_readiness.py`` enumerates an overlapping set for +# a *different* question ("does the machinery import without the partner?"), and +# folding them together would couple two guards whose sets are free to diverge: +# a new module can be partner-neutral without being credential-blind. Two lists +# that each say what they mean beat one that means neither. +# +# What makes the explicit list safe is the root-existence assertion below. That is +# the half that fixes the *class* rather than today's instance. _CREDENTIAL_BLIND = [ - _PKG / "unfao" / "wire", + _PKG / "contract" / "wire", _PKG / "delivery", - _PKG / "unfao" / "historical.py", - _PKG / "unfao" / "track_a_source.py", - _PKG / "unfao" / "frame_extraction.py", + _PKG / "contract" / "historical.py", + _PKG / "contract" / "track_a_source.py", + _PKG / "contract" / "frame_extraction.py", ] _FORBIDDEN_TOKENS = ("os.environ", "getenv", "load_dotenv", "API_KEY", "credentials") @@ -38,20 +53,85 @@ def _python_sources(path: Path): yield from sorted(path.rglob("*.py")) -def test_wire_and_delivery_modules_stay_credential_blind(): +def _scan(roots: list[Path], base: Path = _PKG) -> list[str]: + """Every forbidden-token hit under ``roots``, reported relative to ``base``. + + ``base`` is a parameter only so the guard can be pointed at a synthetic tree and + proven to bite — see ``test_the_guard_would_catch_a_credential_reference``. + """ hits = [] - for root in _CREDENTIAL_BLIND: + for root in roots: for source in _python_sources(root): text = source.read_text() for token in _FORBIDDEN_TOKENS: if token in text: - hits.append(f"{source.relative_to(_PKG)}: {token!r}") + hits.append(f"{source.relative_to(base)}: {token!r}") + return hits + + +def test_every_declared_credential_blind_root_exists(): + """The half that fixes the class, not the instance (register C-74). + + ``_python_sources`` calls ``rglob`` on a directory that is not a file — and + ``rglob`` on a **nonexistent** directory yields an empty iterator rather than + raising. So a missing root and a clean root are indistinguishable to the scan, + and a package move silently narrows it to nothing while the suite stays green. + + That is precisely what happened between #153 and this test: four of five roots + resolved to nothing for two days, and the þing-01 #135 audit was pinned over + roughly a third of what it certified. + """ + missing = [str(r.relative_to(_PKG)) for r in _CREDENTIAL_BLIND if not r.exists()] + assert not missing, ( + f"declared credential-blind root(s) no longer exist: {missing}. A move that " + "leaves this list behind does not fail the scan — it empties it. Re-point the " + "list at where the modules live now (þing-01 #135, register C-74)." + ) + + +def test_wire_and_delivery_modules_stay_credential_blind(): + hits = _scan(_CREDENTIAL_BLIND) assert not hits, ( "credential-blind delivery modules now reference the environment or a " f"credential (þing-01 #135 audit invalidated): {hits}" ) +def test_the_scan_covers_the_whole_audited_surface(): + """A count, so silent narrowing shows up as a number rather than as silence. + + The #135 audit certified the delivery path — the wire, the invariants, and the + three seam modules. It is ~16 files; it was 6 while the roots were stale. The + bound is deliberately loose (a lower bound, not an equality) so adding a module + does not fail the suite, while *losing* most of them does. + """ + scanned = sum(1 for root in _CREDENTIAL_BLIND for _ in _python_sources(root)) + assert scanned >= 15, ( + f"the credential-blind scan covers only {scanned} files. The audited surface " + "is the wire (8), delivery (6) and three seam modules. A number well below " + "that means roots are resolving to nothing again — see C-74." + ) + + +def test_the_guard_would_catch_a_credential_reference(tmp_path): + """The defect it was written for: a delivery module reading the environment. + + A guard nobody has watched fail is a guard nobody knows the shape of. This one + was passing for two days over four roots that resolved to nothing (C-74), which + is exactly what an unproven guard buys you. + """ + (tmp_path / "clean.py").write_text("def build(table):\n return table\n") + assert _scan([tmp_path], base=tmp_path) == [], "a clean module was flagged" + + (tmp_path / "leaky.py").write_text( + "import os\nkey = os.environ['APPWRITE_DATASTORE_API_KEY']\n" + ) + hits = _scan([tmp_path], base=tmp_path) + assert any("leaky.py" in h for h in hits), ( + f"a module reading a credential from the environment was not flagged: {hits}" + ) + + def test_provenance_carries_only_the_declared_closed_keyset(): # The description is uploaded into the store's metadata — it leaves the # machine. Its content must stay exactly the declared delivery primitives. From 729868a9fbbcad6a7fde0e6df3ec6b7ea8c362fc Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 2 Aug 2026 06:38:04 +0200 Subject: [PATCH 13/23] =?UTF-8?q?chore(register):=20close=20C-71=20?= =?UTF-8?q?=E2=80=94=20fixed=20by=20S1=20this=20morning,=20filed=20Open=20?= =?UTF-8?q?all=20day=20(#205)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit assert_env_declared has logged before raising since #193. The entry stayed under Open while eight further stories shipped, in the epic whose entire thesis is that a record should not outlive the thing it describes. S2's guard did not catch it, and that is by design rather than by defect: it matches two declarative phrasings, and this entry's closing language is prose — "a two-line change; the natural place to take it is S8". That is the false negative S2 deliberately accepted, and the cost came due within hours. Widening the guard would be the wrong fix; no regex separates "describes work that is done" from prose. The gap is that #182 was closed without disposing of the entry it named in the same change. The generalisable rule — a story that names a register entry disposes of it in the same PR — goes to S9's ADR question alongside the CIC lagging three consecutive stories. Both are records updated by memory rather than by the change that invalidates them. Co-authored-by: Claude Opus 5 (1M context) --- reports/technical_risk_register.md | 54 ++++++++++++++++++------------ 1 file changed, 32 insertions(+), 22 deletions(-) diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 0f3db06..a0c0a9e 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -6,8 +6,8 @@ | Owner | Dylan Pinheiro / PRIO MD&D Team | | Last Updated | 2026-08-02 | | Total Concerns | 74 | -| Open Concerns | 17 | -| Resolved Concerns | 57 | +| Open Concerns | 16 | +| Resolved Concerns | 58 | --- @@ -464,26 +464,6 @@ Cross-refs: **C-44** (the held bump — same action, opposing rationale), **C-39 --- -### C-71: `appwrite_env.assert_env_declared` raises without logging — ADR-008 non-compliance in an entry-validation seam - -| Field | Value | -|-------|-------| -| ID | C-71 | -| Tier | 4 — the raise is loud and its message is fully diagnostic, so nothing is silently swallowed today. What is missing is the persistent record ADR-008 requires, in the one seam whose whole job is to make a misconfigured launch visible. | -| Source | `review-diff` (2026-07-31) — S1/#149 review; found by mirroring this module when writing its sibling | -| Trigger | When a delivery run refuses on a missing Appwrite variable and the operator goes looking for *why* in the logs rather than the traceback — or when the next entry-validation module is written against this one as the pattern, as `unfao/launch_config.py` was | -| Location | `views_postprocessing/unfao/appwrite_env.py:44-51` (`assert_env_declared`) | - -**ADR-008:48** requires that *"raised structural failures must be logged at `ERROR` level or higher"*, and **:51** that *"raising is not a substitute for logging."* `assert_env_declared` raises `EnvironmentError` naming every missing variable but never logs. A launcher misconfiguration is a structural failure by any reading of that ADR. - -**How it was found, and why that matters:** `unfao/launch_config.py` (S1) was deliberately written to mirror this module — same shape, same failure style, same dependency-light constraint. It **inherited the flaw**, and the S1 diff review caught it in the new code. The new module was fixed to log-before-raise; this one was left alone for scope discipline, which means the pair is now **inconsistent** — the sibling written to match it no longer does. - -Registered rather than fixed in #149 because it is pre-existing (shipped in þing-01 P1 / #134) and outside that story's boundary. It is a two-line change and the natural place to take it is **S8** (#156, epic closeout) or any PR that next touches `appwrite_env.py`. - -Cross-refs: **C-19** (RESOLVED — the ADR-008 log-before-raise sweep whose convention this predates), **C-63** (the declaration-over-inference concern S1 closed), ADR-008, #134, #149, #156. - ---- - ### C-72: The pyarrow pin holds this repo inside a high-severity CVE, and the fix changes the wire bytes | Field | Value | @@ -604,6 +584,36 @@ See also C-40 (the inheritance/representation coupling this migration unwinds), ## Resolved Concerns +### C-71: `appwrite_env.assert_env_declared` raises without logging — ADR-008 non-compliance in an entry-validation seam — RESOLVED + +| Field | Value | +|-------|-------| +| ID | C-71 | +| Resolved | 2026-08-02 | +| Resolution | **Closed by S1 (#182 / #193) on 2026-08-02 — and this entry then sat under Open for the rest of the day while eight further stories shipped.** + +The fix is exactly what the entry predicted: `assert_env_declared` builds its message into a local, logs it at ERROR, then raises it. Verified — one `logger.error(err_msg)` call, and `tests/test_env_declaration.py::test_every_entry_validator_logs_before_it_raises` parametrises the ADR-008 obligation over **both** entry validators so the pair cannot drift apart again, which was the actual hazard: `launch_config` was written by mirroring this module, inherited the flaw, was fixed in review, and left its model as the odd one out. + +**Why this was not caught, stated plainly rather than explained away.** S2 (#183) added exactly the guard for this class — an Open entry whose stated closing condition is already met. It did not fire here, because it matches two declarative phrasings (`closes when \`tests/…\`` and `Mitigation — landed`) and this entry's closing language is prose: *"It is a two-line change and the natural place to take it is S8."* That is the false negative S2 deliberately accepted — *"a guard that cries wolf gets deleted… false negatives are the accepted cost"* — and the cost came due within hours. + +**The gap is process, not tooling, and widening the guard would be the wrong fix.** No regex reliably distinguishes "this entry describes work that is done" from prose. What went wrong is that #182 was closed without disposing of the entry it named in the same change. The generalisable rule — **a story that names a register entry disposes of it in the same PR** — belongs in S9's ADR question, alongside the CIC lagging three consecutive stories. Both are the same shape: a record updated by memory rather than by the change that invalidates it. + +(The entry also pointed at **#156**, epic #148's closeout, which had already closed. A pointer to a finished story is how an item becomes nobody's.) | +| Tier | 4 — the raise is loud and its message is fully diagnostic, so nothing is silently swallowed today. What is missing is the persistent record ADR-008 requires, in the one seam whose whole job is to make a misconfigured launch visible. | +| Source | `review-diff` (2026-07-31) — S1/#149 review; found by mirroring this module when writing its sibling | +| Trigger | When a delivery run refuses on a missing Appwrite variable and the operator goes looking for *why* in the logs rather than the traceback — or when the next entry-validation module is written against this one as the pattern, as `unfao/launch_config.py` was | +| Location | `views_postprocessing/unfao/appwrite_env.py:44-51` (`assert_env_declared`) | + +**ADR-008:48** requires that *"raised structural failures must be logged at `ERROR` level or higher"*, and **:51** that *"raising is not a substitute for logging."* `assert_env_declared` raises `EnvironmentError` naming every missing variable but never logs. A launcher misconfiguration is a structural failure by any reading of that ADR. + +**How it was found, and why that matters:** `unfao/launch_config.py` (S1) was deliberately written to mirror this module — same shape, same failure style, same dependency-light constraint. It **inherited the flaw**, and the S1 diff review caught it in the new code. The new module was fixed to log-before-raise; this one was left alone for scope discipline, which means the pair is now **inconsistent** — the sibling written to match it no longer does. + +Registered rather than fixed in #149 because it is pre-existing (shipped in þing-01 P1 / #134) and outside that story's boundary. It is a two-line change and the natural place to take it is **S8** (#156, epic closeout) or any PR that next touches `appwrite_env.py`. + +Cross-refs: **C-19** (RESOLVED — the ADR-008 log-before-raise sweep whose convention this predates), **C-63** (the declaration-over-inference concern S1 closed), ADR-008, #134, #149, #156. + +--- + ### C-74: The þing-01 redaction guard scans four paths that stopped existing — it reports success while covering one root of five — RESOLVED | Field | Value | From 8a7faf43ddecbc7ae513d6b8e23c0757032d301a Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 2 Aug 2026 06:43:26 +0200 Subject: [PATCH 14/23] =?UTF-8?q?docs(ops):=20S8=20#189=20=E2=80=94=20a=20?= =?UTF-8?q?correction=20procedure=20for=20the=20delivery=20that=20exists?= =?UTF-8?q?=20(#206)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit C-22's trigger fired on 2026-07-27. Run-0 put 64,742 cells x 36 months into the UN FAO's store, live, and the written procedure (#15, June) described disk caches and shapefiles — both deleted with the runtime mapper. There was data in a partner's store and no usable way to correct it. Affected deliveries are identified by run_id and lookup_version, both present by construction. That is why this story waited on S5: until C-60, lookup_version could silently be the string "unknown", and a procedure whose identification step rests on a field that can become a placeholder is not a procedure. The wire mechanism is supersession, not retraction, and the document says so rather than inventing one — manifest-last commit ordering means a run is replaced by publishing a new complete run. It also states the consequence a reader would otherwise find the hard way: views-faoapi selects the newest manifest over a broad filter, so a correction is picked up because it is NEWER, not because it is CORRECT, and a partial correction published to production is indistinguishable from the real one. That is C-73, cited rather than re-solved. Confirmation is offline against committed artifacts, because re-running the pipeline to investigate changes the thing being investigated. And if the fidelity suite passes while a value is still wrong, the fault is upstream — C-43's scope split, restated where an operator will meet it at 22:00. One step is written but NOT decided, and it is the step that reaches the partner: who contacts the UN FAO and whether they expect retraction or supersession. Both belong to the operator. The document states them verbatim and tells the reader to stop and ask rather than improvise — an inconsistent first message to a partner is harder to correct than a slow one. C-22 closes because the procedure exists and says exactly where it stops. What remains is a decision, not engineering, and leaving the entry open would read as unfinished work rather than as a question awaiting an answer. docs/CLONING.md carries it forward: answer your partner's correction questions BEFORE first delivery. This repo did not. Co-authored-by: Claude Opus 5 (1M context) --- README.md | 6 + docs/CLONING.md | 11 ++ docs/operations/correction_procedure.md | 164 ++++++++++++++++++++++++ reports/technical_risk_register.md | 57 +++++--- tests/test_doc_accuracy.py | 60 +++++++++ 5 files changed, 278 insertions(+), 20 deletions(-) create mode 100644 docs/operations/correction_procedure.md diff --git a/README.md b/README.md index ff12b06..b9b74f8 100644 --- a/README.md +++ b/README.md @@ -81,6 +81,12 @@ The pandas metadata-join and history-clip stages were retired with the legacy de in #149; their rules survive as called invariants under `delivery/`. See the [manager README](views_postprocessing/unfao/managers/README.md) for what moved where. +### If a delivered value turns out to be wrong + +`docs/operations/correction_procedure.md` — how to establish which deliveries are +affected, confirm the fault offline, and supersede on the wire. The contract has no +retraction primitive; a correction is a new complete run, manifest last. + ### Output schema (geographic metadata columns) These 9 columns are the delivered geography contract (declared in `contract/gaul_schema.py`): diff --git a/docs/CLONING.md b/docs/CLONING.md index e149ce0..40145e8 100644 --- a/docs/CLONING.md +++ b/docs/CLONING.md @@ -80,6 +80,17 @@ used to select a retired code path silently (register C-63). 4. Run `pytest tests/test_clone_readiness.py` in the clone. If it fails, the boundary moved. +## Before your first delivery, not after + +**Answer your partner's correction questions before you ship to them, not after a bad +delivery.** `docs/operations/correction_procedure.md` is FAO's, and its steps 1–3 and 5 +transfer unchanged — they are contract mechanics, not partner specifics. **Step 4 does +not:** who contacts your partner, through what channel, how fast, and whether they +expect a retraction or a supersession are answers only your partner can give. + +This repo shipped run-0 to the UN FAO on 2026-07-27 with that step undecided (register +C-22), and it is still undecided. Do not inherit that. + ## Where the reasoning lives `docs/ADRs/013_sampled_forecast_wire_contract.md` — the wire contract · diff --git a/docs/operations/correction_procedure.md b/docs/operations/correction_procedure.md new file mode 100644 index 0000000..2161402 --- /dev/null +++ b/docs/operations/correction_procedure.md @@ -0,0 +1,164 @@ +# Correcting a delivery that has already reached the UN FAO + +**Read this when a delivered value is suspect.** It is a runbook, not an argument — a +person following it at 22:00 after a bad delivery should not have to read three other +documents first. Where a step needs background, the link is inline and optional. + +Register **C-22**. Supersedes issue #15, whose procedure described the pre-2026-07 +delivery (disk caches, shapefiles) and no longer applies to anything. + +> **Status: steps 1–3 and 5 are executable today. Step 4 is not** — nobody has decided +> who contacts the UN FAO, or whether FAO expects a retraction or a supersession. Those +> are the operator's calls and are stated in §4 exactly as they need to be answered. + +--- + +## 0. What a delivery is, in one paragraph + +A delivery is one **run**, identified by a `run_id` such as +`rusty_bucket_forecasting_20260727_095355`. It consists of arrow **shards** (one per +target per month), one GAUL **sidecar**, one **historical artifact**, and one **run +manifest** — uploaded **last**, deliberately, because it is the *commit marker* +(ADR-013 §4). A run whose upload died halfway has no manifest and is **invisible to +consumers**, which is the contract's atomicity mechanism and the first thing to +understand before touching anything. + +--- + +## 1. Which deliveries are affected? + +Two fields identify a delivery, and both are on it by construction: + +| field | what it answers | where it lives | +|---|---|---| +| `run_id` | *which run* | the manifest, every shard's header, and each store document's filename | +| `lookup_version` | *which GAUL build produced its geography* | the historical artifact's store-document `description` (a compact JSON provenance record) | + +`lookup_version` has the form `land_gaul@f74d3b2b` — region, then the digest of the +views-datafactory ingestion the lookup was built from. **It cannot silently be +`"unknown"`**: since register C-60 the reader raises rather than degrading, so a stamp +you can read is a stamp you can trust. + +**To enumerate affected runs:** + +1. If the fault is in **geography** (a wrong country, admin unit or coordinate), the + blast radius is *every delivery sharing the suspect `lookup_version`*. Compare the + stamp on each delivered run's provenance record. +2. If the fault is in **forecast values**, the blast radius is the runs whose manifest + names the affected producer run — the manifest carries `run_id` and `targets`. +3. If the fault is in **coverage** (missing or extra cells), the manifest's + `expected_cell_count` and the provenance record's `actual_cell_count` bound it + without opening a single parquet. + +Do this before touching the store. A correction whose scope you have not established +is a second incident. + +--- + +## 2. Reproduce and confirm, offline + +**Do not re-run the pipeline to investigate.** The committed artifacts are enough, and +re-running changes the thing you are diagnosing. + +``` +pytest -q tests/test_gaul_lookup_fidelity.py +``` + +26 tests. The always-on half checks the committed lookup for key uniqueness, region-set +equality, coordinate correctness, and the absence of nulls and `-1` sentinels. With a +views-datafactory checkout present (`VIEWS_DATAFACTORY`), the second half compares all +seven GAUL columns against the producer's own parquets. + +**If those pass and a value is still wrong, the fault is upstream, not here.** That +distinction is register C-43's and it is load-bearing: this repository is verified to +carry views-datafactory's answer faithfully; whether that answer is *right* — in +particular the degree-based area-majority join at high latitudes — is +**views-datafactory#387**. Do not correct a delivery to compensate for a producer +defect; fix the producer and rebuild. + +--- + +## 3. Correct on the wire + +**The contract has no retraction primitive. It has supersession**, and that is +deliberate: the manifest-last commit ordering means the way to replace a run is to +publish a *new complete run*, not to mutate an old one. + +1. Fix the cause — the lookup, the producer, or the code — and land it. +2. Rebuild any affected artifact. For the lookup: + `PYTHONPATH=. python scripts/build_gaul_lookup.py --datafactory --region land_gaul` + The builder now **refuses** to produce an artifact it cannot stamp, so a rebuild is + either traceable or it fails. +3. Re-run the delivery. It publishes a new `run_id`, shards → sidecar → **manifest last**. +4. **Do not delete the superseded run's files before the consumer has moved.** The old + manifest is what makes the old run selectable; removing shards while it stands + produces a run that resolves and then fails to load, which is worse than a wrong + value because it looks like an outage. + +### What the consumer will actually pick up — read this before assuming + +views-faoapi resolves **the newest manifest matching a broad filter** +(`wire/source_selection.py`). So a corrected run is picked up because it is *newer*, not +because it is *correct*. Two consequences: + +- **Publishing the correction is what switches the consumer over.** There is no + "activate" step and no rollback — republishing the old run would mean publishing it + again under a new id. +- **Recency-based selection is register C-73**, open, and fixed upstream in + views-pipeline-core 3.0.0 but not yet taken here. **#133** would have the manifest + declare `{maturity, source, required-schema-version}` so a consumer could select on + intent rather than on timestamp. Until then, do not publish a *test* or *partial* + correction to the production bucket — the consumer cannot tell it from the real one. + +--- + +## 4. Telling the UN FAO — **not decided; the operator must answer this** + +Everything above can be executed by whoever is on the keyboard. This cannot, and it is +the step that matters most to the partner. + +**Two questions, in plain language:** + +> **1. When a delivery is found to be wrong, who contacts the UN FAO, through what +> channel, and how quickly?** +> Right now nobody has said. There is no named person, no address, and no expectation +> about timing — so in practice the answer would be improvised by whoever noticed, +> under time pressure, which is the worst moment to invent a process. + +> **2. Does the UN FAO expect us to *retract* the bad delivery, or to *supersede* it?** +> These need different behaviour. Supersession is what the contract does today: the old +> run stays in the bucket and a newer one wins. Retraction would mean removing or +> marking the old run so it cannot be served — which the wire has no mechanism for, and +> which would need an ADR-013 amendment and agreement from views-faoapi. +> **This is a question for them, not a decision for us.** + +Per `CLAUDE.md`, anything touching an external party is the operator's call. Until both +are answered, treat this step as: **stop, and ask Simon.** Do not contact the partner +ad hoc; an inconsistent first message is harder to correct than a slow one. + +--- + +## 5. Preserve the evidence + +A retraction that destroys the evidence makes the post-mortem impossible. Keep, at +minimum: + +- the superseded run's `run_id`, its manifest, and its provenance record +- the `lookup_version` in force at the time, and the lookup artifact that produced it + (it is committed to this repo, so the git history already holds it) +- what was wrong, how it was found, and which check failed to catch it + +The last one is the point. Every silent-failure entry in +`reports/technical_risk_register.md` exists because something shipped that no gate +objected to. A correction that does not record *which gate was missing* buys a fix and +leaves the hole. + +--- + +## See also + +`docs/ADRs/013_sampled_forecast_wire_contract.md` §4 (commit ordering), §11 (transition +rules) · `views_postprocessing/delivery/provenance.py` (the fields a delivery carries) · +`views_postprocessing/contract/wire/source_selection.py` (how the consumer chooses) · +register **C-22**, **C-43**, **C-60**, **C-73** · issues **#15**, **#131**, **#133**, +views-datafactory **#387** diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index a0c0a9e..c11f869 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -6,8 +6,8 @@ | Owner | Dylan Pinheiro / PRIO MD&D Team | | Last Updated | 2026-08-02 | | Total Concerns | 74 | -| Open Concerns | 16 | -| Resolved Concerns | 58 | +| Open Concerns | 15 | +| Resolved Concerns | 59 | --- @@ -173,24 +173,6 @@ See also C-14 (stale cache without version tracking), C-22 (no post-delivery cor --- -### C-22: No post-delivery correction process for wrong assignments - -| Field | Value | -|-------|-------| -| ID | C-22 | -| Tier | 3 | -| Source | `falsification-audit` (2026-06-02) | -| Trigger | When the run-0 integrity verification (#131 q1) or any FAO/faoapi query surfaces a suspect delivered value — follow the correction procedure; **issue #15 must produce one first.** Re-check at every subsequent delivery until it exists. | -| Location | `views_postprocessing/unfao/managers/unfao.py:442-494` (`_save_contract`), `:518-578` (legacy `_save`); issue #15 (the undocumented procedure) | - -The delivery chain has four stages beyond the code: Appwrite bucket → UN FAO download → FAO systems → operational decisions. When an error is discovered post-delivery, correction requires clearing cache, re-running, re-uploading, notifying FAO, and FAO retracting old data. Steps 3-5 have no documented procedure. - -Part of Cluster B (operational impact dimension). See also C-14 (RESOLVED — mapper-era cache), C-15. - -**Update 2026-07-31 (review-rr — the conditional is spent):** this entry was written conditionally — "*if* wrong data ever reaches FAO." **Run-0 delivered on 2026-07-27** (108 arrow shards + sidecar + manifest to `unfao_bucket`, plus 28,356,996 historical rows at 64,742 cells), and its integrity verification is still open (#131 q1). There is now delivered, unverified data in the partner's store and still no documented correction/recall procedure. Tier held at 3 (process gap, no code defect), but this is the acute member of Cluster J — **issue #15 is now the blocking artifact, not a nice-to-have.** - ---- - ### C-24: Postprocessor output schema diverges from FAO-confirmed API contract | Field | Value | @@ -584,6 +566,41 @@ See also C-40 (the inheritance/representation coupling this migration unwinds), ## Resolved Concerns +### C-22: No post-delivery correction process for wrong assignments — RESOLVED (procedure written; the partner-facing step is an open OPERATOR decision) + +| Field | Value | +|-------|-------| +| ID | C-22 | +| Resolved | 2026-08-02 | +| Resolution | **Closed by S8 (#189)** — `docs/operations/correction_procedure.md`, written against the delivery that exists rather than the one #15 described in June (disk caches and shapefiles, both deleted with the runtime mapper). + +**What it establishes.** Affected deliveries are identified by `run_id` and `lookup_version`, both present by construction — and `lookup_version` can no longer be the string `"unknown"`, because **C-60** made the reader raise instead of degrading. That is the dependency this story waited on: a procedure whose identification step rests on a field that can silently become a placeholder is not a procedure. Confirmation is offline against committed artifacts (`tests/test_gaul_lookup_fidelity.py`, 26 tests), so investigating does not change the thing being investigated. + +**The wire mechanism is supersession, not retraction**, and the document says so plainly rather than inventing one: manifest-last commit ordering (ADR-013 §4) means a run is replaced by publishing a new complete run. It also states the consequence a reader would otherwise discover the hard way — views-faoapi selects **the newest manifest over a broad filter**, so a correction is picked up because it is *newer*, not because it is *correct*, and a test or partial correction published to the production bucket is indistinguishable from the real one. That is **C-73**, cited rather than re-solved, with **#133** named as the fix that would let a consumer select on intent. + +Pinned by three checks in `tests/test_doc_accuracy.py`: the document exists and names the identification fields; it describes ADR-013 mechanisms and none of the deleted ones; and it still flags its undecided step. + +**⚠ RESIDUAL — one step is written but NOT decided, and it is the step that reaches the partner.** Two questions belong to the operator (`CLAUDE.md`: anything touching an external party): + +1. **Who contacts the UN FAO when a delivery is found wrong, through what channel, and how fast?** No named person, no address, no timing expectation. In practice it would be improvised by whoever noticed, under time pressure. +2. **Does FAO expect retraction or supersession?** Supersession is what the contract does. Retraction has **no wire mechanism** and would need an ADR-013 amendment plus agreement from views-faoapi. It is a question for them, not a decision for us. + +The document states both verbatim and instructs the reader to stop and ask rather than improvise. **C-22 closes because the procedure now exists and says exactly where it stops**; what remains is a decision, not engineering. Registered as the standing gap rather than left as an open concern that would read as unfinished work. + +`docs/CLONING.md` carries the same warning forward: a clone should answer its partner's correction questions **before** first delivery. This repo shipped run-0 on 2026-07-27 with that step undecided, and it still is. | +| Tier | 3 | +| Source | `falsification-audit` (2026-06-02) | +| Trigger | When the run-0 integrity verification (#131 q1) or any FAO/faoapi query surfaces a suspect delivered value — follow the correction procedure; **issue #15 must produce one first.** Re-check at every subsequent delivery until it exists. | +| Location | `views_postprocessing/unfao/managers/unfao.py:442-494` (`_save_contract`), `:518-578` (legacy `_save`); issue #15 (the undocumented procedure) | + +The delivery chain has four stages beyond the code: Appwrite bucket → UN FAO download → FAO systems → operational decisions. When an error is discovered post-delivery, correction requires clearing cache, re-running, re-uploading, notifying FAO, and FAO retracting old data. Steps 3-5 have no documented procedure. + +Part of Cluster B (operational impact dimension). See also C-14 (RESOLVED — mapper-era cache), C-15. + +**Update 2026-07-31 (review-rr — the conditional is spent):** this entry was written conditionally — "*if* wrong data ever reaches FAO." **Run-0 delivered on 2026-07-27** (108 arrow shards + sidecar + manifest to `unfao_bucket`, plus 28,356,996 historical rows at 64,742 cells), and its integrity verification is still open (#131 q1). There is now delivered, unverified data in the partner's store and still no documented correction/recall procedure. Tier held at 3 (process gap, no code defect), but this is the acute member of Cluster J — **issue #15 is now the blocking artifact, not a nice-to-have.** + +--- + ### C-71: `appwrite_env.assert_env_declared` raises without logging — ADR-008 non-compliance in an entry-validation seam — RESOLVED | Field | Value | diff --git a/tests/test_doc_accuracy.py b/tests/test_doc_accuracy.py index de7500b..6a81cee 100644 --- a/tests/test_doc_accuracy.py +++ b/tests/test_doc_accuracy.py @@ -296,3 +296,63 @@ def test_the_legacy_ok_marker_still_works(tmp_path): "a marker on the NEXT line excused the mention — the opt-out must stay line-scoped, " "or a single marker silently covers a whole document" ) + + +# --- 4. the post-delivery correction procedure (S8 / #189, register C-22) --------------- + +_CORRECTION = _REPO / "docs" / "operations" / "correction_procedure.md" + + +def test_the_correction_procedure_exists_and_names_how_to_identify_a_delivery(): + """C-22's trigger fired on 2026-07-27 and there was no written procedure. + + Run-0 put 64,742 cells x 36 months into a partner's store, live. The procedure + that existed (#15, June) described disk caches and shapefiles — both deleted with + the runtime mapper. This pins the two things a correction cannot start without: + the fields that say *which* delivery is affected. + + Mirrors ``test_clone_readiness.py::test_the_cloning_guide_exists_and_names_what_ + must_be_supplied`` — the human half of a guarantee, checked mechanically. + """ + assert _CORRECTION.exists(), f"the correction procedure is missing: {_CORRECTION}" + text = _CORRECTION.read_text() + for required in ("run_id", "lookup_version", "manifest"): + assert required in text, ( + f"the correction procedure does not mention {required!r} — without it a " + "reader cannot establish which deliveries are affected before acting" + ) + + +def test_the_correction_procedure_describes_the_delivery_that_exists(): + """It must not describe the pre-#149 pipeline, which is how #15 became useless. + + ``_BANNED`` already covers the deleted symbols; this asserts the *positive* — that + the document names the ADR-013 mechanisms a correction actually runs through. + """ + text = _CORRECTION.read_text() + offenders = [ + line.strip() + for line in text.splitlines() + if "legacy-ok" not in line and _BANNED.search(line) + ] + assert not offenders, f"the correction procedure describes deleted code: {offenders}" + for mechanism in ("supersed", "commit marker", "source_selection", "C-73"): + assert mechanism in text, ( + f"the procedure does not mention {mechanism!r}. A correction that ignores " + "how the consumer SELECTS a run will publish a fix nobody picks up." + ) + + +def test_the_procedure_states_the_questions_only_the_operator_can_answer(): + """The two external-party decisions must stay visible, not quietly become defaults. + + Who contacts the UN FAO, and whether they expect retraction or supersession. Per + CLAUDE.md both are the operator's; the failure mode is that an undecided step gets + silently improvised the first time it is needed, under time pressure. + """ + text = _CORRECTION.read_text() + assert "not decided" in text.lower(), ( + "the procedure no longer flags its undecided step — if it has been decided, " + "replace the marker with the decision and say who made it" + ) + assert "UN FAO" in text and "supersede" in text From 157cc06390f747aa80fd87de4714f7e578f7e797 Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 2 Aug 2026 12:05:19 +0200 Subject: [PATCH 15/23] Record the operator's correction decisions; put the FAO half to them formally (#207) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs(ops): record the operator's correction decisions; put the FAO half to them formally Simon Polichinel von der Maase is responsible for contacting the FAO when a delivery is found wrong — direct email, not an automated alert, as soon as the scope is established rather than after a correction is prepared. The intended treatment of a bad delivery is WITHDRAWAL. What is implemented is SUPERSESSION, and the procedure now states that gap in the section an operator reads at 22:00, because believing a bad delivery becomes unretrievable when it does not is worse than knowing it stays. Withdrawal has no wire mechanism: it needs an ADR-013 amendment and matching views-faoapi work, and whether that is worth building depends on FAO's answer about audit requirements. Both are put to FAO as Pre-Release Note 07 Topic B, with a table of the interim defaults marked as placeholders rather than policy — including the one that is the option we are NOT recommending. Recipients are deliberately not in this repository. It is public. Naming a responsible person on our side is one thing; publishing an external organisation's individual email addresses to a public repo as a side effect of documenting a runbook is another. The contacts live in the FAO-02 project materials. tests/test_doc_accuracy.py now refuses @fao.org anywhere in the repo, mutation-proven. The doc-accuracy guard fired on this change, which is what it was written for: it required that removing the "not decided" marker be accompanied by the decision. Its replacement guards the subtler failure — a half-answered question quietly becoming a whole answer, so the outstanding half stops being asked. Co-Authored-By: Claude Opus 5 (1M context) * fix: the address guard fired on the sentence describing the address guard Third instance in this epic of a guard catching the prose that explains it, after S3's retired contract name and S5's ledger-schema docstring. The fix is the same each time: name the thing without spelling it. Worth keeping the note rather than silently rewording, because the alternative each time was to weaken the guard, and three times would have made it worthless. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) --- docs/operations/correction_procedure.md | 62 +++++++++++++-------- reports/technical_risk_register.md | 10 ++++ tests/test_doc_accuracy.py | 72 +++++++++++++++++++++---- 3 files changed, 113 insertions(+), 31 deletions(-) diff --git a/docs/operations/correction_procedure.md b/docs/operations/correction_procedure.md index 2161402..90fa179 100644 --- a/docs/operations/correction_procedure.md +++ b/docs/operations/correction_procedure.md @@ -80,9 +80,21 @@ defect; fix the producer and rebuild. ## 3. Correct on the wire -**The contract has no retraction primitive. It has supersession**, and that is -deliberate: the manifest-last commit ordering means the way to replace a run is to -publish a *new complete run*, not to mutate an old one. +**The intended policy is WITHDRAWAL. What is implemented is SUPERSESSION.** That gap is +deliberate, recorded, and you need to know about it before you act. + +- **Intended** (operator decision, 2026-08-02): a delivery found to be materially wrong + should be *removed or blocked* so it can no longer be served. Data known to be wrong + should not stay retrievable through the production interface. +- **In force**: supersession. The manifest-last commit ordering means a run is replaced + by publishing a *new complete run*; the old one stays in the bucket and remains + retrievable by a client that asks for it specifically. + +Supersession is in force **because it is what the wire does**, not because it was chosen. +Withdrawal has no mechanism today: it needs an ADR-013 amendment and matching work in +views-faoapi. Put to FAO as **Pre-Release Note 07, Decision Point B.2**, which also asks +whether they have an audit or reproducibility requirement that would argue *against* +withdrawal — that answer decides whether the amendment is worth building. 1. Fix the cause — the lookup, the producer, or the code — and land it. 2. Rebuild any affected artifact. For the lookup: @@ -112,31 +124,37 @@ because it is *correct*. Two consequences: --- -## 4. Telling the UN FAO — **not decided; the operator must answer this** +## 4. Telling the UN FAO -Everything above can be executed by whoever is on the keyboard. This cannot, and it is -the step that matters most to the partner. +**Simon Polichinel von der Maase is responsible for making contact.** Not an automated +alert — a correction needs judgement about scope and impact that an automated message +cannot carry. -**Two questions, in plain language:** +**Recipients and channel are recorded outside this repository.** This repo is **public**, +and publishing named FAO staff members' email addresses in it is not something to do by +default. The contacts are in the FAO-02 project materials +(`brain/2_projects/fao02/…/prerelease_notes/fao_02_pre_release_note_07/`, Decision +Point B.1) and in the operator's address book. -> **1. When a delivery is found to be wrong, who contacts the UN FAO, through what -> channel, and how quickly?** -> Right now nobody has said. There is no named person, no address, and no expectation -> about timing — so in practice the answer would be improvised by whoever noticed, -> under time pressure, which is the worst moment to invent a process. +**Send as soon as the scope of the error is established** — not after a correction has +been prepared. The partner's ability to act is time-sensitive and independent of our +remediation timeline. -> **2. Does the UN FAO expect us to *retract* the bad delivery, or to *supersede* it?** -> These need different behaviour. Supersession is what the contract does today: the old -> run stays in the bucket and a newer one wins. Retraction would mean removing or -> marking the old run so it cannot be served — which the wire has no mechanism for, and -> which would need an ADR-013 amendment and agreement from views-faoapi. -> **This is a question for them, not a decision for us.** +### Still awaiting FAO's answer -Per `CLAUDE.md`, anything touching an external party is the operator's call. Until both -are answered, treat this step as: **stop, and ask Simon.** Do not contact the partner -ad hoc; an inconsistent first message is harder to correct than a slow one. +Two things are proposed but not confirmed, and both are put to FAO in **Pre-Release +Note 07, Topic B**: ---- +1. **Are those the right recipients, and is there an expected notification period?** A + named individual on leave is a single point of failure in exactly the situation where + delay is costly. FAO may prefer a shared address or a rota. +2. **Does FAO expect a correction notice for *every* error, or only one material to + published outputs?** A wrong admin label on a handful of cells and a wrong forecast + across a region are different events; treating them alike either floods them with + immaterial notices or buries a serious one. + +Until FAO answers, follow what is written above. Do not improvise a different channel — +an inconsistent first message to a partner is harder to correct than a slow one. ## 5. Preserve the evidence diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index c11f869..8c60d48 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -580,6 +580,16 @@ See also C-40 (the inheritance/representation coupling this migration unwinds), Pinned by three checks in `tests/test_doc_accuracy.py`: the document exists and names the identification fields; it describes ADR-013 mechanisms and none of the deleted ones; and it still flags its undecided step. +**⚠ UPDATE 2026-08-02, later the same day — the operator answered the PRIO half; the FAO half is now formally asked.** + +- **Who notifies:** **Simon Polichinel von der Maase**, by direct email, as soon as the scope of the error is established. Adopted. +- **Treatment of an affected delivery:** the intended policy is **withdrawal**. What is *implemented* is **supersession**, and the procedure now says so explicitly — supersession is in force because it is what the wire does, not because it was chosen. Withdrawal needs an ADR-013 amendment plus views-faoapi work, and whether that is worth building depends on FAO's answer about audit requirements. +- **Put to FAO** as **Pre-Release Note 07, Topic B** (Decision Points B.1 and B.2), which also records the interim defaults in force as placeholders rather than policy. + +Recipients are deliberately **not in this repository**, which is public; naming a responsible person on our side is one thing, publishing an external organisation's individual email addresses is another. `tests/test_doc_accuracy.py` now refuses partner address strings anywhere in the repo, mutation-proven — and fired on the first draft of this very sentence, which spelled the pattern out. Third time in this epic that a guard has caught the prose explaining it (after S3's retired contract name and S5's ledger-schema docstring); the fix is the same each time — name the thing without spelling it. + +The original residual, kept for the record: + **⚠ RESIDUAL — one step is written but NOT decided, and it is the step that reaches the partner.** Two questions belong to the operator (`CLAUDE.md`: anything touching an external party): 1. **Who contacts the UN FAO when a delivery is found wrong, through what channel, and how fast?** No named person, no address, no timing expectation. In practice it would be improvised by whoever noticed, under time pressure. diff --git a/tests/test_doc_accuracy.py b/tests/test_doc_accuracy.py index 6a81cee..896ba0d 100644 --- a/tests/test_doc_accuracy.py +++ b/tests/test_doc_accuracy.py @@ -343,16 +343,70 @@ def test_the_correction_procedure_describes_the_delivery_that_exists(): ) -def test_the_procedure_states_the_questions_only_the_operator_can_answer(): - """The two external-party decisions must stay visible, not quietly become defaults. +def test_the_procedure_names_who_notifies_and_still_flags_what_fao_has_not_answered(): + """Half of §4 is decided; half is not, and both halves must stay visible. + + **Decided 2026-08-02 (operator):** Simon Polichinel von der Maase makes contact, and + the intended treatment of a bad delivery is *withdrawal*. **Not decided:** whether + FAO agrees the recipients and timing (B.1), and whether they have an audit + requirement arguing against withdrawal (B.2). Both are put to them in Pre-Release + Note 07, Topic B. + + An earlier version of this test asserted the document said "not decided", and its + docstring instructed whoever removed that marker to replace it with the decision. + That is what happened — the guard fired on the operator's answer, which is the + behaviour it was written for rather than a false positive. + + The failure mode it now guards is subtler: a half-answered question quietly becoming + a whole answer, so the outstanding half stops being asked. + """ + text = _CORRECTION.read_text() + assert "Simon Polichinel von der Maase is responsible" in text, ( + "the procedure no longer names who contacts the partner — an unowned step is " + "improvised by whoever notices, under time pressure" + ) + assert "Still awaiting FAO's answer" in text, ( + "the procedure no longer flags what FAO has not answered. If they have " + "answered, record the answer and the date — do not simply drop the question." + ) + assert "Pre-Release Note 07" in text, ( + "the procedure must cite where the outstanding questions were put, or they " + "become questions nobody remembers asking" + ) + + +def test_the_procedure_distinguishes_intended_policy_from_what_is_implemented(): + """The gap that would otherwise be discovered mid-incident. - Who contacts the UN FAO, and whether they expect retraction or supersession. Per - CLAUDE.md both are the operator's; the failure mode is that an undecided step gets - silently improvised the first time it is needed, under time pressure. + Withdrawal is the decision; supersession is what the wire actually does, and it is + in force only because nothing else exists. An operator reading this at 22:00 must + not believe a bad delivery becomes unretrievable when it does not. """ text = _CORRECTION.read_text() - assert "not decided" in text.lower(), ( - "the procedure no longer flags its undecided step — if it has been decided, " - "replace the marker with the decision and say who made it" + assert "intended policy is WITHDRAWAL" in text.replace("**", "") + assert "implemented is SUPERSESSION" in text.replace("**", "") + assert "ADR-013 amendment" in text, ( + "the procedure must say what withdrawal would COST — otherwise the gap reads " + "as an oversight rather than as unbuilt work with a known price" + ) + + +def test_no_partner_contact_details_are_published_in_this_repository(): + """This repo is public. FAO staff email addresses do not belong in it. + + The contacts live in the FAO-02 project materials and the operator's address book. + Naming a responsible person on our side is fine; publishing an external + organisation's individual addresses to a public repository is not something to do + as a side effect of documenting a runbook. + """ + offenders = [] + for doc in (*sorted(_REPO.rglob("*.md")), *sorted(_PKG.rglob("*.py"))): + if ".git" in doc.parts: + continue + for number, line in enumerate(doc.read_text(errors="ignore").splitlines(), 1): + if "@fao.org" in line.lower(): + offenders.append(f"{doc.relative_to(_REPO)}:{number}") + assert not offenders, ( + f"partner contact addresses appear in this public repository: {offenders}. " + "Keep them in the project materials; reference the decision, not the address." ) - assert "UN FAO" in text and "supersede" in text From 97bddb085fc4ef3fe361cb424a955248bc641993 Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 2 Aug 2026 12:52:54 +0200 Subject: [PATCH 16/23] =?UTF-8?q?docs(adr):=20S9=20#190=20=E2=80=94=20epic?= =?UTF-8?q?=20#181=20closeout,=20and=20ADR-014=20for=20the=20rules=20no=20?= =?UTF-8?q?test=20carries=20(#208)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Ten register entries closed, one cluster closed, open count 24 -> 15. Every remaining entry is classified: six blocked on the pipeline-core 3.0.0 publish, five owned by another repo, four deliberately deferred. None unexamined. ADR-014 records what the epic learned. The argument for writing nothing was genuinely strong for most of it: a contributor who violates the register's closing-condition convention already gets a failure naming the exact strings and the fix, at the moment of violation, which teaches better than an ADR nobody opens. What that argument leaves unrecorded is the two clauses no test carries — a deferral needs a trigger and an owner, and a change that names a record disposes of that record in the same change. An ADR for the mechanisable rules would be ceremony; one for the un-mechanisable ones is the only vehicle there is. §5 says explicitly that no test enforces it and none is proposed, and records the attempt so the next person does not spend the same afternoon on it. The evidence that it lapses is the epic's own: C-71 fixed in the morning and filed open all day while eight stories shipped; the CIC lagging three consecutive stories, caught by review each time and by nothing mechanical. The closeout states what was NOT done, at length, because a closeout reporting only successes is the defect this epic exists to fix: Cluster M untouched and correctly so; the CI question decided in writing but unowned; withdrawal of a bad delivery chosen but deliberately unbuilt pending FAO's answer; two questions with the partner; the deploy gate's xfail tuning left alone on purpose; and a local Python below the declared floor that skips three checks. The sharpest thing the epic produced is smaller than any of its stories: existence is not reachability. Co-authored-by: Claude Opus 5 (1M context) --- ...4_claims_and_the_guards_that_carry_them.md | 154 ++++++++++++++++++ docs/ADRs/README.md | 7 + reports/technical_risk_register.md | 34 ++++ 3 files changed, 195 insertions(+) create mode 100644 docs/ADRs/014_claims_and_the_guards_that_carry_them.md diff --git a/docs/ADRs/014_claims_and_the_guards_that_carry_them.md b/docs/ADRs/014_claims_and_the_guards_that_carry_them.md new file mode 100644 index 0000000..13292df --- /dev/null +++ b/docs/ADRs/014_claims_and_the_guards_that_carry_them.md @@ -0,0 +1,154 @@ +# ADR-014: Claims, and the guards that carry them + +**Status:** Accepted +**Date:** 2026-08-02 +**Deciders:** Project maintainers (PRIO MD&D Team) +**Arises from:** epic [#181](https://github.com/views-platform/views-postprocessing/issues/181) +**Related:** [ADR-003](003_authority_of_declarations_over_inference.md) (declarations over +inference), [ADR-005](005_testing_as_mandatory_critical_infrastructure.md) (testing as +infrastructure), [ADR-010](010_technical_risk_register.md) (the register) + +--- + +## Context + +Over two days in August 2026, this repository found **seven** places where it said one +thing and did another. None was a bug in the delivery. All were claims that had outlived +what they described: + +- three register entries filed as open with their own stated closing conditions met + (C-43, C-59, C-61) — one of them saying, in its own body, *"closes when + `tests/test_gaul_lookup_fidelity.py` is committed and green"*, with the file committed + and green; +- two issues closed with their work partly undone (#154, #158) — #158's rename also + broke a URL by find-and-replacing a filename inside it; +- a test file asserting *"the logic tested matches `unfao.py:_validate()` exactly"* about + a method it had stopped resembling seven weeks earlier, while running 43 cases against + a function it defined itself; +- a security-adjacent guard scanning **6 files where it declared 17**, because four of + its five declared roots had been moved and `rglob` on a vanished directory returns + silence rather than an error. + +The pattern is not carelessness. Each was written accurately and became false when +something else moved. What they share is that **nothing was attached to them that would +notice**. + +ADR-003 already forbids inferring what should be declared. This ADR is that rule applied +one level up: **a declaration that nothing validates is an inference with better +grammar.** + +--- + +## Decision + +### §1 A guarantee is attached to a check, or it is not a guarantee + +When this repository asserts something a reader would rely on — in code, in a docstring, +in an ADR, in the register — the assertion is accompanied by something that fails when it +stops being true. If no such check is possible, §4 applies. + +This extends to the repository's own governance artefacts. The register is checked by +`tests/test_register_integrity.py`; the living docs by `tests/test_doc_accuracy.py`; the +package boundary by `tests/test_clone_readiness.py`. **A governance artefact is not exempt +from the rule it exists to enforce.** + +### §2 A guard is mutation-proven, or it is decoration + +A new guard is demonstrated to **fail on the defect it was written for**, and that +demonstration is recorded in the pull request. A guard that has never been watched fail is +a guard whose shape nobody knows. + +Two failure modes, both observed here, and both must be considered: + +- **too narrow** — the þing-01 redaction guard passed while covering a third of its + declared surface; +- **too broad** — a first draft of the coordinate-copy check flagged a *function name* in + this repository as a leaked value. + +Where a guard's inputs are declared (a path list, a module list, a set of names), **assert +that the inputs are real**. `rglob` on a nonexistent directory yields nothing rather than +raising; a list of paths that no longer exist does not fail a scan, it empties it. + +### §3 Prefer a false negative to a false alarm + +A guard that cries wolf gets deleted, and then the rule it carried is unguarded — which is +strictly worse than the narrow guard that would have caught most cases. When a guard fires +on something legitimate, **the first question is whether the matching is wrong, not +whether the scope is too wide.** In the one instance where scope was narrowed instead, the +result missed the shape the defect would actually take. + +Corollary, learned expensively: **existence is not reachability.** A cross-repo pin was +verified to exist, its files were verified to exist at it, and every check written at the +time passed — but it sat on an unmerged branch, declared a version never ratified, and was +withdrawn. Where a claim is about what another repository *says*, check reachability from +that repository's `main`. + +### §4 A deferral is attached to a trigger and an owner, or it is not a deferral + +Deliberate duplication, an unbuilt fix, an unanswered question: each is recorded with the +**named event** that should reopen it. Not "later", not "when convenient" — a thing that +will observably happen. + +Worked examples now in force: the two entry-validation modules stay duplicated until *a +third one is written*; sidecar version-stamping waits for *the next ADR-013 version bump*; +scheduled cross-repo checks wait until *an upstream change reaches the partner through +this repo without anyone noticing first*. + +The counter-example is why this clause exists. Epic #148's retired code path was kept +"until run 0 proves the contract path live" — a real condition, with no owner and no +trigger. Run 0 happened on 2026-07-27 and nobody opened the box; the cleanup took a +subsequent epic. + +### §5 A change that names a record disposes of that record in the same change + +If a pull request cites a register entry, an issue, a CIC or an ADR as the thing it +addresses, it updates that artefact **in the same pull request**. Not in a follow-up, not +at closeout, not from memory. + +**No test enforces this and none is proposed.** It was attempted: no expression reliably +separates *"this entry describes work that is done"* from prose, and a guard that guesses +is one that gets deleted (§3). This clause is therefore a habit, and it is written down +precisely because it is the one rule here that cannot be mechanised — which makes it the +one most likely to lapse. + +The evidence that it does lapse is this epic's own record. C-71 was fixed on the morning of +2026-08-02 and sat filed open for the rest of the day while eight further stories shipped. +The CIC lagged the change in **three consecutive stories** and was corrected by review each +time, never by anything mechanical. + +--- + +## Consequences + +**Cost.** Every guard now carries a mutation demonstration, which is roughly a third again +of the work of writing it. Every deferral needs an argued trigger, which is harder than +writing "later" and is meant to be: if no trigger can be named, the deferral is probably a +decision being avoided. + +**What this does not license.** Not every statement needs a test. §1 is about assertions a +reader would *rely on* — a contract, a boundary, a count, a closing condition. Prose that +explains reasoning is not a claim in this sense, and three times during this epic a guard +fired on the very sentence describing it. The fix each time was to name the thing without +spelling it, **never to weaken the guard** — three weakenings would have left nothing. + +**Where it will be violated first.** §5, because it is the only clause a contributor can +break while every test passes. + +--- + +## Alternatives considered + +**Write nothing; the guards teach themselves.** Genuinely tempting for §1–§3: a +contributor who violates the register's closing-condition convention gets a failure naming +the exact strings and the fix, at the moment of violation — better teaching than an ADR +nobody opens. Rejected because it leaves §4 and §5 unrecorded, and those are the two +clauses no test carries. An ADR that exists for the mechanisable rules would be ceremony; +one that exists for the *un*-mechanisable ones is the only vehicle available. + +**A checklist in the PR template instead.** Cheaper and closer to the moment of work. +Rejected as a replacement, not as a supplement: a checklist records what to do and not why, +and the reasoning is what survives a contributor deciding the rule does not apply to them. +Worth adding later, pointing here. + +**Widen `test_register_integrity` to catch §5.** Attempted and abandoned — see §5. Recorded +so the next person does not spend the same afternoon on it. diff --git a/docs/ADRs/README.md b/docs/ADRs/README.md index 77f4045..9e8ae78 100644 --- a/docs/ADRs/README.md +++ b/docs/ADRs/README.md @@ -72,6 +72,13 @@ These ADRs form the architectural constitution of the repository. sidecar, and the no-collapse boundary (delivery/draws.py). Supersedes the phantom "platform ADR-046". Adopted 2026-07-15 via views-models#149. +- **ADR-014** — Claims, and the Guards That Carry Them + ADR-003 applied to the repository's own statements: a guarantee is attached to a check, + a guard is mutation-proven or it is decoration, a false negative beats a false alarm, a + deferral names a trigger and an owner, and a change that names a record disposes of that + record in the same change. Arises from epic #181, which found seven places where this + repo said one thing and did another. + ADRs numbered 010 and above define: - Domain-specific decisions diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 8c60d48..7230a3f 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -90,6 +90,40 @@ covered a single open entry (see Historical clusters below). **The lesson worth carrying, and it is D-11's:** WET-before-DRY was applied *correctly* — the pandas and frame seams ran as deliberate siblings through the migration, and a premature abstraction would have outlived the implementation it existed to unify. What went wrong was not the duplication; it was that the removal condition (*"until run 0 proves the contract path live"*) was written down without a **named trigger to act on**, so the box expired on 2026-07-27 and nobody opened it. D-11 predicted exactly this. **A deferral needs an owner and a trigger, not just a reason.** +### Epic #181 closeout — "every claim checkable" (2026-08-02) + +**Eleven stories, ten register entries closed, one cluster closed, one ADR written.** Recorded here rather than only in the issue tracker, because an epic that ends in a closed issue ends nowhere a future reader looks. + +| closed | by | proven by | +|---|---|---| +| **C-71** ADR-008 in the entry validator | S1 | a check parametrised over **both** validators, so the pair cannot drift again | +| **C-43, C-59, C-61** Cluster K's build-time guarantees | S2 | `tests/test_gaul_lookup_fidelity.py`, 26 tests | +| **C-03** the `_validate` replica | S4 | 43 self-referential tests replaced by 14 against the real gate | +| **C-60** the lookup's declared version | S5 | the artifact rebuilt, values byte-identical, stamp unchanged | +| **C-57** coordinate-registry drift | S6 | four checks, each mutation-proven | +| **C-46** the machine-specific path | S7 | `grep -rn "/home/"` over the repo → 0 | +| **C-74** the þing-01 redaction guard | S10 | scan coverage 6 → 17 files | +| **C-22** the correction procedure | S8 | written for the delivery that exists; partner-facing step decided 2026-08-02 | + +Plus, outside the register: #158's rename finished and the **broken URL it created** in ADR-013 §7d repaired (Erratum E2); #154's undone half completed across four living documents; #15 superseded; a cross-repo pin re-taken after #196 showed it sat on an unmerged branch. + +**Open count 24 → 15.** Every remaining entry is classified: **six** blocked on the views-pipeline-core 3.0.0 publish (Cluster M), **five** owned by another repo (C-13, C-24, C-26, C-27, C-28), **four** deliberately deferred (C-15, C-30, C-33, C-40). None is unexamined. + +**What the epic did NOT do, stated because a closeout that reports only successes is the defect this epic exists to fix:** + +1. **Cluster M is untouched and correctly so.** Six entries, two of them Tier 2, all resolving on one upstream publish. No engineering here moves them. +2. **The CI question is decided in writing but not implemented.** Three gated cross-repo checks run nowhere automatic. C-46's residual carries the argued recommendation — *do not couple per-PR CI to another repo's default branch; if wanted, a weekly scheduled check that opens an issue on divergence* — with a named trigger. **It is a decision awaiting an owner, not a task awaiting effort.** +3. **Withdrawal of a bad delivery is the chosen policy and is not built.** Supersession is in force because it is what the wire does. Deliberately not started: it needs an ADR-013 amendment plus views-faoapi work, and FAO's answer on audit requirements (Pre-Release Note 07, B.2) decides whether it is wanted at all. +4. **Two questions are with the UN FAO**, not with us — recipients and notification timing (B.1), withdrawal versus supersession (B.2). +5. **`test_datafactory_deploy_readiness`'s `xfail` tuning was left alone**, deliberately: S7 fixed how the checkout is found, not what the gate asserts. If it needs re-pinning now that datafactory has moved past `v1.4.0`, that is a separate judgement. +6. **The local Python is 3.10 while `pyproject` declares `>=3.11`**, so three checks skip on the maintainer's machine and run in CI. Not a repo defect; recorded because "a gate that does not run" is the shape C-46 was open for. + +**The lessons are in [ADR-014](../docs/ADRs/014_claims_and_the_guards_that_carry_them.md)** — including §5, the one rule no test can carry, which is written down *because* it cannot be mechanised. The attempt to mechanise it is recorded there too, so the next person does not repeat it. + +**The sharpest thing the epic produced** is smaller than any of its stories: **existence is not reachability.** A cross-repo pin existed, its files existed at it, every check written at the time passed — and it had never reached `main`. + +--- + ### Cluster M: Six open concerns, one upstream publish **Root cause:** this repo pins `views-pipeline-core >=2.1.3,<3.0.0`, which resolves 2.3.0 from PyPI. Every fix and every removal below exists **only** on pipeline-core's unreleased 3.0.0. None is engineering work here; all five arrive together with one pin bump, and none can be taken before that bump. **Entries:** **C-44** (the bump itself, deliberately held), **C-62** (the transitive drag — 3.4 GB venv, 31 of 32 Dependabot alerts), **C-72** (the pyarrow CVE whose fix our ceiling excludes), **C-73** (the Tier-2 stale-run selection defect, fixed upstream in their #341), **C-58** (the Tier-2 auto-provision-instead-of-raise, fixed upstream in their #322/#331/#332), **C-07** (the undeclared `appwrite` dependency, whose transitive path their #345 withdraws). From 72ce7dac4d71a580d5e09721762d7b229151524b Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Sun, 2 Aug 2026 13:01:49 +0200 Subject: [PATCH 17/23] =?UTF-8?q?chore(deps):=20bump=20views-frames=20to?= =?UTF-8?q?=20>=3D1.10.2,<2=20=E2=80=94=20verified=20wire-byte-identical?= =?UTF-8?q?=20first=20(#209)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit views-frames sits on the byte-producing path: wire/shard.py writes each Hop-B shard through views_frames.io.arrow.save, and the §10 golden fixture is generated the same way. A version bump there is a candidate contract event, so it was checked rather than assumed. Verified in a clean venv at pyarrow 16.1.0 (the pinned version, not this machine's 23.0.1) with views-frames 1.10.2: the fixture generator reproduces ALL FIVE canonical artifacts byte-identically — arrow shard, tap zip, both manifests, sidecar. The committed SHA256SUMS is unchanged and the fixture bytes are untouched. The same venv runs the full suite at 312 passed, 0 failed. The five failures seen locally are purely this machine's pyarrow 23.0.1 against the <17 pin (C-72); they disappear at the pinned version, which is a useful confirmation that the known-failure set really is what it has been claimed to be all along. Lock diff is one package: 1.6.0 -> 1.10.2 plus a docs extra that appeared in the new release's metadata. pyarrow stays at 16.1.0. views-frames 1.10.2 declares pyarrow>=14,<20 for its arrow extra, so the ceiling is compatible. Also corrects the fixture README, which named views_frames 1.0.0 as a pinned tool version. That was stale — the lock had said 1.6.0 for weeks — and it conflated the two dependencies. pyarrow is the version-sensitive one; views_frames is not, and the README now says which is which and what evidence supports it. Co-authored-by: Claude Opus 5 (1M context) --- poetry.lock | 9 +++++---- pyproject.toml | 2 +- tests/fixtures/wire_contract/README.md | 14 +++++++++++--- 3 files changed, 17 insertions(+), 8 deletions(-) diff --git a/poetry.lock b/poetry.lock index 96069ae..3b06a83 100644 --- a/poetry.lock +++ b/poetry.lock @@ -4537,14 +4537,14 @@ dataframe = ["pandas (>=1.5.3,<2.0.0)"] [[package]] name = "views-frames" -version = "1.6.0" +version = "1.10.2" description = "The VIEWS platform data-contract layer: immutable array+identifier frames (numpy only, root of the dependency DAG)." optional = false python-versions = ">=3.10" groups = ["main"] files = [ - {file = "views_frames-1.6.0-py3-none-any.whl", hash = "sha256:bcd09ebf71275f6ad81111dd028fc7e969fc1d514a4364b72f12a3b155d1bf5e"}, - {file = "views_frames-1.6.0.tar.gz", hash = "sha256:081f8f236d9587fe23ed4ae542399f64c0cfe3745b74f37c835a12f905931871"}, + {file = "views_frames-1.10.2-py3-none-any.whl", hash = "sha256:ace7f6a09bdb743edbb39e7e95139a0c647ef7c6100ef4c612a3ce5975078ef0"}, + {file = "views_frames-1.10.2.tar.gz", hash = "sha256:96b4916afe9b64389045043ec1cf05fa2ea9b872ac0d12cb57f0d629ffd88106"}, ] [package.dependencies] @@ -4552,6 +4552,7 @@ numpy = ">=1.26,<3" [package.extras] arrow = ["pyarrow (>=14,<20)"] +docs = ["jupyterlab (>=4,<5)", "matplotlib (>=3.8,<4)", "nbmake (>=1.5,<2)", "pyarrow (>=14,<20)", "scipy (>=1.11,<2)"] [[package]] name = "views-pipeline-core" @@ -4800,4 +4801,4 @@ viz = ["matplotlib", "nc-time-axis", "seaborn"] [metadata] lock-version = "2.1" python-versions = ">=3.11,<3.15" -content-hash = "5f2d192a6a36dd429b473195ca6dc30424b57aed8890fbce99741895015578ef" +content-hash = "81e3ce9892400ac7a3aa4469015b2695f2d3f9c3cee244e5c56c8fe8fee4077a" diff --git a/pyproject.toml b/pyproject.toml index 87de5af..a704039 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -11,7 +11,7 @@ license = "MIT" [tool.poetry.dependencies] python = ">=3.11,<3.15" views-pipeline-core = ">=2.1.3,<3.0.0" -views-frames = ">=1.0,<2" +views-frames = ">=1.10.2,<2" pyarrow = ">=16.1.0,<17.0.0" [build-system] diff --git a/tests/fixtures/wire_contract/README.md b/tests/fixtures/wire_contract/README.md index 64b6299..83bb000 100644 --- a/tests/fixtures/wire_contract/README.md +++ b/tests/fixtures/wire_contract/README.md @@ -29,6 +29,14 @@ injected fixed literals (`run_id="fixture_run_0"`, `generated_at="2026-07-15T00: ## Regeneration `PYTHONPATH=. python3 scripts/build_wire_fixture.py` — byte-reproducible **with the pinned -tool versions** (numpy per lockfile, `pyarrow 16.1.0`, `views_frames 1.0.0`; parquet bytes -vary across pyarrow versions). The committed bytes + `SHA256SUMS` are canonical regardless. -**A change to this fixture is a change to the contract (§10)** — do not regenerate casually. +tool versions** (numpy per lockfile, `pyarrow 16.1.0`). The committed bytes + `SHA256SUMS` +are canonical regardless. **A change to this fixture is a change to the contract (§10)** — +do not regenerate casually. + +**`pyarrow` is the version-sensitive one; `views_frames` is not.** Parquet bytes vary +across pyarrow versions — that is why the pin is `>=16.1.0,<17.0.0` and why a local +pyarrow 23.x fails byte-parity while CI passes (register **C-72**). `views_frames` sits +above it: the fixture was generated under `1.0.0`, CI has reproduced it continuously under +`1.6.0`, and `1.10.2` was verified to regenerate **all five artifacts byte-identically** +before the pin was raised (2026-08-02). Raising it again does not require regenerating the +fixture — but it does require proving that, the same way. From 32c3c16b8797388e85e449fff18651a6f5ec4c40 Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Mon, 3 Aug 2026 02:15:49 +0200 Subject: [PATCH 18/23] =?UTF-8?q?Re-pin=20to=20registry=20v1.4.0=20?= =?UTF-8?q?=E2=80=94=20the=20drift=20detector's=20first=20real=20firing=20?= =?UTF-8?q?(#212)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(seam): re-pin to registry v1.4.0 — the drift detector's first real firing S6 (#187) built a check that fails when views-appwrite's coordinate registry moves without this repo re-verifying against it. It fired on 2026-08-03, naming the version and the action: the Appwrite Seam Contract's registry moved to v1.4.0; this repo declares v1.3.0. Re-verify appwrite_env's declaration against v1.4.0, then bump SEAM_CONTRACT_VERSION and SEAM_CONTRACT_COMMIT together. A true positive on its first firing, and it turned development red rather than letting the divergence sit unnoticed — which is the whole point. The upstream change graduates four CRAFD coordinates from reserved to real and records the CRAFD caller key as issued. Its own header says the bump "imposes no new obligation on any consumer". Re-verified rather than taken on trust, because a bumped pin ASSERTS that someone checked: all 13 names this repo declares are present at v1.4.0 with unchanged class, and nothing was added or removed across connection/target/secret. The claim holds for us mechanically. Pin bumped as a pair — version and commit together, never one alone — to 20dfd0f, confirmed an ancestor of views-appwrite main with both cited files present at it. That reachability check is the one #196 taught us to make: the withdrawn b54928f existed, its files existed at it, and it had never reached main. ADR-013 §7d's two URLs follow the pin, with a dated erratum recording that the detector worked. Co-Authored-By: Claude Opus 5 (1M context) * docs(adr): past-tense the superseded pin Two erratum entries still read "both links now pin 47172af". They pin 20dfd0f as of this change, so the record was asserting a superseded pin as current — the exact stale-claim class ADR-014 exists for, in the document that records the last time it happened. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 5 (1M context) --- .../013_sampled_forecast_wire_contract.md | 24 ++++++++++++++----- views_postprocessing/unfao/appwrite_env.py | 8 +++---- 2 files changed, 22 insertions(+), 10 deletions(-) diff --git a/docs/ADRs/013_sampled_forecast_wire_contract.md b/docs/ADRs/013_sampled_forecast_wire_contract.md index 07b4b88..62cdcfb 100644 --- a/docs/ADRs/013_sampled_forecast_wire_contract.md +++ b/docs/ADRs/013_sampled_forecast_wire_contract.md @@ -689,10 +689,10 @@ a "simplification." this contract governs the **data** crossing the Appwrite seam; the **identity/config** crossing the same seam (which key, which coordinates, sourced from where, named how) is governed by - **[the Appwrite Seam Contract](https://github.com/views-platform/views-appwrite/blob/47172af/docs/ADRs/platform/appwrite_seam_contract.md)**, + **[the Appwrite Seam Contract](https://github.com/views-platform/views-appwrite/blob/20dfd0f/docs/ADRs/platform/appwrite_seam_contract.md)**, homed in views-appwrite and referenced here **by pinned URL, never by copy** — with its coordinate registry - ([`coordinate_registry.toml`](https://github.com/views-platform/views-appwrite/blob/47172af/docs/ADRs/platform/coordinate_registry.toml), v1.3.0) + ([`coordinate_registry.toml`](https://github.com/views-platform/views-appwrite/blob/20dfd0f/docs/ADRs/platform/coordinate_registry.toml), v1.4.0) as the canonical source of the non-secret coordinates this repo's runtime resolves. This repo's declared environment (`unfao/appwrite_env.py`, fail-loud entry validation) follows that registry's names. @@ -705,10 +705,21 @@ a "simplification." a pinned commit, which is what "referenced by URL, never by copy" was always supposed to mean — an unpinned `main` link drifts silently under the reader. + **Re-pinned again 2026-08-03 — the drift detector's first real firing.** The registry + moved to **v1.4.0** on 2026-08-02 (four CRAFD coordinates graduated from reserved to + real, and the CRAFD caller key was recorded as issued). `test_env_declaration.py` + failed on the next run, naming the version and telling the reader what to do, which is + what S6 (#187) built it for — its first firing was a true positive, not a false alarm. + Re-verified before bumping rather than after: all **13** names this repo declares are + present at v1.4.0 with unchanged class, and nothing was added or removed registry-wide, + so the registry's own claim that the bump *"imposes no new obligation on any consumer"* + holds here mechanically and not merely by assertion. Both links now pin `20dfd0f`. + **Re-pinned same day (#196).** The first pin taken here was resolved from a local views-appwrite checkout's `HEAD` — which was sitting on an unmerged branch. That commit declared registry v1.4.0, never reached `main`, and has been withdrawn. Both - links now pin `47172af`, the tip of `main`, carrying the **ratified v1.3.0**. The + links were then pinned to `47172af`, the tip of `main` at that time, carrying the + ratified v1.3.0 (superseded 2026-08-03 — see the erratum above). The lesson is narrow and worth keeping: **resolve a cross-repo pin from the tip of the other repo's `main`, not from whatever its working copy has checked out.** @@ -889,9 +900,10 @@ record execution progress against it. checkout's `HEAD` while it sat on the unmerged `feat/s1-single-writer-rule` branch. That commit declared registry v1.4.0 and has been **withdrawn unmerged** (views-appwrite #30, #27); its sha is deliberately not repeated here, so that grepping for it finds nothing. - Both links now pin **`47172af`** — the tip of `main`, registry **v1.3.0**, ratified at - þing-02. Verified before re-pinning: the withdrawn commit is reachable only from that - branch, `47172af` is `origin/main`, and both cited files exist at it. **Resolve a + Both links were then pinned to **`47172af`** — the tip of `main` at that time, registry + **v1.3.0**, ratified at þing-02 (superseded 2026-08-03 by v1.4.0 @ `20dfd0f`, when S6's + drift check fired). Verified before re-pinning: the withdrawn commit was reachable only + from that branch, `47172af` was `origin/main`, and both cited files existed at it. **Resolve a cross-repo pin from the other repo's `main`, never from its working copy's `HEAD`.** - **2026-07-15 — §4.1a's recorded inconsistency CONFIRMED live.** During the Hop-B legacy-guard work (faoapi PR #200), a read-only audit of the live `unfao_bucket` diff --git a/views_postprocessing/unfao/appwrite_env.py b/views_postprocessing/unfao/appwrite_env.py index 934f372..b107e1f 100644 --- a/views_postprocessing/unfao/appwrite_env.py +++ b/views_postprocessing/unfao/appwrite_env.py @@ -5,9 +5,9 @@ views-appwrite and referenced by pinned URL, never copied — copies were the platform's original failure (þing-01 sáttmál S6): - https://github.com/views-platform/views-appwrite/blob/47172af/docs/ADRs/platform/coordinate_registry.toml + https://github.com/views-platform/views-appwrite/blob/20dfd0f/docs/ADRs/platform/coordinate_registry.toml -That pin is registry **v1.3.0** (ratified, þing-02) — declared below as +That pin is registry **v1.4.0** — declared below as ``SEAM_CONTRACT_VERSION`` / ``SEAM_CONTRACT_COMMIT`` so the pin is a value a test can check rather than a fact buried in prose. A pinned URL does not rot, but it does go stale, and nothing in this repository could previously tell you it had (register C-57). @@ -44,8 +44,8 @@ #: Bumping these is not bookkeeping: it asserts that someone re-checked this module's #: declaration against that edition of the registry. ``tests/test_env_declaration.py`` #: enforces the pair against a local views-appwrite checkout when one is present. -SEAM_CONTRACT_VERSION = "1.3.0" -SEAM_CONTRACT_COMMIT = "47172af" +SEAM_CONTRACT_VERSION = "1.4.0" +SEAM_CONTRACT_COMMIT = "20dfd0f" CONNECTION_ENV = ( "APPWRITE_ENDPOINT", From 19dae3f06e8b0995459f628992f34a6fc4271cb3 Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Mon, 3 Aug 2026 03:35:08 +0200 Subject: [PATCH 19/23] =?UTF-8?q?fix(seam):=20re-pin=20to=20registry=20v1.?= =?UTF-8?q?4.1=20=E2=80=94=20second=20drift=20in=20one=20day=20(#213)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Same procedure as #212, same outcome: our 13 declared names are unchanged in class and membership at v1.4.1, verified rather than assumed before the pair was bumped. The upstream change is a register reframing that imposes nothing on consumers. Noted in ADR-013 rather than acted on: the registry moved twice today, and a pin chased per-PR is a real cost. If views-appwrite stays this active, the check may want a MAJOR/MINOR floor rather than an exact match. That is a decision, not a fix, so it is recorded where the next person re-pinning will read it. Co-authored-by: Claude Opus 5 (1M context) --- docs/ADRs/013_sampled_forecast_wire_contract.md | 13 ++++++++++--- views_postprocessing/unfao/appwrite_env.py | 8 ++++---- 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/docs/ADRs/013_sampled_forecast_wire_contract.md b/docs/ADRs/013_sampled_forecast_wire_contract.md index 62cdcfb..3f8e57c 100644 --- a/docs/ADRs/013_sampled_forecast_wire_contract.md +++ b/docs/ADRs/013_sampled_forecast_wire_contract.md @@ -689,10 +689,10 @@ a "simplification." this contract governs the **data** crossing the Appwrite seam; the **identity/config** crossing the same seam (which key, which coordinates, sourced from where, named how) is governed by - **[the Appwrite Seam Contract](https://github.com/views-platform/views-appwrite/blob/20dfd0f/docs/ADRs/platform/appwrite_seam_contract.md)**, + **[the Appwrite Seam Contract](https://github.com/views-platform/views-appwrite/blob/90fc105/docs/ADRs/platform/appwrite_seam_contract.md)**, homed in views-appwrite and referenced here **by pinned URL, never by copy** — with its coordinate registry - ([`coordinate_registry.toml`](https://github.com/views-platform/views-appwrite/blob/20dfd0f/docs/ADRs/platform/coordinate_registry.toml), v1.4.0) + ([`coordinate_registry.toml`](https://github.com/views-platform/views-appwrite/blob/90fc105/docs/ADRs/platform/coordinate_registry.toml), v1.4.1) as the canonical source of the non-secret coordinates this repo's runtime resolves. This repo's declared environment (`unfao/appwrite_env.py`, fail-loud entry validation) follows that registry's names. @@ -705,7 +705,14 @@ a "simplification." a pinned commit, which is what "referenced by URL, never by copy" was always supposed to mean — an unpinned `main` link drifts silently under the reader. - **Re-pinned again 2026-08-03 — the drift detector's first real firing.** The registry + **Re-pinned a third time, hours later, to v1.4.1 @ `90fc105`.** The registry moved + twice in one day (v1.4.1 adds a register reframing upstream; our 13 names were + re-verified unchanged in class and membership both times). The detector is doing its + job, but a pin that must be chased per-PR is a cost worth naming: if views-appwrite + settles into a slower cadence this is fine, and if it does not, the check may want to + compare a MAJOR/MINOR floor rather than an exact version. Recorded, not acted on. + + **Re-pinned 2026-08-03 — the drift detector's first real firing.** The registry moved to **v1.4.0** on 2026-08-02 (four CRAFD coordinates graduated from reserved to real, and the CRAFD caller key was recorded as issued). `test_env_declaration.py` failed on the next run, naming the version and telling the reader what to do, which is diff --git a/views_postprocessing/unfao/appwrite_env.py b/views_postprocessing/unfao/appwrite_env.py index b107e1f..1035b64 100644 --- a/views_postprocessing/unfao/appwrite_env.py +++ b/views_postprocessing/unfao/appwrite_env.py @@ -5,9 +5,9 @@ views-appwrite and referenced by pinned URL, never copied — copies were the platform's original failure (þing-01 sáttmál S6): - https://github.com/views-platform/views-appwrite/blob/20dfd0f/docs/ADRs/platform/coordinate_registry.toml + https://github.com/views-platform/views-appwrite/blob/90fc105/docs/ADRs/platform/coordinate_registry.toml -That pin is registry **v1.4.0** — declared below as +That pin is registry **v1.4.1** — declared below as ``SEAM_CONTRACT_VERSION`` / ``SEAM_CONTRACT_COMMIT`` so the pin is a value a test can check rather than a fact buried in prose. A pinned URL does not rot, but it does go stale, and nothing in this repository could previously tell you it had (register C-57). @@ -44,8 +44,8 @@ #: Bumping these is not bookkeeping: it asserts that someone re-checked this module's #: declaration against that edition of the registry. ``tests/test_env_declaration.py`` #: enforces the pair against a local views-appwrite checkout when one is present. -SEAM_CONTRACT_VERSION = "1.4.0" -SEAM_CONTRACT_COMMIT = "20dfd0f" +SEAM_CONTRACT_VERSION = "1.4.1" +SEAM_CONTRACT_COMMIT = "90fc105" CONNECTION_ENV = ( "APPWRITE_ENDPOINT", From 94977bd7282b1a9534d8ee3ec8aa1a4bc2e44b67 Mon Sep 17 00:00:00 2001 From: "Simon Polichinel von der Maase, Ph.D" Date: Mon, 3 Aug 2026 03:37:39 +0200 Subject: [PATCH 20/23] =?UTF-8?q?feat(enrichment):=20S4=20(#89)=20?= =?UTF-8?q?=E2=80=94=20numpy/pyarrow=20keyed=20gather,=20drop=20pandas=20m?= =?UTF-8?q?erge=20(#210)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(enrichment): S4 (#89) — numpy/pyarrow keyed gather, drop the pandas merge Rewrite GaulLookupEnricher's lookup side to be pandas-free (epic #85): - Read the 888 KB GAUL lookup with pyarrow; hold it as numpy arrays + a sorted key index. Attaching metadata is a keyed `searchsorted` gather, not a pandas merge — absent gids gather to None so the downstream null gate still fails loud. The artifact is no longer materialised as a pandas frame, and the join no longer needs it to carry pandas index metadata (which unblocks S5/#90's pyarrow-native builder). - pandas becomes a TYPE_CHECKING-only import (it's in the enricher's interface — callers hand it DataFrames — not its implementation). - test_enrichment: the `lookup` fixture loads the artifact directly (the enricher no longer exposes a private pandas frame), tolerant of both the current pandas-index shape and S5's index-free one. - test_doc_accuracy: the pandas-importer guard now checks by AST, asserting ZERO runtime importers + type-only in enrichment.py (a regex couldn't tell a TYPE_CHECKING guard from a real import). Recovered from an uncommitted working tree on this (correctly-named) branch — complete, its 31 tests pass. (The repo's 5 wire shard/sidecar byte-parity + toolchain-pin failures are pre-existing and unrelated to this change.) Co-Authored-By: Claude Opus 4.8 * fix(enrichment): S4 review findings — refuse an empty lookup, and say what the dtype change costs Four findings from the diff review, applied to the commit that preceded them. An empty lookup raised IndexError from inside the gather. The guard read `(len(self._keys) > 0) & (self._keys[clipped] == wanted)`, which LOOKS like a guard and is not one: `&` evaluates both operands, so the index happened anyway. Guard where the condition is knowable — __init__ now refuses an empty lookup with a message about the lookup, rather than letting the delivery later complain about missing metadata. The parity proof passed check_dtype=False, check_categorical=False and compared after reset_index — three flags that hide exactly what a merge-to-assign rewrite is most likely to change. Re-checked without them. Two results worth having: - name columns went category -> object. Measured, not guessed: on a 200-row output, category is 1,795,191 bytes and object is 60,061, because a categorical carries the artifact's full 64,742-entry dictionary whatever the output size. Lighter for the frames this object actually sees. Recorded in the code beside the C-32 note it might otherwise be read as contradicting. - on EMPTY input the old merge replaced the input's RangeIndex with an object-dtype Index; the gather leaves it alone. The only behavioural difference found, and it is in the direction the CIC already claimed. The CIC is corrected — fourth story running where it lagged the change, which is the pattern ADR-014 §5 was written for, recurring in the first story after the ADR landed. It now states the gather, the dtype change, the empty-lookup refusal, and that pandas is interface-only. Also dropped a dead alias and a duplicated row count. Co-Authored-By: Claude Opus 5 (1M context) * fix(enrichment): S4 review round 1 — a declared conversion, and the guards the claims needed Six findings from /code-review max (five parallel reviewers), all reproduced before fixing. The blind `np.asarray(gids, dtype=np.int64)` was wrong in three ways, one of them a regression against development: - a NaN gid became INT64_MIN with only a RuntimeWarning, and then the warning line itself raised `ValueError: cannot convert float NaN to integer` because it iterated the PRE-cast array. The path whose job is to report the problem gracefully was the path that crashed. A missing gid is ordinary data quality. - a non-integral float truncated SILENTLY: 54220.000000001 -> 54220, matching a real, different cell. Verified against the pandas merge this replaced — it returns null there, and even warns. A silent wrong match is worse than a crash. - pd.NA raised a bare TypeError with no log and no contract-shaped error. Replaced by `_as_cell_ids`, which DECLARES what a cell id is and marks anything else unusable rather than coercing it. Unusable ids gather to null exactly as unknown gids do, so the downstream gate sees a hole rather than a guess. Verified case by case against the old implementation: NaN, drifted float and object-with-None now match it exactly, and pd.NA is strictly better (old crashed). The lookup side had the same hole: a null key would take the same sentinel, collide with an unusable query id, and report FOUND — handing back another cell's metadata. That is the fabricated value this module's docstring forbids. __init__ now refuses it. The warning then named cell 0 for an unusable id — a cell nobody asked about. It now names unknown cells and separately counts rows that carried no usable id, because an id that is not an id has none to report. The AST guard added in this same PR had the defect it was written to catch. It walked the whole `if TYPE_CHECKING` node including `else:`, and substring-matched the test, so `not TYPE_CHECKING` read as a guard. Both evasions reproduced, both now caught, and pinned by a test rather than a comment — the `&`-not-short-circuiting shape, committed twice in one pull request. The CIC's guarantees were prose resting on a throwaway script. Committed: empty and null-key refusals, the four unusable-id behaviours, row/order/index preservation across five shapes plus a non-default index, and the object dtype. 29 tests. ADR-012 updated — my own test docstring said "ADR-012 names the seam, so update it rather than letting the claim drift", and the first draft did not. Three claims in the CIC that PR #200 retired in the manager's CIC were still standing here. Registered C-75: this class has no production caller and now holds a second copy of the delivery path's keyed gather. Tier 3 — the verification path and the delivery path implement the same algorithm twice, and the tests run through the copy that does not ship. Co-Authored-By: Claude Opus 5 (1M context) * fix(enrichment): S4 review round 2 — bound the conversion, stop parsing, prove the mask Round 1 removed one silent coercion and introduced two more. Round 2 found them. The object branch used `float(value)`, which PARSES. It accepted the string "54220" as a cell id — where the pandas merge this replaces raised "You are trying to merge on object and int64 column". A string gid column is a declaration error and the old path said so; parsing it is inference, and it is the drifted-float defect pointing the other way. It now accepts only values that already ARE integers. `bool` is excluded for the same reason: True is not cell 1. Neither branch bounded int64. `uint64` max wrapped to -1 and `1e30` to INT64_MIN, both flagged VALID — an id nobody wrote, marked usable. Harmless only because real gids are 1..259,200, which is luck of the key range rather than a property the code enforced. Both branches now bound explicitly. `& usable` in `_gather` was the one new guard that survived its own removal: unusable ids substitute to 0, and 0 is not a real gid, so the key comparison already failed. It looked like decoration. It is not, and the case is now committed: with a lookup containing gid 0, a NaN row silently receives cell 0's geography without the mask. That is the fabricated value the docstring forbids, produced from an input the caller never wrote. The out-of-range test I wrote in round 2 built every case with dtype=object, so all of them entered the element-wise branch and the int and float bounds it claimed to cover were never executed — removing them left the suite green. Each case now enters the branch it tests. Mutation matrix after: dropping `& usable`, the float bound, the int bound, or the isinstance check each fails the suite. `np.isfinite` is the exception and is now documented as such rather than left to be rediscovered: inf fails the upper bound, NaN fails `arr == rint(arr)`, so the check is subsumed and defensive, not load-bearing. Saying so is cheaper than the next person mutation-testing it and wondering. 37 tests in test_enrichment.py; 337 in the suite. Wire untouched. Co-Authored-By: Claude Opus 5 (1M context) * fix(enrichment): S4 review round 3 — the float bound was off by one ULP, and my mutation proof was a replica Second /code-review max pass. Three correctness defects and a vacuous test, all reproduced before fixing. The float upper bound admitted exactly what it was added to exclude. _INT64_MAX is 2**63-1, which is odd and NOT representable in float64, so comparing a float against it promotes and rounds UP to 2**63. float(2**63) passed isfinite, integral and both bounds, then wrapped to INT64_MIN and was flagged valid. Now bounded strictly below 2.0**63, which IS exact. _INT64_MIN needed no change: -2**63 is a power of two. timedelta64 was accepted as a cell id, because isinstance(np.timedelta64(5,'ns'), np.integer) is True — a numpy quirk, and datetime64 is not, so the hole was specific to one type. A duration is not a cell id any more than True is cell 1. The test I wrote to prove the AST guard bites was itself a replica. It defined a local copy of the classifier and asserted against that; reintroducing the polarity bug into the REAL guard left all 16 tests green. That is the defect S4 of epic #181 retired from test_validation.py two days ago — 43 tests against a function the file defined itself — committed again by me, in the PR whose message boasts about catching its cousin. The classifier is now one module-level function with two callers, and both regressions now fail it. And the fix for that had a sting: routing the out-of-range cases through the public API, as the review recommended, LOST the proof. Publicly, "refused" and "accepted but wrapped to an absent id" both produce null, so reverting the float bound left the test green. The public assertion is the caller's guarantee; the mask assertion is the mutation proof; both are needed and both are now present, with a comment saying why. _gather returned four values, three of which existed only to build one log message. The warning moved inside it, matching this file's own convention where __init__ co-locates detection and logging at every guard. It returns one dict. My own error message credited the builder with refusing null keys. It does not: its null check runs after the key becomes the index, and DataFrame.isna() does not inspect an index. A null key is unreachable there only because an earlier astype('int64') raises — protection by accident. Registered as C-76, along with the empty-output case, which IS reachable and writes cells=0 without complaint. Also: the pandas error message was misquoted, ADR-012 cited a path that does not exist (unfao/wire/ -> contract/wire/), the CIC claimed eight input shapes where the pinned class has six and named one it does not exercise, the artifact size was wrong by 2% in four places, and a register quote was singular where C-40 is plural. C-75 gained the owner ADR-014 §4 requires and its first draft omitted while citing §4. Co-Authored-By: Claude Opus 5 (1M context) --------- Co-authored-by: Claude Opus 4.8 --- docs/ADRs/012_revised_ontology.md | 6 +- docs/CICs/GaulLookupEnricher.md | 63 ++++- reports/technical_risk_register.md | 50 +++- tests/test_doc_accuracy.py | 115 +++++++- tests/test_enrichment.py | 264 ++++++++++++++++++- views_postprocessing/contract/enrichment.py | 238 +++++++++++++++-- views_postprocessing/contract/gaul_lookup.py | 2 +- 7 files changed, 687 insertions(+), 51 deletions(-) diff --git a/docs/ADRs/012_revised_ontology.md b/docs/ADRs/012_revised_ontology.md index bf9fa32..79dcc3e 100644 --- a/docs/ADRs/012_revised_ontology.md +++ b/docs/ADRs/012_revised_ontology.md @@ -47,7 +47,7 @@ unfao/ who a delivery is FOR — one partner's product and manager | **Delivery Invariants** | Representation-free rules over primitives that a delivery must satisfy: coverage, no-collapse, gid parity, observed-range, provenance. Live in `delivery/` — **nothing there imports pandas or views_frames**. *Forecast identity was one of these until 2026-07-31 — see the amendment below.* | Authoritative — they define what a valid delivery is | Stable — changes are governance decisions | | **Representation Seam** | `contract/frame_extraction.py` — turns a `views_frames` frame into the primitives the invariants consume. **One seam.** Its pandas sibling `unfao/extraction.py` was deleted in #151 once the pandas delivery was retired; the two ran as deliberate WET siblings through the migration. | Derived — isolates the representation so invariants stay representation-free | Evolving | | **Wire Mechanism** | `contract/wire/` — the ADR-013 contract: header, shard, sidecar, run manifest, sink, source selection. Partner-neutral: it takes its consumer name and collapse floor as **arguments** (#153). | Authoritative — the contract with the consumer | Stable — changes are contract amendments | -| **Enrichment Asset** | The precomputed GAUL lookup (`data/gaul_lookup.parquet`), its identity in `contract/gaul_lookup.py`, its schema in `contract/gaul_schema.py`, and the pandas merge that joins it (`contract/enrichment.py`, the build/verification path). | Authoritative for geographic metadata | Stable — rebuilt only when the producer releases new GAUL data | +| **Enrichment Asset** | The precomputed GAUL lookup (`data/gaul_lookup.parquet`), its identity in `contract/gaul_lookup.py`, its schema in `contract/gaul_schema.py`, and the keyed gather that joins it (`contract/enrichment.py`, the build/verification path — numpy/pyarrow since #89; see register **C-75** on whether that class should survive at all). | Authoritative for geographic metadata | Stable — rebuilt only when the producer releases new GAUL data | | **Artifact Builders** | `contract/historical.py` — turns a frame plus the lookup into the partner-facing artifact. | Derived | Evolving | | **External Facts** | Facts read from systems this repo does not own: the producer's (`contract/source_metadata.py` — `last_valid_month_id`, D-07) and the store's (`contract/store_metadata.py`). | Authoritative (the owning system is the source of truth) | Evolving | | **Launch Declarations** | `contract/launch_config.py` — the delivery mode the launcher must declare. Omitting a key is **refused by name**, never inferred (ADR-003, register C-63). | Authoritative | Stable | @@ -60,7 +60,7 @@ unfao/ who a delivery is FOR — one partner's product and manager 636 lines holding two of everything, and it called `unfao/extraction.py` *"the **single** pandas-aware module"* when pandas lived in three. Both drifted the same way: the ADR described the intended end state of a migration that then stopped one step short. Both are now true — -pandas has exactly one importer again (`contract/enrichment.py`) — and the load-bearing ones are +pandas has **zero runtime importers** (#89 made `contract/enrichment.py`'s a type-only import under `if TYPE_CHECKING` — pandas is in that class's interface, not its implementation) — and the load-bearing ones are **mechanically checked** by `tests/test_doc_accuracy.py`, so the next drift fails CI instead of waiting for an audit. @@ -117,7 +117,7 @@ looking for deleted code. **The rule is not weaker — it is enforced from better evidence.** The retired `assert_forecast_identity` compared one selected store document's `name`/`loa` against the configured ensemble. Since the ADR-013 contract path became the only path (#149), the same -guarantee is enforced in `unfao/wire/source_selection.py:73-81`: `TargetLease.load()` +guarantee is enforced in `contract/wire/source_selection.py:73-81`: `TargetLease.load()` checks **every shard header's declared `provenance.ensemble`** against the launched ensemble, and refuses the run on a mismatch. Identity now comes from the artifact's own declared content rather than from a metadata field on a single document, and it is checked diff --git a/docs/CICs/GaulLookupEnricher.md b/docs/CICs/GaulLookupEnricher.md index 4d0938e..a49cfc5 100644 --- a/docs/CICs/GaulLookupEnricher.md +++ b/docs/CICs/GaulLookupEnricher.md @@ -25,23 +25,45 @@ views-datafactory area-majority join), so this class does only a table join. - This class does **not** build the lookup table (that is `scripts/build_gaul_lookup.py`, run offline). - This class does **not** fill, impute, or invent metadata for unmatched cells. -- This class does **not** validate the result — null/coverage validation is the - manager's `_validate()` responsibility. +- This class does **not** validate the result. Null/coverage enforcement lives on the + delivery path — `contract/historical.assert_metadata_complete` at artifact build and + `delivery/coverage.py` for the region contract. It is **not** the manager's + `_validate()`, which stopped null-gating in #149 and now asserts only that the read + resolved. (Corrected 2026-08-02; PR #200 retired the same claim in the manager's CIC + and this one was left standing.) - This class does **not** read from the datafactory, viewser, or Appwrite. --- ## 3. Responsibilities and Guarantees -- Loads exactly one lookup Parquet at construction and verifies it carries the 9 - contract columns; missing columns raise at construction. +- Loads exactly one lookup Parquet at construction and verifies it carries the key + plus the 9 contract columns; missing columns raise at construction, and so does an + **empty** lookup — every cell would gather to null and the delivery would then + complain about missing metadata rather than about a missing lookup (S4 / #89). - Returns the input frame augmented with exactly the 9 columns of - `gaul_schema.METADATA_COLS`, with their dtypes preserved from the lookup - (codes numeric, coordinates float, names/iso categorical). + `gaul_schema.METADATA_COLS`: codes numeric, coordinates float, **names and iso as + `object`**. + *Changed in S4 (#89).* They were `category`, inherited from the pandas merge that + read the artifact's dictionary encoding. The gather that replaced it assigns plain + values. Measured on a 200-row output: category 1,795,191 bytes, object 60,061 — + a categorical carries the artifact's full 64,742-entry dictionary whatever the + output size. The builder still writes the artifact with dictionary-encoded names + (`# names/iso categorical (C-32 memory)`); that governs the file, not this output. - A cell id present in the lookup is enriched with that cell's metadata. - A cell id **absent** from the lookup yields **null** metadata for that row — never a sentinel, never a fabricated value (fail-loud downstream). -- Row count and row order of the input are preserved (left merge). +- Row count, row order **and index** of the input are preserved. Pinned by + `tests/test_enrichment.py::TestFramePropertiesPreserved` across six shapes — + ordered, reversed, duplicated, single, empty, and a non-default index. + (An earlier draft of this line claimed *eight* shapes "including unknown gids", + counting a throwaway development script rather than the committed suite, and naming + a shape that class does not exercise. Unknown gids are covered, for null-value + correctness, by `TestFailLoud` — a different guarantee.) + On **empty** input this is now *more* true than before: the pandas merge replaced the + input's `RangeIndex` with an object-dtype `Index`, where the gather leaves it + untouched. The only behavioural difference found, and it is in the direction the + guarantee above already claimed. --- @@ -60,7 +82,12 @@ views-datafactory area-majority join), so this class does only a table join. ## 5. Outputs and Side Effects - Output: the input frame (or, with `only_metadata=True`, just `pg_id_col` + - `time_id_col`) left-merged with the 9 metadata columns. + `time_id_col`) with the 9 metadata columns attached by **keyed gather**. + *Changed in S4 (#89):* this was a pandas left-merge on the lookup's index. The + lookup is now read with pyarrow, sorted once, and addressed by `np.searchsorted`. + Attaching metadata to a frame is a keyed gather, not frame algebra, and it never + needed a merge — which is also what frees the builder to stop writing pandas index + metadata (S5 / #90). - Public attribute: `lookup_version` — a short, stampable id read from the lookup's **declared** `lookup_version` metadata key at construction (`@<8-char source digest>`). Delegates to @@ -84,16 +111,21 @@ views-datafactory area-majority join), so this class does only a table join. column. - **Raises** `ValueError` if `pg_id_col` is not in the input. - **Does not raise** on unmatched cells — it surfaces them as nulls and logs a - WARNING. This is deliberate: the manager's `_validate()` null gate is the - single enforcement point, so a coverage hole fails loudly there (one place), - not in two. Passing a sentinel for unmatched cells would be a **bug** (it would + WARNING naming the unknown cells, and separately counting rows that carried no + usable cell id at all. This is deliberate: enforcement is a single point on the + delivery path (`historical.assert_metadata_complete`), so a coverage hole fails + loudly there, not in two places. Passing a sentinel for unmatched cells would be a **bug** (it would bypass that gate). Aligns with ADR-003 (fail loud on semantic ambiguity). --- ## 7. Boundaries and Interactions -- Allowed to depend on: pandas, `gaul_schema`, and a local Parquet file. +- Allowed to depend on: numpy, pyarrow, `gaul_schema`, and a local Parquet file. + **pandas is interface-only** — callers hand this class DataFrames and get one back, + but nothing here constructs, reads or joins one, and its import is under + `if TYPE_CHECKING` (S4 / #89). `tests/test_doc_accuracy.py` asserts by AST that the + package has **zero** runtime pandas importers. - Must **not** depend on: geopandas/shapely, the runtime mapper, the datafactory, viewser, Appwrite, or any network resource. - Treats the lookup table as an opaque, trusted artifact produced by the build @@ -112,7 +144,12 @@ out = enricher.enrich_dataframe_with_pg_info( # out has the 9 metadata columns; unmatched cells are null. ``` -Drop-in for the manager's existing call (same method name and key kwargs). +Signature-compatible with the runtime mapper it replaced (same method name and key +kwargs), which is why the mapper-only kwargs are still accepted and ignored. +**The manager does not call this class** — it has no `enrich` reference at all, and +reads the lookup directly via `gaul_lookup.load()` (register C-66). This example is a +build/verification-path usage. Whether the class should survive that is register +**C-75**. --- diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 7230a3f..0b0a8e0 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -5,8 +5,8 @@ | Project | views-postprocessing | | Owner | Dylan Pinheiro / PRIO MD&D Team | | Last Updated | 2026-08-02 | -| Total Concerns | 74 | -| Open Concerns | 15 | +| Total Concerns | 76 | +| Open Concerns | 17 | | Resolved Concerns | 59 | --- @@ -532,6 +532,52 @@ Cross-refs: **C-25** (whose resolution this corrects), **C-40** (the inherited p --- +### C-75: `GaulLookupEnricher` has no production caller, and now implements a second copy of the delivery path's keyed gather + +| Field | Value | +|-------|-------| +| ID | C-75 | +| Tier | 3 — no correctness impact today: the class is off the delivery path, so a defect in it cannot reach the UN FAO. The cost is that **the verification path and the delivery path now implement the same algorithm twice**, and the tests that check the artifact run through the copy that does *not* ship. A fix applied to one and not the other makes the verification stop verifying what ships — quietly, because both would still pass their own tests. | +| Source | `code-review max` (2026-08-02) — PR #210, five parallel reviewers; two reached this independently | +| Trigger | When a bug is fixed in `contract/historical.py`'s gather (the one that ships), check whether `contract/enrichment.py`'s copy needs the same fix — nothing links them. Also fires at **S5 (#90)**: once the builder is pyarrow-native, the enricher's pandas interface is the last one in the package, and the question "does this class survive?" has to be answered rather than deferred again. | +| Owner | Whoever takes **#90** — the keep-or-retire decision is theirs to make and record, not to defer a third time. Added 2026-08-03: the first draft of this entry named two triggers and no owner, while citing ADR-014 §4 in its own body. This register had already learned that twice — *"a deferral needs an owner and a trigger, not just a reason"* (Cluster L) and *"a decision awaiting an owner, not a task awaiting effort"* (epic #181 closeout). | +| Location | `views_postprocessing/contract/enrichment.py` (the whole class; `_gather` specifically); the shipping twin is `views_postprocessing/contract/historical.py:54-68` | + +**Verified, not inferred (2026-08-02):** `grep -rn "GaulLookupEnricher\|enrich_dataframe_with_pg_info"` across the package finds **zero** production callers — the two hits are docstring mentions in `gaul_lookup.py`. The manager calls `gaul_lookup.load()` directly and has zero `enrich` references. **C-66**'s resolution already said this plainly: *"the pandas enricher leaves the delivery path entirely."* + +**What PR #210 did, and why that raises the question.** S4 (#89) rewrote this class's lookup side from a pandas merge to a numpy/pyarrow keyed gather: a measured dtype analysis, an empty-lookup guard, a mutation-proven bug fix, a corrected CIC, and five reviewers' attention. All of it spent on a method with no reachable caller outside its own test suite. The engineering is sound; what is missing is anyone having **decided** that the class should exist. + +**The duplication is the concrete consequence.** `_gather`'s `argsort → searchsorted → clip → equality-mask` is the same shape as `historical.py:54-68`. The policies differ deliberately — `historical` **raises** on an absent gid (*"geography must never silently vanish"*), the enricher returns nulls for the downstream gate to catch — so extracting a shared helper would mean parameterising the failure policy, which is the guessed abstraction **WET before DRY** exists to prevent. Two copies that are understood is the right call *today*. The trigger above is what stops "today" lasting indefinitely, per **ADR-014 §4**. + +**The precedent is C-45**, `unfao/frames.py`: an unused adapter carried on no live path, resolved by deleting it. This is the same shape with a different module, and the same question — keep it as the declared verification/reference implementation, or retire it and let the fidelity suite test `historical.py` directly. + +**Deliberately NOT registered from the same review** (defects in unmerged code, all fixed in #210 before merge rather than tracked): a NaN gid crashing the warning path, the unvalidated int64 coercion at both ends, the AST guard's `else`-branch blind spot, ADR-012's stale pandas-merge claim, and three CIC claims retired elsewhere by #200. The register tracks standing risk; a defect fixed before it ships is not one. They are recorded in the PR. + +Cross-refs: **C-45** (RESOLVED — the same shape, resolved by deletion), **C-66** (RESOLVED — established the enricher left the delivery path), **C-40** (which calls `enrichment.py` and `extraction.py` together *"the retired-in-place `enrichment.py`/`extraction.py` legacy seams"*), **#89** / **#90** / epic **#85**, ADR-014 §4. + +--- + +### C-76: `build_gaul_lookup.py` will write an empty lookup without complaint + +| Field | Value | +|-------|-------| +| ID | C-76 | +| Tier | 4 — no silent corruption. A zero-row artifact fails downstream at `historical.build_historical_table`, which raises on cells absent from the lookup. The cost is that it fails **late and confusingly**: the message names missing geography rather than an empty lookup, and the artifact is committed by then. | +| Source | `code-review max` (2026-08-03) — PR #210 second pass, while checking whether the consumer's new guards duplicated a producer guarantee. They do not. | +| Trigger | When `build_gaul_lookup.py` is next run with a new or renamed `--region`, or against a datafactory whose `gaul_admin` parquets have changed shape — check the printed `cells=` count is non-zero before committing the artifact. Nothing else will tell you. | +| Owner | Whoever next runs the builder. It is a two-line guard in a script one person runs by hand, not a scheduling decision. | +| Location | `scripts/build_gaul_lookup.py` — the invariant block at `:246-268` and the write at `:284` | + +The builder's invariant block is thorough about what it checks: index uniqueness (C-59), nulls in the metadata columns, `-1` sentinels in the code columns (C-35). It does not check that any rows survived. A `--region` argument that filters every cell out, or an upstream join that produces nothing, writes a zero-row parquet and prints `cells=0` as though that were a result. + +**Verified 2026-08-03, and the neighbouring worry is NOT real.** The same review asked whether the builder also fails to reject a null key, since `df.isna().sum().sum()` runs *after* `priogrid_gid` becomes the index and `DataFrame.isna()` does not inspect the index. It does not check it — but the null key is unreachable anyway: `df.index.astype("int64")` raises `IntCastingNaNError` two lines earlier. Protection by accident rather than by declaration, which is worth knowing, but not a defect to fix. **Only the empty case is reachable.** + +**Why this was found now.** PR #210 added consumer-side refusals for both an empty lookup and a null key to `GaulLookupEnricher.__init__`, and the review challenged them as duplicating a producer guarantee. Checking established the opposite: for the empty case there is no producer guarantee to duplicate, and for the null key the producer's protection is incidental. The consumer guards stay, and this entry records the producer-side half rather than quietly assuming someone will notice. + +Cross-refs: **C-59** and **C-61** (RESOLVED — the invariant block this sits beside, and the reason it is otherwise thorough), **C-35** (the `-1` defect class it does check for), **C-75** (the consumer whose guards prompted the check), #210. + +--- + ## Disagreements ### D-12: Post-Run-0 infrastructure & naming intents — repo rename, internal-store transport, compute co-location diff --git a/tests/test_doc_accuracy.py b/tests/test_doc_accuracy.py index 896ba0d..ea16dd2 100644 --- a/tests/test_doc_accuracy.py +++ b/tests/test_doc_accuracy.py @@ -16,6 +16,7 @@ from __future__ import annotations +import ast import re from pathlib import Path @@ -139,16 +140,80 @@ def test_internal_doc_links_resolve(): _MANAGER_LINE_BUDGET = 450 # epic #148's bound; 406 at close, 636 before #149 -def test_pandas_has_exactly_one_importer(): - """ADR-012's 'Representation Seam' row says pandas is isolated to one module.""" - importers = sorted( - f.relative_to(_PKG).as_posix() - for f in _PKG.rglob("*.py") - if re.search(r"^\s*(?:import pandas|from pandas\b)", f.read_text(), re.M) +def _is_type_checking(test: ast.expr) -> bool: + """`TYPE_CHECKING` or `typing.TYPE_CHECKING`, and nothing else. + + Substring-matching `ast.dump(test)` also matched `not TYPE_CHECKING`, which means + the opposite. Polarity is the whole point of a guard. + """ + if isinstance(test, ast.Name): + return test.id == "TYPE_CHECKING" + return isinstance(test, ast.Attribute) and test.attr == "TYPE_CHECKING" + + +def _classify_pandas_imports(source: str) -> list[tuple[str, ast.AST]]: + """``[("runtime" | "type-only", node)]`` for every pandas import in ``source``. + + **Module-level on purpose.** Both the guard and the test that proves the guard + bites call THIS function. An earlier draft had the meta-test define its own copy + of this logic — so it asserted against a replica, and when the polarity bug was + reintroduced into the real guard, all 16 tests still passed. That is the exact + defect S4 of epic #181 retired from ``test_validation.py`` (43 tests against a + function the file defined itself), committed again in the pull request whose + message boasts about catching its cousin. One implementation, two callers. + """ + tree = ast.parse(source) + # BODY only, never `orelse`. Walking the whole `If` node classified a real runtime + # `import pandas` sitting in the `else:` branch as type-only — the guard passed on + # the exact thing it exists to catch. Reproduced before fixing. + guarded = { + id(n) + for node in ast.walk(tree) + if isinstance(node, ast.If) and _is_type_checking(node.test) + for stmt in node.body + for n in ast.walk(stmt) + } + out = [] + for node in ast.walk(tree): + if not isinstance(node, (ast.Import, ast.ImportFrom)): + continue + names = [getattr(node, "module", None) or a.name for a in node.names] + if any((n or "").split(".")[0] == "pandas" for n in names): + out.append(("type-only" if id(node) in guarded else "runtime", node)) + return out + + +def test_pandas_is_not_imported_at_runtime_anywhere_in_the_package(): + """ADR-012's 'Representation Seam' row — now a stronger claim than it was. + + It used to be *one* pandas-aware module (``contract/enrichment.py``). After S4 + (#89) that module's lookup side is numpy + pyarrow and its pandas import is + **type-only**, under ``if TYPE_CHECKING``: pandas is in the enricher's interface + (callers hand it DataFrames) but no longer in its implementation. + + So the assertion is now *zero runtime importers*, and it is checked by **AST** + rather than by regex — a regex matching an indented ``import pandas`` cannot tell a + real import from one inside a ``TYPE_CHECKING`` guard, and would have passed + unchanged while the meaningful property changed underneath it. + """ + runtime, type_only = [], [] + for source in sorted(_PKG.rglob("*.py")): + for kind, node in _classify_pandas_imports(source.read_text()): + # The FILE is the claim; the line number is not. Pinning a line means any + # edit above it fails a test about imports, which is how a guard earns a + # reputation for crying wolf (ADR-014 §3). + where = source.relative_to(_PKG).as_posix() + (type_only if kind == "type-only" else runtime).append(where) + + assert not runtime, ( + f"pandas is imported at RUNTIME in the package: {runtime}. The delivery path " + "is numpy/pyarrow end to end (epic #85); a runtime pandas import re-materialises " + "the representation the migration removed. If it is genuinely needed, put it " + "behind `if TYPE_CHECKING` or say in ADR-012 why it is not." ) - assert importers == ["contract/enrichment.py"], ( - f"ADR-012 claims one pandas-aware module; found {importers}. Either the claim " - "or the code moved — fix whichever is wrong, do not leave the ADR lying." + assert type_only == ["contract/enrichment.py"], ( + f"the type-only pandas imports moved: {type_only}. Not necessarily wrong — but " + "ADR-012 names the seam, so update it rather than letting the claim drift." ) @@ -410,3 +475,35 @@ def test_no_partner_contact_details_are_published_in_this_repository(): f"partner contact addresses appear in this public repository: {offenders}. " "Keep them in the project materials; reference the decision, not the address." ) + + +def test_the_runtime_import_guard_catches_both_ways_of_evading_it(): + """A guard that cannot fail is decoration (ADR-014 §2) — and this one could not. + + Its first draft walked the whole ``if TYPE_CHECKING`` node and substring-matched + ``ast.dump(test)``. Two evasions passed it, both reproduced before the fix: + + - a real runtime ``import pandas`` in the ``else:`` branch (walked as if guarded); + - ``if not TYPE_CHECKING:`` (substring match ignores polarity, which is the entire + point of a guard). + + **And the first draft of THIS test could not have caught either**, because it + defined its own copy of the classifier and asserted against that. Reintroducing + the polarity bug into the real guard left all 16 tests green. It now calls + ``_classify_pandas_imports`` — the same function the guard calls — so a regression + there fails both. + """ + def kinds(src: str) -> list[str]: + return [kind for kind, _ in _classify_pandas_imports(src)] + + assert kinds( + "from typing import TYPE_CHECKING\nif TYPE_CHECKING:\n pass\nelse:\n import pandas\n" + ) == ["runtime"], "an import in the else: branch was classified type-only" + + assert kinds( + "from typing import TYPE_CHECKING\nif not TYPE_CHECKING:\n import pandas\n" + ) == ["runtime"], "`not TYPE_CHECKING` was treated as a type-checking guard" + + assert kinds( + "from typing import TYPE_CHECKING\nif TYPE_CHECKING:\n import pandas\n" + ) == ["type-only"], "a legitimate type-only import was flagged — the guard cries wolf" diff --git a/tests/test_enrichment.py b/tests/test_enrichment.py index 367aa36..ca1ea6c 100644 --- a/tests/test_enrichment.py +++ b/tests/test_enrichment.py @@ -6,8 +6,14 @@ """ import pandas as pd +import logging + +import numpy as np +import pyarrow as pa +import pyarrow.parquet as pq import pytest +from views_postprocessing.contract import gaul_lookup from views_postprocessing.contract.enrichment import ( GaulLookupEnricher, METADATA_COLS, @@ -28,8 +34,24 @@ def enricher(): @pytest.fixture(scope="module") -def lookup(enricher): - return enricher._lookup +def lookup(): + """The committed artifact as a pandas frame, built here rather than reached for. + + Until S4 (#89) this was ``enricher._lookup`` — the enricher's own private pandas + frame. It no longer has one: the lookup side is numpy + pyarrow, and pandas is a + type-only import there. These tests are about the **artifact**, not the enricher, + so they load it directly. + + The ``set_index`` is deliberate and forward-looking. ``to_pandas()`` currently + restores ``priogrid_gid`` as the index from the parquet's embedded pandas + metadata — metadata that **S5 (#90) removes** when the builder becomes + pyarrow-native. Handling both shapes means these tests do not have to change + again then. + """ + df = gaul_lookup.load().to_pandas() + if "priogrid_gid" in df.columns: + df = df.set_index("priogrid_gid") + return df class TestLookupIntegrity: @@ -157,3 +179,241 @@ def test_missing_pg_id_col_raises(self, enricher): enricher.enrich_dataframe_with_pg_info( pd.DataFrame({"x": [1]}), pg_id_col="priogrid_gid", ) + + +# ── the guarantees the CIC states, pinned (S4 / #89; ADR-014 §1) ───────────── +# +# The CIC claims row/order/index preservation "verified across eight input shapes" +# and names the output dtypes. That verification was a throwaway script run once +# during development — a guarantee resting on a claim rather than on a check, which +# is precisely what ADR-014 §1 forbids, written by the story after the ADR landed. +# These commit it. + + +class TestConstructionRefusals: + """Degenerate lookups must fail at construction, naming the lookup.""" + + def _write(self, path, gids): + cols = {"priogrid_gid": pa.array(gids, pa.int64())} + for c in METADATA_COLS: + cols[c] = pa.array( + [0.0] * len(gids) if c in ("pg_xcoord", "pg_ycoord") or c.endswith("_code") + else ["x"] * len(gids), + pa.float64() if c in ("pg_xcoord", "pg_ycoord") or c.endswith("_code") else pa.string(), + ) + t = pa.table(cols).replace_schema_metadata({b"lookup_version": b"t@00000000"}) + pq.write_table(t, path) + return path + + def test_an_empty_lookup_is_refused_at_construction(self, tmp_path): + """It used to raise IndexError from inside the gather instead — the guard + `(len(self._keys) > 0) & (...)` read as a guard and was not one, because `&` + evaluates both operands.""" + path = self._write(tmp_path / "empty.parquet", []) + with pytest.raises(ValueError, match="is empty"): + GaulLookupEnricher(path) + + def test_a_null_key_is_refused_at_construction(self, tmp_path): + """A null key would coerce to the same sentinel as an unusable query id, the + two would collide, and the row would be reported FOUND — receiving another + cell's metadata. That is the fabricated value this module forbids (cf. C-35).""" + cols = {"priogrid_gid": pa.array([1, None, 3], pa.int64())} + for c in METADATA_COLS: + numeric = c in ("pg_xcoord", "pg_ycoord") or c.endswith("_code") + cols[c] = pa.array([0.0] * 3 if numeric else ["x"] * 3, + pa.float64() if numeric else pa.string()) + path = tmp_path / "nullkey.parquet" + pq.write_table(pa.table(cols).replace_schema_metadata({b"lookup_version": b"t@0"}), path) + with pytest.raises(ValueError, match="null values"): + GaulLookupEnricher(path) + + +class TestUnusableCellIds: + """A value that is not a cell id gathers to null — never to a guessed cell. + + Each case was verified against the pandas merge this replaced; the behaviour + below is that merge's, not an invention. The drifted-float case is the one that + matters most: a blind `astype(np.int64)` truncates `54220.000000001` to `54220` + and matches a real, *different* cell, silently. + """ + + def _out(self, enricher, col): + return enricher.enrich_dataframe_with_pg_info( + pd.DataFrame({"priogrid_gid": col, "month_id": [1] * len(col)}), + pg_id_col="priogrid_gid", time_id_col="month_id", + ) + + def test_a_missing_gid_yields_null_and_does_not_crash(self, enricher, lookup): + good = lookup.index[:2].tolist() + out = self._out(enricher, [float(good[0]), float("nan"), float(good[1])]) + assert len(out) == 3 + assert out["country_iso_a3"].isna().sum() == 1 + + def test_a_non_integral_gid_never_matches_a_neighbouring_cell(self, enricher, lookup): + gid = int(lookup.index[100]) + out = self._out(enricher, [float(gid) + 1e-9]) + assert out["country_iso_a3"].isna().all(), ( + "a drifted float matched a cell — truncation turned it into a different, " + "real gid and fabricated that cell's geography" + ) + + def test_pandas_na_yields_null_rather_than_a_bare_typeerror(self, enricher, lookup): + good = lookup.index[:2].tolist() + out = self._out(enricher, pd.array([good[0], pd.NA, good[1]], dtype="Int64")) + assert len(out) == 3 and out["country_iso_a3"].isna().sum() == 1 + + def test_an_unusable_id_cannot_match_the_cell_it_was_substituted_with(self, tmp_path): + """The `& usable` term in `_gather`, proven load-bearing (ADR-014 §2). + + It survived removal against the committed artifact: an unusable id converts to + `0`, and `0` is not a real gid, so the key comparison already failed. That made + it look like decoration. It is not — it is the only thing standing between an + unusable id and the row it was substituted onto, and the substitute is a real + key the moment a lookup contains gid `0`. + + Without the mask, the NaN row below silently receives cell 0's geography. That + is the fabricated value this module's docstring forbids, produced from an input + the caller never wrote. + """ + cols = {"priogrid_gid": pa.array([0, 5, 9], pa.int64())} + for c in METADATA_COLS: + numeric = c in ("pg_xcoord", "pg_ycoord") or c.endswith("_code") + cols[c] = pa.array( + [0.0] * 3 if numeric else ["ZERO", "FIVE", "NINE"], + pa.float64() if numeric else pa.string(), + ) + path = tmp_path / "withzero.parquet" + pq.write_table( + pa.table(cols).replace_schema_metadata({b"lookup_version": b"t@00000000"}), path + ) + + out = GaulLookupEnricher(path).enrich_dataframe_with_pg_info( + pd.DataFrame({"priogrid_gid": [5.0, float("nan"), 9.0], "month_id": [1, 1, 1]}), + pg_id_col="priogrid_gid", time_id_col="month_id", + ) + assert out["country_iso_a3"].tolist() == ["FIVE", None, "NINE"], ( + "an unusable id matched the cell it was substituted with — `& usable` is " + "what prevents that, and this is the input that proves it" + ) + + def test_a_string_gid_column_is_refused_rather_than_parsed(self, enricher, lookup): + """A string gid is a declaration error, and the merge this replaced said so. + + `pandas` raised *"You are trying to merge on object and int64 columns. If you + wish to proceed you should use pd.concat"*. An + earlier draft of `_as_cell_ids` used `float(value)`, which happily parsed + `"54220"` and matched — inference where the old path declared (ADR-003). + """ + gid = str(int(lookup.index[0])) + out = self._out(enricher, [gid]) + assert out["country_iso_a3"].isna().all(), ( + f"the string {gid!r} was parsed into a cell id; a string column is a " + "declaration error, not a gid to be guessed at" + ) + + @pytest.mark.parametrize( + "label,array", + [ + # Each case must enter the branch it is testing. An earlier draft built + # them all with dtype=object, which routed every one through the + # element-wise branch — so the int and float bounds it claimed to test + # were never executed, and removing them left the suite green. + ("uint64 max (int branch)", np.array([np.iinfo(np.uint64).max], dtype=np.uint64)), + ("1e30 (float branch)", np.array([1e30], dtype=np.float64)), + ("inf (float branch)", np.array([np.inf], dtype=np.float64)), + ("-inf (float branch)", np.array([-np.inf], dtype=np.float64)), + ("string (object branch)", np.array(["123"], dtype=object)), + ("bool (object branch)", np.array([True], dtype=object)), + ("float exactly 2**63", np.array([2.0**63], dtype=np.float64)), + ("timedelta64", np.array([1], dtype="timedelta64[ns]")), + ], + ) + def test_out_of_range_and_non_numeric_values_are_unusable_not_wrapped( + self, enricher, label, array + ): + """Outside int64 the cast WRAPS rather than raising, and a wrapped id flagged + valid is exactly the fabricated value this conversion exists to prevent. + + `uint64` max becomes `-1`; `1e30` and `float(2**63)` become `INT64_MIN`. Each + passes a naive finite-and-integral test, so the bounds are separate conditions + rather than consequences of the others. + + Asserted through the PUBLIC method. An earlier draft called the private + `_as_cell_ids` and checked its mask — pinning an implementation detail rather + than the guarantee a caller relies on, and skipping `_gather`'s use of that + mask entirely. Every dtype below is reachable through a DataFrame column, so + there was never a reason to reach inside. + """ + out = enricher.enrich_dataframe_with_pg_info( + pd.DataFrame({"priogrid_gid": array, "month_id": [1] * len(array)}), + pg_id_col="priogrid_gid", time_id_col="month_id", + ) + assert out["country_iso_a3"].isna().all(), ( + f"{label} produced a match — an id the caller never wrote was treated as " + "a cell and that cell's geography was fabricated onto the row" + ) + + # AND at the mask, because the public assertion above cannot tell "refused" + # from "accepted but wrapped to an id that happens to be absent" — both give + # null. Reverting the float bound or re-admitting timedelta64 left the public + # assertion green while the value was silently accepted; only this sees it. + # The public check is the caller's guarantee; this one is the mutation proof. + _, usable = GaulLookupEnricher._as_cell_ids(array) + assert not usable.any(), ( + f"{label} was ACCEPTED as a cell id (it only looked refused because the " + f"wrapped id is absent from this lookup — against a lookup that contains " + f"it, this row would receive that cell's geography)" + ) + + def test_the_warning_names_unknown_cells_but_invents_no_id_for_unusable_ones( + self, enricher, lookup, caplog + ): + good = int(lookup.index[0]) + with caplog.at_level(logging.WARNING): + self._out(enricher, [float(good), float("nan"), 999999.0]) + message = " ".join(r.getMessage() for r in caplog.records) + assert "999999" in message, "the genuinely unknown cell must be named" + assert "no usable cell id" in message + assert "[0]" not in message, ( + "an unusable id was reported as cell 0 — a cell nobody asked about" + ) + + +class TestFramePropertiesPreserved: + """Row count, order and index survive the gather. Was CIC prose; now a check.""" + + @pytest.mark.parametrize("shape", ["ordered", "reversed", "duplicated", "single", "empty"]) + def test_row_count_and_order_are_preserved(self, enricher, lookup, shape): + gids = { + "ordered": lookup.index[:50].tolist(), + "reversed": lookup.index[:50].tolist()[::-1], + "duplicated": lookup.index[:5].tolist() * 3, + "single": lookup.index[:1].tolist(), + "empty": [], + }[shape] + out = enricher.enrich_dataframe_with_pg_info( + pd.DataFrame({"priogrid_gid": gids, "month_id": [1] * len(gids)}), + pg_id_col="priogrid_gid", time_id_col="month_id", + ) + assert len(out) == len(gids) + assert out["priogrid_gid"].tolist() == gids + + def test_a_non_default_index_is_preserved(self, enricher, lookup): + gids = lookup.index[:20].tolist() + df = pd.DataFrame({"priogrid_gid": gids, "month_id": [1] * 20}, + index=range(1000, 1020)) + out = enricher.enrich_dataframe_with_pg_info( + df, pg_id_col="priogrid_gid", time_id_col="month_id") + assert out.index.equals(df.index), ( + "the index moved. The pandas merge this replaced mutated it on empty " + "input; the gather must not mutate it on any input." + ) + + def test_name_columns_are_object_not_categorical(self, enricher, lookup): + """The CIC states this dtype. It changed in #89 and was measured, not guessed.""" + out = enricher.enrich_dataframe_with_pg_info( + pd.DataFrame({"priogrid_gid": lookup.index[:5].tolist(), "month_id": [1] * 5}), + pg_id_col="priogrid_gid", time_id_col="month_id", + ) + for c in NAME_COLS: + assert out[c].dtype == object, f"{c} is {out[c].dtype}, CIC says object" diff --git a/views_postprocessing/contract/enrichment.py b/views_postprocessing/contract/enrichment.py index 8f40a31..75baa77 100644 --- a/views_postprocessing/contract/enrichment.py +++ b/views_postprocessing/contract/enrichment.py @@ -7,10 +7,19 @@ area-majority GAUL parquets) onto the input DataFrame by PRIO-GRID cell id. No geopandas, no shapefiles, no spatial computation. The lookup contains only -fully-complete cells; an unknown or incomplete cell id left-merges to NaN, so -the manager's ``_validate()`` null gate still crashes the delivery (fail-loud) -rather than shipping a hole. This is intentional and matches the old mapper's -behaviour (it returned ``None`` for such cells). +fully-complete cells; an unknown or incomplete cell id gathers to null, so the +delivery's null gate still crashes (fail-loud) rather than shipping a hole. This +is intentional and matches the old mapper's behaviour (it returned ``None`` for +such cells). + +**The lookup side is pandas-free (S4 / #89, epic #85).** It is read with pyarrow +and held as numpy arrays plus a sorted key index; attaching metadata to a frame is +a **keyed gather**, not frame algebra, and it never needed a pandas merge. The +input frame is still whatever the caller passes — this class is the *build and +verification* path's object, and its callers hand it DataFrames. What changed is +that the ~880 KiB artifact is no longer materialised as a pandas frame, and the +join no longer depends on the artifact carrying pandas index metadata — which is +what unblocks S5 (#90) making the builder pyarrow-native. Produces exactly the 9-column contract enforced at the delivery's artifact builders and at @@ -21,10 +30,18 @@ import logging from pathlib import Path +from typing import TYPE_CHECKING -import pandas as pd +import numpy as np +import pyarrow as pa from views_postprocessing.contract import gaul_lookup + +if TYPE_CHECKING: # pragma: no cover — pandas is in this module's INTERFACE, not its + import pandas as pd # implementation. Callers hand it DataFrames; nothing here + # constructs, reads or joins one. Removing the runtime import is the point of + # S4 (#89): the ~880 KiB lookup is no longer materialised as a pandas frame, and the + # join no longer needs the artifact to carry pandas index metadata. from views_postprocessing.contract.gaul_schema import METADATA_COLS logger = logging.getLogger(__name__) @@ -33,6 +50,23 @@ # enricher's own default working without re-deriving the path. _DEFAULT_LOOKUP = gaul_lookup.LOOKUP_PATH +#: int64 bounds, named because they are a correctness condition rather than trivia: +#: a value outside them does not raise on cast, it WRAPS, and a wrapped id that is +#: flagged valid is the fabricated value this module forbids. +_INT64_MIN = int(np.iinfo(np.int64).min) +_INT64_MAX = int(np.iinfo(np.int64).max) + +#: Float-comparison bounds. Deliberately NOT the int bounds above: 2**63-1 is odd and +#: unrepresentable in float64, so comparing a float against it rounds up to 2**63 and +#: lets through the very values the bound excludes. 2**63 and -2**63 are both powers +#: of two and exact, so the float check is `>= -2**63` and `< 2**63`. +_INT64_MIN_F = -(2.0**63) +_INT64_MAX_EXCLUSIVE_F = 2.0**63 + +#: The lookup's key column. Named once — the artifact calls it `priogrid_gid`, the +#: wire calls it `priogrid_id` (§5.1), and confusing the two is a silent join failure. +_KEY = "priogrid_gid" + class GaulLookupEnricher: """Merge precomputed GAUL metadata onto an input DataFrame by cell id.""" @@ -46,19 +80,188 @@ def __init__(self, lookup_path: str | Path | None = None) -> None: ) logger.error(err_msg) raise FileNotFoundError(err_msg) - self._lookup = pd.read_parquet(self._lookup_path) - # Index is priogrid_gid; columns are the 9 metadata columns. - missing = [c for c in METADATA_COLS if c not in self._lookup.columns] + table = gaul_lookup.load(self._lookup_path) + missing = [c for c in (_KEY, *METADATA_COLS) if c not in table.column_names] if missing: err_msg = f"Lookup table is missing contract columns: {missing}" logger.error(err_msg) raise ValueError(err_msg) + + if table.column(_KEY).null_count: + err_msg = ( + f"GAUL lookup at {self._lookup_path} has null values in {_KEY!r}. A " + "null key cannot identify a cell, and coercing it would make it " + "collide with any unusable id on the query side — the row would then " + "be reported as FOUND and receive another cell's metadata. Rebuild " + "with scripts/build_gaul_lookup.py. (That script does not check the " + "key column either — its null check runs after the key becomes the " + "index, and DataFrame.isna() does not inspect an index. A null key is " + "unreachable there only because the earlier astype('int64') raises. " + "See register C-76.)" + ) + logger.error(err_msg) # ADR-008: logged persistently AND raised + raise ValueError(err_msg) + + if table.num_rows == 0: + err_msg = ( + f"GAUL lookup at {self._lookup_path} is empty. Every cell would gather " + "to null and the delivery would fail downstream complaining about " + "missing metadata rather than about a missing lookup. Rebuild it with " + "scripts/build_gaul_lookup.py." + ) + logger.error(err_msg) # ADR-008: logged persistently AND raised + raise ValueError(err_msg) + + # Sort the whole table once by key, in arrow, so the gather below is a binary + # search per row rather than a scan. `take` reorders every column together — + # doing it column-by-column in python was measurably quadratic and is exactly + # the mistake this comment exists to stop the next person repeating. + keys = table.column(_KEY).to_numpy(zero_copy_only=False).astype(np.int64) + order = np.argsort(keys, kind="stable") + table = table.take(pa.array(order)) + self._keys = keys[order] + # One `to_pylist` per column, not per row. Lists rather than numpy arrays + # because the columns are of mixed kind (float codes, string names) and the + # output is assembled per column anyway. + # + # DTYPE CHANGE, deliberate and measured (S4 / #89). The pandas merge this + # replaced produced `category` name columns, inherited from the artifact's + # dictionary encoding; assigning lists produces `object`. On a 200-row output: + # category 1,795,191 bytes vs object 60,061 — a categorical carries the full + # 64,742-entry dictionary whatever the output size, so this is far lighter for + # the small frames this object actually sees and heavier only past the point + # where the dictionary amortises. The builder's `# names/iso categorical (C-32 + # memory)` note governs the ARTIFACT; it never governed this method's output. + self._values = {col: table.column(col).to_pylist() for col in METADATA_COLS} self.lookup_version = self._read_version(self._lookup_path) logger.info( "Loaded GAUL lookup: %d cells from %s (version=%s)", - len(self._lookup), self._lookup_path, self.lookup_version, + len(self._keys), self._lookup_path, self.lookup_version, ) + @staticmethod + def _as_cell_ids(gids) -> tuple[np.ndarray, np.ndarray]: + """``(int64 ids, usable mask)`` — DECLARED, never coerced (ADR-003). + + A cell id is an integer. Anything that is not one — a missing value, a + non-integral float, a ``pd.NA`` — is marked **unusable** rather than cast, + and unusable ids gather to null exactly as an unknown gid does. The + downstream gate then sees a hole, which is what it exists for. + + **Why not just cast.** ``np.asarray(gids, dtype=np.int64)`` looks equivalent + and is not, in three ways found in review of #210: + + - a ``NaN`` becomes ``INT64_MIN`` with only a ``RuntimeWarning``. If a lookup + key were ever null it would take the same sentinel, the two would collide, + and the row would be reported FOUND — receiving another cell's metadata. + That is the fabricated value this module's docstring forbids (cf. C-35). + - a non-integral float **truncates silently**: ``54220.000000001`` becomes + ``54220`` and matches a real, *different* cell. Verified against the pandas + merge this replaced: it returns null there, and even warns. A silent wrong + match is strictly worse than the crash it would replace. + - a ``pd.NA`` raises a bare ``TypeError`` from numpy — no log, no + contract-shaped error, unlike every other guard in this file. + """ + arr = np.asarray(gids) + + if arr.dtype.kind in ("i", "u"): + # In range, or it is not the id the caller wrote. uint64 max silently + # wraps to -1 under `astype(np.int64)` and would be flagged VALID. + usable = (arr >= _INT64_MIN) & (arr <= _INT64_MAX) + return np.where(usable, arr, 0).astype(np.int64), usable + + if arr.dtype.kind == "f": + # Finite, integral, AND representable. `1e30` passes the first two and + # then overflows the cast to INT64_MIN — an id nobody asked for, marked + # valid. Same silent-coercion class this method exists to remove. + # STRICTLY below 2**63, not `<= _INT64_MAX`. `_INT64_MAX` is 2**63-1, + # which is odd and NOT representable in float64 — comparing against it + # promotes to float and rounds UP to 2**63, so the bound admitted exactly + # the values it was added to exclude. `float(2**63)` passed every check + # and then wrapped to INT64_MIN, flagged valid. `_INT64_MIN` has no + # equivalent hole: -2**63 is a power of two and exactly representable. + # + # `isfinite` is kept for intent and is SUBSUMED — `inf` fails the upper + # bound, `NaN` fails `arr == rint(arr)` (NaN equals nothing). It is the + # one condition here that survives its own removal, so it is defensive + # rather than load-bearing; the other three each fail the suite. + usable = ( + np.isfinite(arr) + & (arr == np.rint(arr)) + & (arr >= _INT64_MIN_F) + & (arr < _INT64_MAX_EXCLUSIVE_F) + ) + return np.where(usable, arr, 0).astype(np.int64), usable + + # object / pandas-nullable. Accept only values that ALREADY ARE integers. + # + # An earlier draft used `float(value)`, which parses. That accepted the string + # `"54220"` as a cell id — where the pandas merge this replaced raised + # `ValueError: You are trying to merge on object and int64 columns`. A string + # gid column is a declaration error and the old path said so; parsing it is + # inference (ADR-003) and it is the same defect as the drifted float, just + # pointing the other way. `bool` is excluded for the same reason: `True` is + # not cell 1. + ids = np.zeros(len(arr), dtype=np.int64) + usable = np.zeros(len(arr), dtype=bool) + for i, value in enumerate(arr): + # `np.timedelta64` IS an `np.integer` instance — a numpy quirk, and + # `np.datetime64` is not, so the hole was specific to that one type. A + # duration is not a cell id any more than `True` is cell 1. + if isinstance(value, (bool, np.timedelta64)): + continue + if not isinstance(value, (int, np.integer)): + continue + if _INT64_MIN <= int(value) <= _INT64_MAX: + ids[i], usable[i] = int(value), True + return ids, usable + + def _gather(self, gids) -> dict: + """The metadata for each gid, warning about the ones that got none. + + A sorted-key ``searchsorted`` rather than a hash map: the lookup is 64,742 + rows read once per process, and the gather is over the delivery's row count. + Absent gids yield ``None`` in every column — the null the downstream gate + exists to catch, not a sentinel it would pass. An id that is not a usable + cell id is treated the same way: absent, never guessed at. + """ + wanted, usable = self._as_cell_ids(gids) + # `self._keys` is non-empty — __init__ refuses an empty lookup. An earlier draft + # guarded with `(len(self._keys) > 0) & (...)`, which READS as a guard and is + # not one: `&` evaluates both operands, so the index happened regardless and an + # empty lookup raised IndexError from inside the gather rather than ValueError + # from the constructor. Guard where the condition is knowable, not where it bites. + idx = np.clip(np.searchsorted(self._keys, wanted), 0, len(self._keys) - 1) + # `& usable` is NOT belt-and-braces, and no substitute value would make it so. + # Unusable ids substitute to 0, but int64 reserves nothing — the lookup is + # arbitrary data and may legally contain 0, or -1, or INT64_MIN, or whatever + # else one might pick instead. The mask is what carries correctness; the + # substitute is only a placeholder. Pinned by + # `test_an_unusable_id_cannot_match_the_cell_it_was_substituted_with`. + found = (self._keys[idx] == wanted) & usable + out = { + col: [vals[i] if hit else None for i, hit in zip(idx, found)] + for col, vals in self._values.items() + } + + # Warn HERE rather than handing the caller three arrays to reassemble one + # message. `__init__` already co-locates detection and logging at every guard + # (ADR-008); this is the same shape. An earlier draft returned four values, + # three of which existed only to build the string below. + absent = ~found + n_unmapped = int(absent.sum()) + if n_unmapped: + unknown = sorted({int(i) for i, miss, ok in zip(wanted, absent, usable) if miss and ok}) + n_unusable = int((absent & ~usable).sum()) + detail = f"unknown cells {unknown[:20]}" if unknown else "no unknown cells" + if n_unusable: + detail += f"; {n_unusable} row(s) carried no usable cell id" + logger.warning( + "%d/%d rows have no lookup match (will fail validation): %s", + n_unmapped, len(wanted), detail, + ) + return out + @staticmethod def _read_version(path: Path) -> str: """The lookup's build stamp. Delegates to ``gaul_lookup.version`` (#152) — @@ -102,19 +305,12 @@ def enrich_dataframe_with_pg_info( else: base = df.copy() - merged = base.merge( - self._lookup, left_on=pg_id_col, right_index=True, how="left", - ) + gids = base[pg_id_col].to_numpy() + gathered = self._gather(gids) - n_total = len(merged) - n_unmapped = int(merged["country_iso_a3"].isna().sum()) - if n_unmapped: - logger.warning( - "%d/%d rows have no lookup match (will fail validation): %s", - n_unmapped, n_total, - sorted(merged.loc[merged["country_iso_a3"].isna(), pg_id_col] - .unique().tolist())[:20], - ) + merged = base.copy() + for col in METADATA_COLS: + merged[col] = gathered[col] return merged # Convenience alias for new call sites that don't need the legacy name. diff --git a/views_postprocessing/contract/gaul_lookup.py b/views_postprocessing/contract/gaul_lookup.py index 6907f31..23f17bb 100644 --- a/views_postprocessing/contract/gaul_lookup.py +++ b/views_postprocessing/contract/gaul_lookup.py @@ -3,7 +3,7 @@ One concept — *the precomputed geography asset* — which until #152 was scattered: its path was a private name in ``enrichment.py`` imported across module boundaries (register C-68), its version stamp was a ``@staticmethod`` on ``GaulLookupEnricher`` -that never touched the instance, and the delivery read the 888 KB parquet **three +that never touched the instance, and the delivery read the ~880 KiB parquet **three times per run** (C-66) — once eagerly into a pandas enricher it never used, then twice more through pyarrow. From 9799e8717a91eb169f0096917b74b67c84925e78 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Sun, 2 Aug 2026 17:54:54 +0200 Subject: [PATCH 21/23] =?UTF-8?q?feat(crafd):=20CRAF'd=20producer=20packag?= =?UTF-8?q?e=20=E2=80=94=20clone=20of=20unfao/=20(per=20CLONING.md)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stand up the views-postprocessing producer for the second consumer, CRAF'd (views-crafdapi). Per CLONING.md this is "replace three, keep everything else": - crafd/product.py — TARGETS = the same three GED fatality series (CRAF'd is FAO extended, same forecasts/geography; additional targets are a later Amendment A1 edit), CONSUMER_DOCUMENT_NAME = "un_crafd" (ADR-034 §6), S_MIN = 2, UPLOAD_ENABLED = False (§11.4 interlock — staged locally, zero store calls, until the views-crafdapi selection guard is deployed). - crafd/appwrite_env.py — CRAFD_ENV (APPWRITE_CRAFD_* coordinates, filled by the operator at views-crafdapi S9 / views-appwrite #38); connection + prod-forecasts env shared with FAO (same internal shelf source). Contract pinned at 1.3.0 / 47172af (same edition as unfao; the CRAFD names are reserved there). - crafd/managers/crafd.py — CRAFDPostProcessorManager: the pipeline-core seam, reads the shared production_forecasts shelf and delivers the same wire to crafd_bucket. Mechanical retarget of the FAO manager; no partner-neutral machinery touched. - test_doc_accuracy: the pipeline-core-importer guard now expects BOTH partner managers (unfao + crafd) — a new partner manager is expected, not blast-radius creep; anything OUTSIDE the managers still fails. VALIDATION: ruff-clean and AST-parses locally; the FULL suite is validated by CI, because this machine (Python 3.13) cannot build the repo's pinned pyarrow 16.1.0 (no 3.13 wheel) — CI runs the correct 3.11/3.12 + 16.1.0 toolchain. Co-Authored-By: Claude Opus 4.8 --- tests/test_doc_accuracy.py | 8 +- views_postprocessing/crafd/__init__.py | 0 views_postprocessing/crafd/appwrite_env.py | 95 ++++ .../crafd/managers/__init__.py | 1 + views_postprocessing/crafd/managers/crafd.py | 406 ++++++++++++++++++ views_postprocessing/crafd/product.py | 42 ++ 6 files changed, 549 insertions(+), 3 deletions(-) create mode 100644 views_postprocessing/crafd/__init__.py create mode 100644 views_postprocessing/crafd/appwrite_env.py create mode 100644 views_postprocessing/crafd/managers/__init__.py create mode 100644 views_postprocessing/crafd/managers/crafd.py create mode 100644 views_postprocessing/crafd/product.py diff --git a/tests/test_doc_accuracy.py b/tests/test_doc_accuracy.py index ea16dd2..869daca 100644 --- a/tests/test_doc_accuracy.py +++ b/tests/test_doc_accuracy.py @@ -228,9 +228,11 @@ def test_views_pipeline_core_has_exactly_one_importer(): for f in _PKG.rglob("*.py") if "views_pipeline_core" in f.read_text() ) - assert importers == ["unfao/managers/unfao.py"], ( - f"views_pipeline_core is imported by {importers}. ADR-012 and register C-40 both " - "state it is one file wide; a second importer widens C-40's blast radius." + assert importers == ["crafd/managers/crafd.py", "unfao/managers/unfao.py"], ( + f"views_pipeline_core is imported by {importers}. ADR-012 and register C-40 state it " + "is confined to the per-partner manager seam — one file per delivery (unfao, crafd). " + "An importer OUTSIDE those managers widens C-40's blast radius; a new partner manager " + "is expected and joins this list." ) diff --git a/views_postprocessing/crafd/__init__.py b/views_postprocessing/crafd/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/views_postprocessing/crafd/appwrite_env.py b/views_postprocessing/crafd/appwrite_env.py new file mode 100644 index 0000000..4758f12 --- /dev/null +++ b/views_postprocessing/crafd/appwrite_env.py @@ -0,0 +1,95 @@ +"""The Appwrite-seam environment this package requires, DECLARED (þing-01 P1, #134). +Clone of ``unfao/appwrite_env.py`` for the CRAF'd delivery (its own outbound bucket). + +Names follow **the Appwrite Seam Contract**'s coordinate registry (connection + +target classes) plus the one operator-issued secret slot. The contract is homed in +views-appwrite and referenced by pinned URL, never copied — copies were the +platform's original failure (þing-01 sáttmál S6): + + https://github.com/views-platform/views-appwrite/blob/47172af/docs/ADRs/platform/coordinate_registry.toml + +That pin is registry **v1.3.0** (ratified, þing-02) — declared below as +``SEAM_CONTRACT_VERSION`` / ``SEAM_CONTRACT_COMMIT`` so the pin is a value a test can +check rather than a fact buried in prose. The ``APPWRITE_CRAFD_*`` names are declared +at that edition (reserved slots); their *values* were filled by the operator at +views-crafdapi S9 (views-appwrite PR #38) — but this module declares names, never +reads values. A pinned URL does not rot, but it does go stale, and nothing in this +repository could previously tell you it had (register C-57). + +**Pin from the tip of `main`, never from a sibling checkout's `HEAD`** (#196). + +The LAUNCHER assembles the environment +(views-models M3: run.sh reads the owned registry; the secret stays the operator +slot) — this package loads no dotenv and validates fail-loud instead (verdict D6). + +Deliberately dependency-light: no pipeline-core imports, so the declaration is +importable (and testable) everywhere. +""" + +from __future__ import annotations + +import logging +import os + +logger = logging.getLogger(__name__) + +#: The Appwrite Seam Contract edition these names were verified against, and the commit +#: this repo cites. **A version string and a sha are not coordinate values** — the +#: registry forbids copying its values, and nothing here copies one. What is recorded is +#: *which edition was read*, which is exactly what makes drift detectable. +SEAM_CONTRACT_VERSION = "1.3.0" +SEAM_CONTRACT_COMMIT = "47172af" + +CONNECTION_ENV = ( + "APPWRITE_ENDPOINT", + "APPWRITE_DATASTORE_PROJECT_ID", + "APPWRITE_DATASTORE_API_KEY", # secret slot — the value must never be logged +) +PROD_FORECASTS_ENV = ( + "APPWRITE_PROD_FORECASTS_BUCKET_ID", + "APPWRITE_PROD_FORECASTS_BUCKET_NAME", + "APPWRITE_PROD_FORECASTS_COLLECTION_ID", + "APPWRITE_PROD_FORECASTS_COLLECTION_NAME", + "APPWRITE_METADATA_DATABASE_ID", + "APPWRITE_METADATA_DATABASE_NAME", +) +CRAFD_ENV = ( + "APPWRITE_CRAFD_BUCKET_ID", + "APPWRITE_CRAFD_BUCKET_NAME", + "APPWRITE_CRAFD_COLLECTION_ID", + "APPWRITE_CRAFD_COLLECTION_NAME", + "APPWRITE_METADATA_DATABASE_ID", + "APPWRITE_METADATA_DATABASE_NAME", +) + + +def assert_env_declared(names: tuple, *, store: str) -> None: + """Entry validation (þing-01 D6): every required name resolved and non-empty, + or raise naming ALL missing variables — never a partial config that half-works. + + Logs at ERROR before raising (ADR-008): the refusal must survive in the run's + log, not only in a traceback the operator no longer has. Names only — the + resolved value of a variable is never read, and never logged. + + Args: + names: the environment-variable names this store requires. + store: the store being configured, for the message. + + Raises: + EnvironmentError: naming every missing variable. + """ + missing = [name for name in names if not os.getenv(name)] + if missing: + # `missing` holds NAMES, never values: membership is decided by + # `os.getenv(name)` being falsy, and the resolved value is never read. + # That is what makes this loggable at all — CONNECTION_ENV carries the + # APPWRITE_DATASTORE_API_KEY secret slot. + err_msg = ( + f"{store}: the launcher did not assemble the required environment — " + f"missing {missing}. Coordinates come from the Appwrite Seam Contract's " + "coordinate registry, homed in views-appwrite (views-models run.sh " + "declares its sourcing); the secret is the operator slot. This package " + "no longer loads any dotenv (#134)." + ) + logger.error(err_msg) # ADR-008: logged persistently AND raised + raise EnvironmentError(err_msg) diff --git a/views_postprocessing/crafd/managers/__init__.py b/views_postprocessing/crafd/managers/__init__.py new file mode 100644 index 0000000..fe29bf0 --- /dev/null +++ b/views_postprocessing/crafd/managers/__init__.py @@ -0,0 +1 @@ +from .crafd import CRAFDPostProcessorManager as CRAFDPostProcessorManager \ No newline at end of file diff --git a/views_postprocessing/crafd/managers/crafd.py b/views_postprocessing/crafd/managers/crafd.py new file mode 100644 index 0000000..8613160 --- /dev/null +++ b/views_postprocessing/crafd/managers/crafd.py @@ -0,0 +1,406 @@ +from views_pipeline_core.managers.postprocessor.postprocessor import ( + PostprocessorManager, + PostprocessorPathManager, +) +import logging + +from views_pipeline_core.modules.appwrite.file import AppwriteConfig +from views_pipeline_core.modules.datastore import DatastoreModule +from views_pipeline_core.managers.model import ForecastingModelManager + +from views_pipeline_core.managers.ensemble import EnsemblePathManager +from datetime import datetime +import os +from views_pipeline_core.modules.dataloaders.datafactory_contract import declared_data_format +from views_postprocessing.contract import frame_extraction, gaul_lookup, historical, launch_config, source_metadata, store_metadata +from views_postprocessing.crafd import appwrite_env, product +from views_postprocessing.contract.wire import sink as wire_sink +from views_postprocessing.contract.wire import source_selection +from views_postprocessing.delivery import coverage, observed_range, provenance +from pathlib import Path + +logger = logging.getLogger(__name__) + + +class _ContractStorePort: + """Adapts ``DatastoreModule`` to the wire ports (ADR-013 epic #105; DIP — + ``wire/source_selection`` and ``wire/sink`` never see Appwrite types).""" + + def __init__(self, datastore: DatastoreModule) -> None: + self._dsm = datastore + + def latest_file_id(self, filters: dict): + return self._dsm.get_latest_file_id(filters=filters) + + def file_metadata(self, file_id: str) -> dict: + return store_metadata.file_metadata(self._dsm.get_file_metadata(file_id)) + + def download(self, file_id: str) -> bytes: + return ( + self._dsm.download_prediction(file_id).to_dict().get("data", {}).get("file_bytes", None) + ) + + def upload(self, file_path, *, filename, name, doc_type, category, loa, targets, description=None) -> None: + result = self._dsm.upload_data( + file=file_path, + filename=filename, + name=name, + type=doc_type, + category=category, + loa=loa, + targets=targets, + description=description, + ) + # The store module degrades gracefully (its ADR-046 policy) and only LOGS + # metadata failures — which strands an invisible orphan file (run-0 + # historical, 2026-07-27). The delivery fails loud instead. + success = getattr(result, "success", None) + if success is None and hasattr(result, "to_dict"): + success = result.to_dict().get("success") + if success is False: + error = getattr(result, "error", None) or "unknown store error" + raise RuntimeError( + f"upload of {filename!r} did not fully succeed (file may be an " + f"orphan without a metadata document): {error}" + ) + + +class CRAFDPostProcessorManager(PostprocessorManager, ForecastingModelManager): + def __init__( + self, + model_path: PostprocessorPathManager, + wandb_notifications: bool = True, + use_prediction_store: bool = False, + ) -> None: + super().__init__(model_path, wandb_notifications, use_prediction_store) + + # Add your custom initialization below + logger.info(f"Initializing {self.__class__.__name__}") + self._forecast_resolution = None # {target: TargetLease}, set by _read + self._historical_frame = None # views_frames.FeatureFrame, set by _read + self.ensemble_path_manager = None + + def _read_historical_frame(self): + """#126: historical actuals as a views_frames.FeatureFrame — the first + production consumer of pipeline-core's frame-native fetch. Same clip + policy as the legacy path (producer-sourced boundary, degrade-open).""" + frame = self._data_loader.get_feature_frame( + partition="forecasting", use_saved=False, level="pgm", validate=True + ) + try: + lv = source_metadata.last_valid_month_id(self.configs.get("zarr_url")) + except Exception: + logger.warning("last_valid_month_id unavailable; skipping clip (degrade-open, C-26).", exc_info=True) + lv = None + if lv is None: + self._historical_frame = frame + return + fabricated = observed_range.fabricated_months(frame_extraction.months_of(frame), lv) + if len(fabricated): + logger.warning( + "Dropping %d fabricated (unobserved) month(s) above last_valid_month_id=%d.", + len(fabricated), lv, + ) + frame = frame_extraction.drop_months_above(frame, lv) + self._historical_frame = frame + + def _read_historical_data(self): + """Historical actuals, frame-native only (#126). + + The queryset must DECLARE ``data_format: feature_frame`` — pipeline-core's + ``declared_data_format`` is the one gate, and a queryset that declares + anything else is refused rather than quietly read through a retired pandas + path (register C-63, #149). + """ + # Declaration first: the check reads the queryset, not the loader, so a + # refused config must not pay for loader construction. + launch_config.assert_frame_native_historical( + declared_data_format(self._model_path.get_queryset()) + ) + self._initialize_data_loader() + self._read_historical_frame() + + def _prod_forecasts_datastore(self) -> DatastoreModule: + """The shared internal store (ADR-013's 'shared shelf'), configured from the + launcher-assembled environment (validated fail-loud; þing-01 #134 — no dotenv + is loaded here). + + pipeline-core's ``DatastoreModule.get_predictions_by_metadata`` injects an + automatic ``name == model_name`` filter on every lookup. The contract read + must **not** have it: ADR-013 artifacts are named + ``{run_id}__{target}__m{month}.arrow.parquet`` (never the bare ensemble + name), so an injected ``name == "rusty_bucket"`` matches nothing and also + clobbers the wire layer's own run-id / target / name filters. Suppressed + unconditionally below — the retired legacy reader was the only caller that + needed it on (#149).""" + ensemble_name = self.configs.get("ensemble", None) + if not ensemble_name: + err_msg = "Ensemble name must be provided in configs with the `ensemble` key for forecasting. Cannot proceed." + logger.error(err_msg) + raise ValueError(err_msg) + self.ensemble_path_manager = EnsemblePathManager(ensemble_name_or_path=ensemble_name, validate=False) + # ensemble_configs = EnsembleManager( + # ensemble_path=self.ensemble_path_manager, + # ).configs + + # loa = ensemble_configs.get("level", None) + loa = "pgm" + if not loa: + err_msg = "level must be defined in the ensemble configurations (e.g, pgm, cm). Cannot proceed." + logger.error(err_msg) + raise ValueError(err_msg) + + appwrite_env.assert_env_declared( + appwrite_env.CONNECTION_ENV + appwrite_env.PROD_FORECASTS_ENV, + store="production_forecasts datastore", + ) + appwrite_config = AppwriteConfig( + path_manager=self.ensemble_path_manager, + endpoint=os.getenv("APPWRITE_ENDPOINT"), + project_id=os.getenv("APPWRITE_DATASTORE_PROJECT_ID"), + credentials=os.getenv("APPWRITE_DATASTORE_API_KEY"), + auth_method="api_key", + cache_ttl_hours=24, + bucket_id=os.getenv("APPWRITE_PROD_FORECASTS_BUCKET_ID"), + bucket_name=os.getenv("APPWRITE_PROD_FORECASTS_BUCKET_NAME"), + collection_id=os.getenv("APPWRITE_PROD_FORECASTS_COLLECTION_ID"), + collection_name=os.getenv("APPWRITE_PROD_FORECASTS_COLLECTION_NAME"), + database_id=os.getenv("APPWRITE_METADATA_DATABASE_ID"), + database_name=os.getenv("APPWRITE_METADATA_DATABASE_NAME"), + ) + datastore = DatastoreModule(appwrite_file_manager_config=appwrite_config) + # Suppress the automatic name==model_name filter (see docstring). model_path + # is used by DatastoreModule only for that injection and for uploads; the + # contract read neither uploads nor performs any model-scoped lookup. + datastore.model_path = None + return datastore + + def _read_forecast_data_contract(self): + """ADR-013 contract inbound (epic #105; streaming since the run-0 OOM fix): + RESOLVE the newest fully-manifested run — manifests + pinned shard + file_ids only, no heavy bytes. Frames materialize one target at a time + inside the sink at _save (each lease loads → verifies → curates → is + released). The `wire/` package owns the policy; this method only adapts + the store (DIP) and declares the product facts (region curation + + coverage expectations live in the lease, where frames exist).""" + port = _ContractStorePort(self._prod_forecasts_datastore()) + region = self.configs.get("region") + self._forecast_resolution = source_selection.resolve_run( + port, + expected_targets=product.TARGETS, + expected_ensemble=self.configs["ensemble"], + excluded_gids=coverage.excluded_for(region), + expected_cells=coverage.expected_for(region), + ) + run_id = next(iter(self._forecast_resolution.values())).run_id + logger.info( + "Contract inbound resolved: run %s (%d targets leased; region=%r; " + "heavy fetch deferred to delivery).", + run_id, + len(self._forecast_resolution), + region, + ) + + def _read_forecast_data(self): + """Forecast inbound — ADR-013 contract only. + + The launcher must DECLARE ``wire_contract: True``. The pandas reader this + key used to select was retired in #149; omitting the key is a refusal, not + a fallback (register C-63). + """ + launch_config.assert_contract_mode(self.configs) + self._read_forecast_data_contract() + + def _read(self) -> any: + self._read_historical_data() + self._read_forecast_data() + + def _transform(self) -> None: + """No-op by design. + + Geography is not joined into either payload: the historical artifact + attaches it at build time (``contract/historical.py``) and the forecast ships + it as the §5 GAUL sidecar, built in the sink at ``_save``. The hook stays + so the Template Method's phases remain truthful. + """ + return + + def _validate(self) -> None: + """Assert the read produced what the save needs, then check coverage. + + Neither payload is null-gated here. The historical artifact's metadata + null-gate fires at build time (``historical.assert_metadata_complete``); + the forecast's guarantees are the wire's own verified chain — content + hashes, header/payload asserts, and per-target coverage inside each + lease's ``load()``, plus the §6 no-collapse gate and gid parity at + ``_save``. This phase asserts the RESOLUTION happened, keeping the + Template Method's phases truthful: ``_read`` resolves, ``_save`` + materializes. + """ + if self._historical_frame is None: + raise ValueError("no historical frame — _read did not run.") + if self._forecast_resolution is None: + raise ValueError("no resolved forecast run — _read did not resolve.") + logger.info( + "Historical is frame-native: the metadata null-gate is enforced at " + "artifact build (historical.assert_metadata_complete)." + ) + self._check_coverage() + + def _check_coverage(self) -> None: + """Log delivered cell counts and enforce the region coverage contract (S1/C-34). + + Orchestration only: extract primitives via ``frame_extraction``, then call + the representation-free ``delivery.coverage`` invariant — the rule is + *called*, not embedded. The count-gate fires only for regions pinned in + ``coverage.EXPECTED_CELLS``; an unpinned/unresolved region logs a skipped gate + rather than guessing. + + Only the historical leg is checked here. Forecast coverage is asserted + inside each lease's ``load()``, where the frame actually exists — see + ``wire/source_selection``. + """ + region = self.configs.get("region") + expected = coverage.expected_for(region) + excluded = coverage.excluded_for(region) + for label, cells, n_rows in [self._historical_coverage_source()]: + logger.info( + "%s delivery coverage: %d distinct cells, %d rows.", + label, + len(cells), + n_rows, + ) + # GAUL-uncovered cells the curated region must drop (S4/C-30) — checked + # before the count gate so a leaked island names itself, not "over-coverage + # by 1". Empty for unpinned regions (e.g. africa_me_legacy keeps its ocean + # cells), so this is a no-op there. + if excluded: + coverage.assert_no_excluded_cells(cells, excluded, label=label) + if expected is not None: + coverage.assert_complete_coverage(cells, expected, label=label) + else: + logger.warning( + "Coverage count-gate skipped for %s: region %r is not pinned in " + "delivery.coverage.EXPECTED_CELLS — verify and pin before relying on it.", + label, + region, + ) + + def _save_contract(self) -> dict: + """ADR-013 contract outbound (epic #105): the composed sink delivers the run. + + The §11.4 interlock is enforced by ``wire.sink`` itself: with the default + ``product.UPLOAD_ENABLED=False`` (overridable only by the explicit + ``wire_upload_enabled`` launch-config key), artifacts are staged locally and + ZERO store calls occur. First live enablement is gated on C-161 closure. + """ + if self._forecast_resolution is None: + raise ValueError( + "contract _save called without a resolved run — _read must run first." + ) + # ONE read of the 888 KB lookup per delivery (#152/C-66), threaded to both + # consumers — each takes it as a parameter (DIP), so neither reaches for the + # file itself. + lookup = gaul_lookup.load() + upload_enabled = bool(self.configs.get("wire_upload_enabled", product.UPLOAD_ENABLED)) + store = _ContractStorePort(self._crafd_datastore()) if upload_enabled else None + # The wire is partner-neutral (#153): the manager supplies CRAF'd's product + # facts explicitly rather than the mechanism reaching for them. + summary = wire_sink.deliver_run( + self._forecast_resolution, + lookup=lookup, + staging_dir=Path(self._model_path.data_generated) / "wire_contract", + consumer_name=product.CONSUMER_DOCUMENT_NAME, + s_min=product.S_MIN, + store=store, + upload_enabled=upload_enabled, + ) + # Historical leg (#126): the CRAF'd product ships actuals alongside the wire — + # frame-built, same artifact shape faoapi already ingests, same interlock. + if self._historical_frame is None: + raise ValueError( + "contract _save: no historical frame — the un_crafd descriptor must " + "declare data_format: feature_frame (#126)." + ) + hist_path, hist_description, _ = self._build_historical_artifact( + Path(summary["staging_dir"]), lookup + ) + if upload_enabled: + store.upload( + hist_path, + filename=hist_path.name, + name=self._model_path.model_name, + doc_type="model", + category="historical", + loa="pgm", + targets=list(self.configs.get("targets", [])), + description=hist_description, + ) + logger.info("uploaded %s (historical, run %s)", hist_path.name, summary["run_id"]) + else: + logger.info( + "Interlock holding: historical artifact staged at %s (no store calls).", + hist_path, + ) + summary["historical"] = hist_path.name + return summary + + def _crafd_datastore(self) -> DatastoreModule: + """The CRAF'd-facing store (`crafd_bucket`).""" + return DatastoreModule(appwrite_file_manager_config=self._crafd_appwrite_config()) + + def _crafd_appwrite_config(self) -> AppwriteConfig: + appwrite_env.assert_env_declared( + appwrite_env.CONNECTION_ENV + appwrite_env.CRAFD_ENV, store="crafd_bucket datastore" + ) + return AppwriteConfig( + path_manager=self._model_path, + endpoint=os.getenv("APPWRITE_ENDPOINT"), + project_id=os.getenv("APPWRITE_DATASTORE_PROJECT_ID"), + credentials=os.getenv("APPWRITE_DATASTORE_API_KEY"), + auth_method="api_key", + cache_ttl_hours=24, + bucket_id=os.getenv("APPWRITE_CRAFD_BUCKET_ID"), + bucket_name=os.getenv("APPWRITE_CRAFD_BUCKET_NAME"), + collection_id=os.getenv("APPWRITE_CRAFD_COLLECTION_ID"), + collection_name=os.getenv("APPWRITE_CRAFD_COLLECTION_NAME"), + database_id=os.getenv("APPWRITE_METADATA_DATABASE_ID"), + database_name=os.getenv("APPWRITE_METADATA_DATABASE_NAME"), + ) + + def _save(self) -> dict: + """Deliver the run — ADR-013 contract only (#149).""" + return self._save_contract() + + def _historical_coverage_source(self): + """(label, cells, n_rows) for the historical frame.""" + return ( + "historical", + frame_extraction.cells_of(self._historical_frame), + self._historical_frame.n_rows, + ) + + def _historical_frame_description(self, table, timestamp: str) -> str: + """The C-15 provenance description for the frame-built historical artifact.""" + region = self.configs.get("region") + prov = provenance.build_provenance( + lookup_version=gaul_lookup.version(), + region=region, + expected_cell_count=coverage.expected_for(region), + actual_cell_count=len(frame_extraction.cells_of(self._historical_frame)), + unmapped_count=historical.unmapped_cell_count(table), + ) + return provenance.compact_description(prov) + + def _build_historical_artifact(self, directory, lookup) -> tuple: + """Frame-built historical artifact staged into ``directory``; returns + (path, description, timestamp). Null-gate enforced here (fail loud). + + ``lookup`` is the already-read GAUL table (injected, not fetched — #152). + """ + timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") + table = historical.build_historical_table(self._historical_frame, lookup) + historical.assert_metadata_complete(table) + path = Path(directory) / f"historical_dataset_{timestamp}.parquet" + historical.write_historical_artifact(table, path) + return path, self._historical_frame_description(table, timestamp), timestamp diff --git a/views_postprocessing/crafd/product.py b/views_postprocessing/crafd/product.py new file mode 100644 index 0000000..b378a9a --- /dev/null +++ b/views_postprocessing/crafd/product.py @@ -0,0 +1,42 @@ +"""The declared CRAF'd delivery product (ADR-013 §4.2a's "views-postprocessing +configuration"). Clone of ``unfao/product.py`` for the second external partner — +the Complex Risk Analytics Fund (CRAF'd), served by views-crafdapi. + +Everything here is a **declaration**, never an inference: the delivery refuses to +ship until reality matches these constants, and changing the product is a human +decision plus an edit here — reviewed, git-historied, fail-loud. One reason to +change: the CRAF'd partner relationship. + +CRAF'd is FAO *extended* (same VIEWS forecasts, same PRIO-GRID geography, same +cadence): for now the same three conflict-fatality series, delivered to CRAF'd's +own bucket. The uncertainty *surface* CRAF'd adds — exceedance probabilities +alongside HDI/MAP — lives in the consumer (views-crafdapi ADR-034), not here; the +producer ships the same posterior-sample wire the FAO producer ships. Additional +targets, when CRAF'd names them, are an Amendment A1 edit to ``TARGETS`` below. + +The constants and their contract homes: + +- ``TARGETS`` — the expected target set (§4.2a): a run is *complete* only when every + target listed here has a manifested Hop-A leg. Adding a target follows Amendment + A1 (§7a): maintainer names it, producer's mapping gains an entry, this tuple + gains an entry. Wire vocabulary only — never internal model names. +- ``S_MIN`` — the §6 no-collapse floor passed to ``delivery.draws``. +- ``CONSUMER_DOCUMENT_NAME`` — the §4.1a store-document ``name`` pin. views-crafdapi's + query layer filters on it unconditionally; a document under any other name is + invisible to the consumer. Config-owned by views-crafdapi (ADR-034 §6); changing + it is a contract amendment. +- ``UPLOAD_ENABLED`` — the §11.4 upload interlock: ``False`` means the sink writes + artifacts locally and never calls the store. Overriding requires an explicit + launch-config declaration, and the first live enablement is gated on the + views-crafdapi consumer's selection guard being deployed in production. +""" + +from __future__ import annotations + +TARGETS: tuple[str, ...] = ("lr_ged_sb", "lr_ged_ns", "lr_ged_os") + +S_MIN: int = 2 + +CONSUMER_DOCUMENT_NAME: str = "un_crafd" + +UPLOAD_ENABLED: bool = False From 6d587d1a3718f55b36c35dc0019853da3d83e8ca Mon Sep 17 00:00:00 2001 From: Polichinl Date: Mon, 3 Aug 2026 12:06:19 +0200 Subject: [PATCH 22/23] =?UTF-8?q?fix(guards):=20#211=20=E2=80=94=20every?= =?UTF-8?q?=20partner-scoped=20guard=20was=20scoped=20to=20ONE=20partner?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `crafd/` landed exempt from eight guards, because each named `unfao` as a string. Five review passes; each found defects in the previous pass's fixes. The defect crafd shipped with - `crafd/appwrite_env.py` pinned registry v1.3.0/47172af — an edition at which its own four APPWRITE_CRAFD_* coordinates had no values, while the file's docstring cited the PR that filled them. Re-pinned to v1.4.1/90fc105. What was unguarded, and is now - partner list declared ONCE (`tests/conftest.py:PARTNER_PACKAGES`), asserted against the filesystem, and consumed by every guard that needs it - þing-01 dotenv borrow: covered 2 files of 30. A live `load_dotenv(find_dotenv())` in `unfao/appwrite_env.py` — the entry validator whose docstring says the borrow is dead — ran on import with the suite green. Now a package-wide AST walk, which sees imports and calls and cannot see the prose that legitimately describes it - ADR-002 both legs: `contract/` could import `crafd`, and `delivery/ -> contract/` had no test at all. Both are now AST import walks plus a subprocess import-purity check. Regexes escaped three times (relative, bare-package, function-body) before being replaced - line budget: the last hardcoded list; a 906-line third manager passed it. Derived, and applied to the whole `managers/` directory - consumer document name: nothing asserted crafd's. Pinned, plus a gated cross-seam check reading views-crafdapi's own `APIPathManager` literal and its filter field - `_MACHINERY` had no completeness assertion, so `contract/enrichment.py` was exempt from both purity guards Production code - `_ContractStorePort.upload`'s comment claimed the pipeline-core store "only LOGS" metadata failures. It returns `success=False, code="PARTIAL_SUCCESS"`. The comment was self-defeating: if it only logged, the guard it explains could never fire Records disposed of in the same change (ADR-014 §5) - ADR-012 ontology, CLONING.md, role_and_seams, the CIC, unfao/managers/README, README tree, `contract/__init__` — all said "the" partner or "only one" importer - CLONING.md over-claimed þing-02 S24(5) as a flat prohibition; the verdict binds the repos cut from views-faoapi, not an in-repo partner package of the producer. It also dropped #146's named trigger and asserted a tracker state #146 did not have (commented) - register: C-77, C-78, C-79 added; C-33, C-40, C-47, C-57 amended; 76/17/59 -> 79/19/60 Also - retired `test_version_bumped_past_latest_tag`: it asserted a version is NOT tagged, so it went red whenever views-datafactory was in a clean released state. Its purpose is covered by `test_land_gaul_commit_is_in_a_release_tag` - `sibling_repo` briefly raised on a bad override and turned a typo into "0 tests run". Moved to its own assertion - C-33's measurement was wrong six times, twice by edits that shifted lines it had just counted. No entry states manager line numbers now — they name symbols, and the docs state the `diff` command instead of its result 361 passed, 40 xfailed, 0 failed (conda env, pipeline-core 3.0.0 editable, pyarrow 16.1.0); ruff clean. Every guard mutation-proven; ~25 mutations, all reverted. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 17 +- docs/ADRs/012_revised_ontology.md | 30 +- docs/CICs/UNFAOPostProcessorManager.md | 2 +- docs/CLONING.md | 105 +++++- docs/architecture/role_and_seams.md | 17 +- reports/technical_risk_register.md | 149 +++++++- tests/conftest.py | 70 ++++ tests/test_clone_readiness.py | 223 ++++++++++-- tests/test_datafactory_deploy_readiness.py | 72 ++-- tests/test_doc_accuracy.py | 136 +++++++- tests/test_env_declaration.py | 328 ++++++++++++++---- tests/test_frame_extraction.py | 2 +- tests/test_product.py | 169 ++++++++- tests/test_views_frames_conformance.py | 2 +- views_postprocessing/contract/__init__.py | 12 +- .../contract/wire/__init__.py | 5 +- views_postprocessing/crafd/appwrite_env.py | 30 +- views_postprocessing/crafd/managers/crafd.py | 12 +- views_postprocessing/unfao/managers/README.md | 9 +- views_postprocessing/unfao/managers/unfao.py | 12 +- 20 files changed, 1169 insertions(+), 233 deletions(-) diff --git a/README.md b/README.md index b9b74f8..f2e071f 100644 --- a/README.md +++ b/README.md @@ -135,17 +135,24 @@ views-postprocessing/ │ ├── source_metadata.py # producer (datafactory) facts │ ├── store_metadata.py # prediction-store facts │ └── launch_config.py # the delivery mode the launcher must declare - ├── unfao/ # WHO A DELIVERY IS FOR — the only FAO-specific code + ├── unfao/ # WHO A DELIVERY IS FOR — the FAO-specific code │ ├── product.py # targets, consumer name, S_MIN, upload interlock │ ├── appwrite_env.py # the declared store coordinates │ └── managers/unfao.py # UNFAOPostProcessorManager + ├── crafd/ # WHO A DELIVERY IS FOR — the CRAF'd-specific code + │ ├── product.py # same three files, same shape (register C-33 on + │ ├── appwrite_env.py # why the manager is a copy, and what would + │ └── managers/crafd.py # make it time to stop copying) └── data/gaul_lookup.parquet # the precomputed GAUL lookup (ADR-011) ``` -**Dependencies point one way only:** `unfao/` → `contract/` → `delivery/`. Nothing in -`contract/` may import `unfao/` — that is what lets a new partner reuse the machinery -without inheriting FAO, and it is enforced by `tests/test_clone_readiness.py`, not by -convention. See [`docs/CLONING.md`](docs/CLONING.md). +**Dependencies point one way only:** `/` → `contract/` → `delivery/`. Nothing +in `contract/` may import a partner package — that is what lets a new partner reuse the +machinery without inheriting another partner's product, and it is enforced by +`tests/test_clone_readiness.py`, not by convention. The partner list lives in one place +(`tests/conftest.py`) and is itself checked against the filesystem, so a package added +without being declared fails rather than passing quietly. +See [`docs/CLONING.md`](docs/CLONING.md). --- diff --git a/docs/ADRs/012_revised_ontology.md b/docs/ADRs/012_revised_ontology.md index 79dcc3e..433e70a 100644 --- a/docs/ADRs/012_revised_ontology.md +++ b/docs/ADRs/012_revised_ontology.md @@ -40,8 +40,21 @@ packages answer three different questions, and every category below names the on delivery/ what makes a delivery VALID — representation-free invariants contract/ how a delivery is BUILT — partner-neutral machinery unfao/ who a delivery is FOR — one partner's product and manager +crafd/ who a delivery is FOR — another partner's product and manager ``` +**Amended 2026-08-03 (#211): the third row repeats.** `crafd/` joined `unfao/` as a +second partner package. This does not add a fourth category — a partner package is a +partner package, and the closed set is unchanged. What it changes is that *"who a +delivery is FOR"* is answered N times rather than once, and every claim below that said +**"the"** partner or **"one"** module now says how many. + +A partner package is the shape a partner takes **inside this repository**. It is not +the same thing as a partner *repository*: `views-crafdapi` and `views-productionapi` +are consumer APIs cut from `views-faoapi`, and this repository is the single producer +that serves all of them. `docs/CLONING.md` was written before that was settled and +described the cut-a-repo case; it now says which is which. + | Category | Purpose | Authority | Stability | |----------|---------|-----------|-----------| | **Delivery Invariants** | Representation-free rules over primitives that a delivery must satisfy: coverage, no-collapse, gid parity, observed-range, provenance. Live in `delivery/` — **nothing there imports pandas or views_frames**. *Forecast identity was one of these until 2026-07-31 — see the amendment below.* | Authoritative — they define what a valid delivery is | Stable — changes are governance decisions | @@ -51,8 +64,8 @@ unfao/ who a delivery is FOR — one partner's product and manager | **Artifact Builders** | `contract/historical.py` — turns a frame plus the lookup into the partner-facing artifact. | Derived | Evolving | | **External Facts** | Facts read from systems this repo does not own: the producer's (`contract/source_metadata.py` — `last_valid_month_id`, D-07) and the store's (`contract/store_metadata.py`). | Authoritative (the owning system is the source of truth) | Evolving | | **Launch Declarations** | `contract/launch_config.py` — the delivery mode the launcher must declare. Omitting a key is **refused by name**, never inferred (ADR-003, register C-63). | Authoritative | Stable | -| **Partner Product** | `unfao/product.py` (targets, consumer document name, collapse floor, upload interlock) and `unfao/appwrite_env.py` (the store coordinates). **This is what a clone replaces.** | Authoritative — one reason to change: the partner relationship | Evolving | -| **Pipeline Manager** | `unfao/managers/unfao.py` — a concrete pipeline-core postprocessor (Template-Method subclass) that *orchestrates* read/transform/validate/save and **calls** the invariants, never inherits them. **It is the only module in the repository that imports `views_pipeline_core`** — the coupling C-40 describes is one file wide. It is not yet *thin*: 406 lines, down from 636 (#149). | Derived | Evolving | +| **Partner Product** | Per partner: `/product.py` (targets, consumer document name, collapse floor, upload interlock) and `/appwrite_env.py` (the store coordinates). Two exist — `unfao/` and `crafd/` (#211). **This pair plus the Pipeline Manager below is what a new partner supplies** — three files, as `docs/CLONING.md` states them. | Authoritative — one reason to change: that partner relationship | Evolving | +| **Pipeline Manager** | One per partner: `/managers/.py` — a concrete pipeline-core postprocessor (Template-Method subclass) that *orchestrates* read/transform/validate/save and **calls** the invariants, never inherits them. **These are the only modules in the repository that import `views_pipeline_core`**, pinned to an explicit allowlist by `tests/test_doc_accuracy.py` — the coupling C-40 describes is one file per partner. Neither is yet *thin* — each sits just under the 450-line budget `tests/test_doc_accuracy.py` holds them to, down from 636 (#149), and since #211 the second is a near-verbatim copy of the first — deliberate WET with a named extraction trigger, recorded in register **C-33**. | Derived | Evolving | | **Derived Outputs** | Arrow shards, the GAUL sidecar, the run manifest and the historical parquet, produced per run and delivered to the partner store. | Ephemeral | Ephemeral | **Two claims this ADR made until 2026-08-01, both now corrected rather than quietly dropped** @@ -78,9 +91,16 @@ waiting for an audit. - **Screaming architecture:** the categories match the package layout — `delivery/` (invariants), `contract/` (the machinery: `wire/`, the `frame_extraction.py` seam, the - GAUL asset, artifact builders, external-fact readers), `unfao/` (the partner's product and - its manager). A reader can infer responsibilities from the structure, and the one-way - dependency `unfao/ → contract/ → delivery/` is enforced by test, not convention. + GAUL asset, artifact builders, external-fact readers), and one package per partner — + `unfao/`, `crafd/` — each holding that partner's product and its manager. A reader can + infer responsibilities from the structure, and the one-way dependency + `/ → contract/ → delivery/` is enforced by test, not convention — **both legs + of it, for every declared partner, since #211.** Neither was fully true before: + `test_contract_package_does_not_import_any_partner` named `unfao`, so `contract/` was + free to import `crafd`; and the `contract/ → delivery/` leg had no test at all until + `test_the_invariants_do_not_import_the_machinery` was written — a documented arrow + that half existed, which is worse than an undocumented one because a reader stops + checking. - **DIP / OCP:** primitives are the abstraction the invariants depend on; the representation seam is the single point of change for a representation migration (C-40), so the invariants are closed against it. diff --git a/docs/CICs/UNFAOPostProcessorManager.md b/docs/CICs/UNFAOPostProcessorManager.md index 2629c93..b1a4d77 100644 --- a/docs/CICs/UNFAOPostProcessorManager.md +++ b/docs/CICs/UNFAOPostProcessorManager.md @@ -105,7 +105,7 @@ The following **must never** fail silently: - PRIO-GRID geometry details - The internals of how the lookup table was built -This anchors the class within ADR-002 (topology): `unfao/` → `contract/` → `delivery/`, one way only. It is **the repository's only importer of `views_pipeline_core`** (mechanically pinned by `tests/test_doc_accuracy.py`), which is what makes C-40's blast radius one file wide. 406 lines as of epic #148, down from 636 — not yet *thin*, and held under a 450-line budget by the same test. +This anchors the class within ADR-002 (topology): `unfao/` → `contract/` → `delivery/`, one way only — and since #211 the same holds for `crafd/`, the second partner package. It is one of **the repository's only two importers of `views_pipeline_core`** (both mechanically pinned to an allowlist by `tests/test_doc_accuracy.py`), which keeps C-40's blast radius at one file per partner. Not yet *thin*: it came down from 636 lines at epic #148 and now sits just under a **450-line budget**, which `tests/test_doc_accuracy.py` applies to the whole `managers/` directory of each partner rather than to this file alone — a seam that holds its line count by moving 800 lines into a sibling module has not held anything. The exact figure is deliberately not repeated here; the test carries it. --- diff --git a/docs/CLONING.md b/docs/CLONING.md index 40145e8..dd1c20c 100644 --- a/docs/CLONING.md +++ b/docs/CLONING.md @@ -1,7 +1,19 @@ -# Cloning this repository for a new partner - -Read this **before** cutting `views-crafdapi`, `views-productionapi`, or any future -partner delivery. It is short on purpose. +# Adding a new partner delivery + +Read this **before** adding a partner package to this repository, or cutting a partner +API repository that consumes one. It is short on purpose. + +> **Corrected 2026-08-03 (#211). This document used to say "before cutting +> `views-crafdapi`", and that framed the job wrongly.** `views-crafdapi` and +> `views-productionapi` are **consumer** APIs, cut from `views-faoapi`. They do not +> clone *this* repository. This repository is the single **producer** that serves every +> partner, and a new partner is added here as a package alongside `unfao/` and +> `crafd/` — not as a new producer repo. +> +> The three-things-to-supply structure below survives that correction unchanged, because +> it was always describing the same three files. What changes is where they go: into a +> new directory in this repo, not into a new repository. The "Hard rules" section below +> is where the distinction actually mattered, and it is corrected there too. ## What you get for free @@ -16,10 +28,18 @@ partner package arrives with them). ## What you must supply -Three things. They are the only FAO-specific files in the repository, so the shape of -your work is: **replace these three, keep everything else.** +Three things. They are the only partner-specific files in the repository, so the shape +of your work is: **copy these three from an existing partner, change them, keep +everything else.** `crafd/` is the worked example. One caveat if you read it as a +template: it did *not* copy `unfao/managers/README.md`, the operational summary that +sits beside the FAO manager. That was an omission rather than a decision — write one. + +Register them as you add them: `tests/conftest.py` holds the repository's single +declared partner list (`PARTNER_PACKAGES`), and a partner missing from it is exempt +from every guard below. A test asserts that list against the filesystem, so forgetting +fails CI rather than passing quietly — which is what happened when `crafd/` landed. -### 1. Your product — `unfao/product.py` +### 1. Your product — `/product.py` Four declarations, and nothing may be inferred: @@ -32,25 +52,78 @@ Four declarations, and nothing may be inferred: - **`UPLOAD_ENABLED`** — the interlock. **Leave it `False`** until your consumer's selection guard is deployed in production, not merely merged. -### 2. Your store coordinates — `unfao/appwrite_env.py` +### 2. Your store coordinates — `/appwrite_env.py` The env-var names your delivery requires, validated fail-loud before any store is constructed. Names come from the **Appwrite Seam Contract's coordinate registry** (homed in views-appwrite) and are referenced **by URL at a pinned commit, never copied**. The secret stays an operator slot. -### 3. Your manager — `unfao/managers/unfao.py` +### 3. Your manager — `/managers/.py` + +The pipeline-core seam. **The partner managers are the only modules in the repository +that import `views_pipeline_core`**, and a test holds them to an explicit allowlist. +Yours will orchestrate read → transform → validate → save and *call* the invariants — +never inherit them. + +Today the two managers are near-identical. See for yourself rather than trusting a +number here — the number went stale twice while this paragraph was being written: + +``` +diff views_postprocessing/unfao/managers/unfao.py \ + views_postprocessing/crafd/managers/crafd.py +``` + +Sixteen lines differ on each side and **none of them changes behaviour**: the import, +the class name, the two partner-named methods and their two call sites, the four +env-var literals, one line that both selects which env tuple is validated and labels +the store, one runtime refusal message, and four lines of prose. -The pipeline-core seam. **This is the only module in the repository that imports -`views_pipeline_core`**, and a test keeps it that way. Yours will orchestrate -read → transform → validate → save and *call* the invariants — never inherit them. +**That is deliberate** — WET before DRY, and the second copy is what finally showed the +seam is a config object rather than a behavioural one. Register **C-33** carries the +extraction trigger: a **third** in-repo partner, or the first bug that has to be +hand-patched identically in both files. If you are the third, read C-33 before copying +a fourth time. ## Hard rules, and why each exists -**Do not import `views_pipeline_core.modules.{appwrite,datastore}`.** -þing-02 **S24(5)**, binding. This repository's own import of those is how a two-repo -defect became three (register C-40); pipeline-core declines to offer the surface, and -that refusal is deliberate. Write a thin client against the SDK, as views-faoapi did. +**Your manager may import `views_pipeline_core.modules.{appwrite,datastore}`. +Nothing else in this repository may — and you add yourself to the allowlist by hand.** +`tests/test_doc_accuracy.py::test_views_pipeline_core_is_confined_to_the_partner_managers` pins the +importer set to an explicit list of manager files. Adding a partner means editing that +list deliberately. That is the cost of a new partner, not a formality: the coupling is +bounded only because someone has to write the file's name down. + +Pipeline-core declines to export that surface, and this repository's own import of it is +how a two-repo defect became three (register **C-40**). Unwinding it is deferred under +issue **#146** behind a **named trigger — þ01-D8's supply trigger firing on the C-221 +decomposition, explicitly not on this repository's convenience**. Do not read the +deferral as "not done yet"; it is a decision with a condition attached (ADR-014 §4). + +**If you are cutting a consumer API repo, the rule inverts: do *not* import them.** +þing-02 **S24(5)** binds the repositories cut from views-faoapi — `views-crafdapi` +(the þing records call it `un-crafdapi`) and `views-productionapi`. It does not reach a +partner package inside this producer, which is why `crafd/managers/crafd.py` may import +what a consumer API may not. An earlier version of this document cited the verdict as a +flat prohibition and over-claimed it. Write a thin client against the SDK, as +views-faoapi did. + +**Check the store's result. It is load-bearing, not boilerplate.** +`_ContractStorePort.upload` inspects `result.success` and raises. It looks like +defensive noise and is not. When metadata storage fails after the file is already +uploaded, the pipeline-core store logs the error and **returns +`OperationResult(success=False, code="PARTIAL_SUCCESS")`** — it reports the failure +faithfully and simply does not raise. A caller that discards the result therefore +proceeds as though the delivery were complete, leaving a file with no metadata +document: invisible to the consumer, exactly like a wrong document name. That happened +to run-0's historical artifact on 2026-07-27. + +This matters more than it reads, because of a date. þing-02 **D10/S30** required this +repository's legacy path to be guarded or retired **before 2026-11-30**, when the +current key expires — an unguarded path on that day *reports success and ships nothing*. +The legacy path itself was retired in #149 (register C-63), so what the obligation now +amounts to is keeping this guard on the contract path. If you copy a manager you inherit +it; do not tidy it away. **Get your own key before the first run, not after.** Free at t=0, a migration later. One key per identity per environment (the Appwrite Seam Contract diff --git a/docs/architecture/role_and_seams.md b/docs/architecture/role_and_seams.md index 7813b2c..1a4e99a 100644 --- a/docs/architecture/role_and_seams.md +++ b/docs/architecture/role_and_seams.md @@ -14,7 +14,10 @@ metadata, guards their integrity, and delivers them to a partner store** — it **post-forecast delivery layer**, not a spatial-mapping library and not a statistical post-processor. -The only live consumer today is the **UN FAO** delivery (`views_postprocessing/unfao/`). +Two partner deliveries live here: the **UN FAO** one +(`views_postprocessing/unfao/`), delivering to FAO-FSFC since 2026-07-27, and +**CRAF'd** (`views_postprocessing/crafd/`), added 2026-08-03 with its upload interlock +still closed. They are peers — one producer, one partner package each. --- @@ -182,11 +185,17 @@ views_postprocessing/ │ ├── source_metadata.py producer (datafactory) facts, e.g. last_valid_month_id │ ├── store_metadata.py prediction-store facts │ └── launch_config.py the delivery mode the launcher must declare -├── unfao/ WHO A DELIVERY IS FOR — the only FAO-specific code +├── unfao/ WHO A DELIVERY IS FOR — the FAO-specific code, and only that │ ├── product.py targets, consumer document name, S_MIN, upload interlock │ ├── appwrite_env.py the declared store coordinates -│ └── managers/unfao.py UNFAOPostProcessorManager (406 lines; the only importer -│ of views_pipeline_core) +│ └── managers/unfao.py UNFAOPostProcessorManager +├── crafd/ WHO A DELIVERY IS FOR — the CRAF'd-specific code (same three +│ │ files, same shape; register C-33 on why it is a copy) +│ ├── product.py +│ ├── appwrite_env.py +│ └── managers/crafd.py CRAFDPostProcessorManager +│ the two managers are the ONLY importers of +│ views_pipeline_core — one per partner, allowlisted by test └── data/gaul_lookup.parquet the precomputed GAUL lookup (ADR-011) ``` diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 0b0a8e0..787473f 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -4,10 +4,10 @@ |-------------------|--------------------------------------| | Project | views-postprocessing | | Owner | Dylan Pinheiro / PRIO MD&D Team | -| Last Updated | 2026-08-02 | -| Total Concerns | 76 | -| Open Concerns | 17 | -| Resolved Concerns | 59 | +| Last Updated | 2026-08-03 | +| Total Concerns | 79 | +| Open Concerns | 19 | +| Resolved Concerns | 60 | --- @@ -32,6 +32,7 @@ covered a single open entry (see Historical clusters below). ### Cluster G: Inherited pipeline-core surface **Root cause:** this repo *is-a* pipeline-core postprocessor by double inheritance, so it inherits that project's data loader, container, store I/O, and dependency tree — defects in that surface land in FAO delivery without this repo owning the fix. **Entries:** C-40 (root), C-07, C-13, C-26, C-27, C-28, C-29, C-44, C-58, C-62 +**Amended 2026-08-03:** the root's defining measurement — pipeline-core imported by exactly one module — became **two** when `crafd/managers/crafd.py` landed (PR #211). The count is still pinned by an explicit allowlist, so the cluster's boundary holds; what changed is that every fix in it now has two landing sites. See C-33 for why the second copy is deliberate and what triggers its removal. **Highest tier:** 1 (C-26) **Fix strategy:** the thin-shell de-inheritance C-40 prescribes — and which is **half-built**: the sink side landed (`_ContractStorePort`, `unfao.py:37-78`) and the invariants are already pipeline-core-free modules the manager calls (`delivery/*`, `unfao/historical.py`, `unfao/wire/`). The remaining half is the **input** side (loader + `PGMDataset`), gated on pipeline-core Epic #186/#207. **Resolution scope:** Partial — C-26/C-27/C-28 are upstream-owned; de-inheritance makes them visible and testable, not fixed. @@ -316,8 +317,9 @@ See also D-10 (handling decision), C-43 (the *value*-correctness sibling — run | ID | C-33 | | Tier | 2 — two to three additional Appwrite stores are planned imminently; the current design forces copy-pasting a 273-line manager per store | | Source | `expert-code-review` (2026-06-12) | -| Trigger | When the second Appwrite prediction store is configured (issue #97 scoping), verify store identity comes from configuration — the env **names** are now centrally declared, but the three `AppwriteConfig` constructions, the targets list, and the category strings are still inline per-store | -| Location | `views_postprocessing/unfao/managers/unfao.py:148-204` (`_prod_forecasts_datastore`), `:495-517` (`_unfao_datastore`/`_unfao_appwrite_config`), `:528-534` (legacy `_save`); declared names in `views_postprocessing/unfao/appwrite_env.py` | +| Trigger | **Fired 2026-08-03 — see the update below.** The remaining trigger is the *extraction* one, and it is now named: a **third** in-repo partner package, **or** the first bug that must be hand-patched identically in both manager files — whichever comes first. | +| Owner | Whoever adds the third partner package, or hits the first double-patch. Until one of those happens the duplication is the deliberate WET position, not a task anyone is behind on. | +| Location | `views_postprocessing//managers/.py` — `_prod_forecasts_datastore`, `__datastore`, `__appwrite_config`, and the four hardcoded `os.getenv("APPWRITE__*")` literals inside the last of those; declared names in each partner's `appwrite_env.py`. **Symbols, not line numbers** — see the note under the measurement below. | Mitigation: a small `DeliveryProfile` (bucket/collection/database ids, category, targets) passed to the manager — one manager class, N store configs. Scheduled **after** the FAO global delivery ships (D-09); the only immediate action is deleting the commented-out config blocks at lines 80-107, which are a mis-uncomment hazard during deadline work. @@ -327,7 +329,38 @@ Mitigation: a small `DeliveryProfile` (bucket/collection/database ids, category, 3. **Partially mitigated by þing-01 #134.** `unfao/appwrite_env.py` now declares the env **names** centrally (`CONNECTION_ENV`, `PROD_FORECASTS_ENV`, `UNFAO_ENV`) and validates them fail-loud before every `AppwriteConfig` construction, following the PLATFORM-001 coordinate registry. Names are no longer scattered string literals. **What is still hardcoded is store *identity*** — which names apply to which store, the targets list, and the category strings — so the `DeliveryProfile` case stands. Tier held at 2. 4. **The deferral condition has expired**: D-09 scheduled this "after the FAO global delivery ships." It shipped 2026-07-27. Ready for the "calm 1-day job" whenever #97 scoping lands. -See also C-24 (schema contract per store), D-09 (the deferral, now expired), #97 (second-store scoping). +**Update 2026-08-03 (PR #211) — the thing this entry warned about has happened, and it is being kept on purpose.** + +This entry's own Tier-2 rationale was that the design *"forces copy-pasting a 273-line manager per store."* PR #211 added `views_postprocessing/crafd/` — a second partner package whose `managers/crafd.py` is a **line-for-line copy** of `unfao/managers/unfao.py`. Measured with + + diff views_postprocessing/unfao/managers/unfao.py \ + views_postprocessing/crafd/managers/crafd.py | grep -c '^[<>]' + +**32** — sixteen differing lines on each side. Substitute every form of the partner name (case-insensitively, including `un_fao`/`un_crafd` and `faoapi`) and it falls to **2**: one line per side. + +The sixteen are, by category: one import, one class name, two partner-named method definitions, their two call sites, one refusal-message string, one line that is *both* the `*_ENV` tuple reference and the store label, the four env-name literals, and four lines of prose. + +**None of the difference is behaviour, but "byte-identical" is too strong for one method.** `_read`, `_transform`, `_validate`, `_check_coverage` and `_build_historical_artifact` are byte-identical. `_save_contract` is not: five of the sixteen fall inside it — the datastore call, two comments, and the refusal string. All five are partner-name substitutions; none changes what the method does. + +*(**This paragraph was wrong five times, and how it was wrong is the entry's most useful content.** (1) "roughly ten lines", carried from the review that found it and never measured. (2) A normalised count of 4 and a claim that `_save_contract` was byte-identical, neither checked. (3) A story that #211 "fixed two divergences that already existed" — false: at `9799e87` the second line was **byte-identical in both files**, an inherited inaccuracy rather than a divergence, and rewording CRAF'd's copy is what *created* a divergence there. Only the `:222` pair was real. (4) and (5) An exact list of sixteen line numbers and a line count, invalidated twice within the hour by comment corrections elsewhere in the same file.* + +*The fix was not a sixth careful re-count. **Neither this measurement nor any `Location` field in C-33, C-40, C-77 or C-79 states a line number any more** — they name symbols, which grep can find and which survive an edit above them. There was a sixth failure, and it is why: a draft of this very paragraph announced that the entry "no longer states line numbers" while its own `Location` row still carried six, every one of them shifted by four lines by a comment correction made in the same commit. `tests/test_doc_accuracy.py` had already written the rule down — "The FILE is the claim; the line number is not." A count a reader relies on is a claim under ADR-014 §1, and each of these six was written by someone who believed it.)* + +**The duplication is the right call today, and this is the record that says why.** CLAUDE.md's WET rule asks for a *second incident* before extracting, and this is genuinely it: one implementation showed nothing, two show that the seam is trivial — a partner-config object, not a behavioural one. The two candidate abstractions are both worse than the copy. A shared base class would stack a second, repo-owned Template Method under pipeline-core's imposed one, producing a three-tier inheritance chain to deduplicate ~10 lines; it would also have to live somewhere, and `contract/` is pinned pipeline-core-free by `tests/test_clone_readiness.py`. A factory solves a dispatch problem this system does not have — each partner is wired explicitly by its own launcher config, and nothing selects a manager class at runtime. + +**What was missing was the trigger, and ADR-014 §4 says a deferral without one is not a deferral.** It is now in the Trigger field above. Both halves matter: a *third* package is the point at which "two copies you can hold in your head" becomes sprawl, and the *first double-patched bug* is the point at which the copies start costing correctness rather than bytes. + +**Drift is the live risk, and both directions of it showed up immediately.** + +*A real divergence the copy created.* At `9799e87`, `unfao/managers/unfao.py:222` named the artifact builder as `unfao/historical.py` — a path retired by #153 — while the fresh copy said `contract/historical.py` and was correct. **The clone silently fixed a stale reference in the original and the fix never propagated back.** #211 fixed the original too. + +*An inherited inaccuracy that was not a divergence — until fixing it made one.* Both files carried *"same artifact shape faoapi already ingests"*. Identical, so no diff flagged it; wrong for CRAF'd, whose consumer is not faoapi. #211 reworded CRAF'd's copy, which is correct for both files and **puts that line into the raw diff for the first time**. A copy can therefore drift by being corrected, and a rising count is not by itself evidence of anything going wrong. + +Both were prose, both were harmless, and together they are how a 16-line diff becomes a 40-line one — in under a day, with no contributor doing anything careless. #211 also extended the line-budget guard (`tests/test_doc_accuracy.py`) to cover **both** managers rather than only `unfao.py`, which had left the second copy of the file epic #148 shrank from 636 lines with no regrowth protection at all. + +**What this does not license.** The four env-name literals in `__appwrite_config` duplicate names that `appwrite_env.py` already declares as data, in both files. Removing that is not an abstraction and does not wait for the trigger — it is not encoding the partner's identity twice in the same package. Left as a follow-up rather than folded into #211, which is a partner-addition PR. + +See also C-24 (schema contract per store), C-77 (the fourth home for partner identity, in the duplicated `name=` argument of the historical upload), D-09 (the deferral, now expired), ADR-014 §4 (a deferral needs a trigger and an owner), #97 (second-store scoping), #211. --- @@ -339,7 +372,7 @@ See also C-24 (schema contract per store), D-09 (the deferral, now expired), #97 | Tier | 2 | | Source | `expert-code-review` (2026-06-24) | | Trigger | **(a) Upstream change:** when pipeline-core changes `PGMDataset` / the data loader / the postprocessor base (mid-migration: their #186/#188/#161), verify the inherited surface this repo depends on still holds. **(b) Standing work item:** the input-side de-inheritance (the sink side landed — see the 2026-07-31 update) — schedule it, don't wait for a trigger. | -| Location | `views_postprocessing/unfao/managers/unfao.py:80` (double inheritance); `:148-204`, `:495-534` (inline env/AppwriteConfig/DatastoreModule); `:276-287` (`_append_metadata`), `:300-349` (`_validate`); DIP sink adapter at `:37-78` (`_ContractStorePort`) | +| Location | `views_postprocessing//managers/.py` — the `class PostProcessorManager(PostprocessorManager, ForecastingModelManager)` statement (double inheritance); `_prod_forecasts_datastore`, `__datastore`, `__appwrite_config` (inline env/AppwriteConfig/DatastoreModule); `_validate` and `_check_coverage`; the DIP sink adapter `_ContractStorePort`. **Since 2026-08-03 all of it exists twice** — `unfao` and `crafd` are the same file with the partner name changed (C-33). Symbols rather than lines, deliberately: an earlier version of this row was invalidated by a comment edit four lines long. | `UNFAOPostProcessorManager` subclasses **two concrete** pipeline-core base classes (`PostprocessorManager`, `ForecastingModelManager`) and **interleaves infrastructure** (env reading, `AppwriteConfig` construction, `DatastoreModule`, path resolution) with the FAO **business logic** (GAUL enrichment, the 9-column null gate) inside the lifecycle hooks. Consequences: (a) the FAO logic cannot be instantiated or unit-tested without the full framework + Appwrite env + viewser; (b) **pandas cannot leave the delivery path** because the inherited data loader and `PGMDataset` are pandas — gated on pipeline-core's own DataFrame retirement; (c) **SDP exposure** — heavy *inheritance* coupling to a pipeline-core that is itself unstable (mid-migration), so upstream changes break far from their cause (cf. C-27, C-29); (d) it's the repo's only composition-over-inheritance violation. The dependency itself is correct (`unfao.py` genuinely *is* a pipeline-core postprocessor) — the issue is its **blast radius**. Mitigation (does **not** fight the Template-Method framework): keep the subclass as a **thin shell** but extract `enrich` + `validate` + the 9-column contract into a pipeline-core-free core object the manager *calls*, and wrap the Appwrite I/O behind a small delivery-sink adapter (DIP). This makes the FAO logic testable standalone and insulates it from pipeline-core churn. @@ -369,7 +402,15 @@ See also C-24 (schema contract per store), D-09 (the deferral, now expired), #97 *Did:* the surrounding surface shrank sharply. The manager is **406 lines** (from 636); it imports neither pandas nor `PGMDataset`; the partner-neutral machinery moved out to `contract/` (#153); and **`views_pipeline_core` is still imported by exactly one module — this one — now pinned mechanically** by `tests/test_doc_accuracy.py` and `tests/test_clone_readiness.py`. That property is what keeps this entry's blast radius one file wide, and it is no longer a claim anyone has to re-check by hand. -*Did not:* the double inheritance at `unfao.py:80` stands, and so do consequences (a) — the FAO logic still cannot be instantiated without the framework — and (c)/(d). **This entry remains open on exactly that scope.** Its remaining fix is gated on views-pipeline-core's 3.0.0 (C-44/C-62), which is a release signal rather than engineering work. +**⚠ Superseded 2026-08-03 (PR #211): "exactly one module" is now exactly TWO.** `views_postprocessing/crafd/managers/crafd.py` is the second, and it imports the same `views_pipeline_core.modules.{appwrite,datastore}` surface at the same lines. The claim above was true when written and is left visible rather than edited away, per ADR-014 §5. + +**What actually changed, and what did not.** The blast radius is no longer *one file wide* — it is **one file, twice**, which is a different and slightly worse property: an upstream change now has two identical landing sites and no mechanism guarantees they are patched together (C-33). What did **not** change is the more important half: the count is still **bounded and pinned**. `test_views_pipeline_core_is_confined_to_the_partner_managers` (renamed in #211 — it had asserted *two* under a name that said *one*) was widened to an explicit allowlist, not deleted, so a *third* importer still fails CI. Every other module in the repository remains pipeline-core-free, including the whole of `contract/` and `delivery/`, and `tests/test_clone_readiness.py` still proves the machinery imports in a subprocess without it. + +**On þing-02 S24(5).** `docs/CLONING.md` cited that verdict as forbidding these imports outright. Reading it directly (`þingit/02_credential_identity_key_ownership/sáttmál.md:240-242` — precondition (5) itself; the section opens at `:232` under the heading *"§5 — The clone (`un-crafdapi`)"* — and `orð_dómr.md:418-441`), it binds *"the clone"* — `un-crafdapi` and `views-productionapi`, repositories **git-cloned from views-faoapi** — and does not reach an in-repo partner package of the producer. CLONING.md over-claimed; PR #211 corrects the citation rather than weakening the rule. This entry's own scope is unaffected: the coupling is a design concern here regardless of what the verdict binds, and issue **#146**'s deferred unwind now covers two files instead of one. + +Tier held at 2. The residual scope — the double inheritance and the framework-bound instantiation — is unchanged, and is still gated on views-pipeline-core 3.0.0 (C-44/C-62). + +*Did not:* the double inheritance (the `class UNFAOPostProcessorManager(...)` statement — this row cited `unfao.py:80` when written, and that number has moved twice since) stands, and so do consequences (a) — the FAO logic still cannot be instantiated without the framework — and (c)/(d). **This entry remains open on exactly that scope.** Its remaining fix is gated on views-pipeline-core's 3.0.0 (C-44/C-62), which is a release signal rather than engineering work. See also C-07/C-27/C-29 (pipeline-core coupling symptoms), C-39 (the dead-mapper cleanup that precedes any unfao restructuring), **#45** (the delivery-side draw carrier — ship `(N, S)` uncollapsed as a native frame, the producer half of this same problem), and **epic #85** (the migration backlog). @@ -578,6 +619,62 @@ Cross-refs: **C-59** and **C-61** (RESOLVED — the invariant block this sits be --- + + +--- + +### C-79: `_ContractStorePort.upload`'s result check is called "the whole mechanism" and has no test, and it fails open + +| Field | Value | +|-------|-------| +| ID | C-79 | +| Tier | 3 — the check works today and is correct for what the store actually returns, so nothing is shipping wrong. What is missing is any assertion that it keeps working, plus a polarity that would swallow an unrecognised result rather than refuse it. | +| Source | `code-review max` (2026-08-03) — PR #211 fourth pass, while verifying the corrected comment beside it | +| Trigger | When views-pipeline-core changes what `DatastoreModule.upload_data` returns — a different result type, a renamed field, or a raise where it used to report — check this port still refuses a partial upload. The 3.0.0 bump (C-44) is the next occasion. | +| Owner | Whoever takes the pipeline-core 3.0.0 bump; it is the same reading of the same return contract. | +| Location | `_ContractStorePort.upload` in `views_postprocessing/unfao/managers/unfao.py` and `views_postprocessing/crafd/managers/crafd.py` (byte-identical in both) | + +The port exists because the store **reports** a metadata failure without raising: after the file is uploaded it logs, then returns `OperationResult(success=False, code="PARTIAL_SUCCESS")`. A caller that discards the result ships a file with no metadata document — invisible to the consumer, which is what happened to run-0's historical artifact on 2026-07-27. This check is what converts that into a refusal. + +**Two things are wrong with how it is held.** + +*It is untested.* `grep -rn _ContractStorePort tests/` returns exactly one hit, in a docstring in `tests/test_selection_guard.py` noting that the port is **not** asserted. So the code the comment beside it calls *"the whole mechanism"* is carried by no check at all — ADR-014 §1, in the file that this change edited to say so. + +*It fails open.* The refusal is `if success is False`, and `success` is resolved by `getattr(result, "success", None)` with a `to_dict()` fallback. A result object that is neither shape yields `None`, which is not `False`, so the upload is accepted. That is the wrong polarity for a repository whose ADR-003 forbids inferring what should be declared: an unrecognised result is exactly the case where refusing is cheap and guessing is not. The `to_dict()` branch is also dead on the real path — `OperationResult` has a `success` attribute — so it is untested code guarding an untested case. + +Neither is urgent, because `OperationResult.success` is typed `bool` and is never `None` today. Both become live the moment the return contract moves, which is precisely when nobody will be looking at this file. + +Cross-refs: **C-40** (the pipeline-core surface this port wraps), **C-44** (the 3.0.0 bump that is the named trigger), **C-77** (the other unguarded thing on the same delivery leg), ADR-014 §1, #211, #146. + +--- + +### C-77: The historical leg names its document from the model path, not from the declared consumer name — and nothing checks the two agree + +| Field | Value | +|-------|-------| +| ID | C-77 | +| Tier | 2 — structural fragility with a clear trigger, affecting **both** partners. Not Tier 1: the failure is a document the consumer cannot find, not a wrong value inside one. But it is the **F1 invisibility shape** — ADR-013 §4.1a, the defect that left six `orange_ensemble` forecast documents stranded in `unfao_bucket` while forecast serving read empty for months. Nobody notices a delivery that simply is not there. | +| Source | `code-review max` (2026-08-03) — PR #211, cross-checking the crafd producer against the views-crafdapi consumer | +| Trigger | When a postprocessor's directory is renamed in views-models, or a new partner package is added whose directory name differs from its `CONSUMER_DOCUMENT_NAME` — check that the historical artifact is still retrievable by the consumer's filter. The forecast leg will keep working, so a green delivery run is not evidence. | +| Owner | Whoever takes the guard. It is a one-line assertion plus a test, not a design decision — but it must be taken deliberately, because the current agreement is a coincidence nobody has written down. | +| Location | The historical-artifact upload in `views_postprocessing//managers/.py` — the call passing `name=self._model_path.model_name`, in `_save_contract`. For contrast, the correct leg is the `consumer_name=product.CONSUMER_DOCUMENT_NAME` argument a few lines above, which reaches the wire as `common["name"]` in `contract/wire/sink.py::deliver_run`. | + +The forecast leg is right. It threads the declared constant through: the manager passes `consumer_name=product.CONSUMER_DOCUMENT_NAME` into `deliver_run`, which sets `common = {"name": consumer_name, ...}`. One declaration, carried to the wire as a parameter — the shape C-69 credited as already correct. + +**The historical-actuals leg does not use that constant at all.** It passes `name=self._model_path.model_name` — a value that comes from the postprocessor's *directory name* in views-models, not from any declaration in this repository. The consumer filters on exactly the string this repo declares: `filters["name"] = self.model_path.model_name`, where the path manager is constructed as `APIPathManager("un_crafd")`. + +**For FAO the two agree; for CRAF'd nobody can yet say.** `views-models/postprocessors/` contains `un_fao` and nothing else — there is **no `un_crafd` postprocessor directory**, so CRAF'd's historical `name=` has never been resolved, let alone compared against its consumer's filter. That makes this worse rather than better: for the live partner the agreement is a coincidence nobody wrote down, and for the new one it is an assumption that will first be tested by a production run. Whoever creates that directory decides, without knowing it, whether CRAF'd's actuals are retrievable. + +**Nothing in this repository asserts they agree.** `tests/test_product.py` asserts `CONSUMER_DOCUMENT_NAME` for the forecast leg; `tests/test_hop_b_sink_e2e.py` checks `consumer_name` on the forecast leg. Neither touches the historical leg's `name=`. A rename of the views-models directory — an ordinary, plausible act, done in a different repository by someone who has never read this file — silently detaches the historical artifact from the consumer's filter while every test here stays green and every delivery run reports success. + +This is ADR-003's rule broken in the quiet direction: the delivery **infers** its consumer identity from a path instead of reading the declaration that exists three lines away. It is also the fourth home for partner identity, where C-69's 2026-07-31 note counted three and recommended consolidation rather than relocation. Consolidation did not reach this line. + +**Scope note:** the crafd package inherited this unchanged from `unfao`; PR #211 did not introduce it, it doubled it. Registering it against both partners rather than against the PR. + +Cross-refs: **C-01** (RESOLVED — the metadata-completeness gate; same partner, same delivery, different field), **C-69** (RESOLVED — "partner identity has THREE homes"; this is the fourth and the note's consolidation recommendation is the fix), **C-33** (the duplication that turned one instance into two), ADR-013 §4.1a (F1 invisibility), ADR-003 (declarations over inference), #211. + +--- + ## Disagreements ### D-12: Post-Run-0 infrastructure & naming intents — repo rename, internal-store transport, compute co-location @@ -646,6 +743,29 @@ See also C-40 (the inheritance/representation coupling this migration unwinds), ## Resolved Concerns +### C-78: A partner package without an `__init__.py` is invisible to the guard that inventories them — RESOLVED same day + +| Field | Value | +|-------|-------| +| ID | C-78 | +| Tier | 4 — FIXED in the same change that found it; recorded because the *reasoning* is what future guards need, not because work is outstanding. No delivery was ever affected. | +| Source | `code-review max` (2026-08-03) — PR #211 second pass, attacking the new scope guard | +| Trigger | When a future guard inventories the package tree, check what it uses as its "is this a package" test. If it asks for `__init__.py`, it disagrees with every other scan in this suite and with the repository's own root. | +| Owner | Discharged. | +| Location | `tests/test_clone_readiness.py` (the criterion); `views_postprocessing/` (which has no `__init__.py` of its own) | + +`test_the_declared_partner_list_is_the_real_one` was written to stop a partner package going unguarded — the defect that let `crafd/` land exempt from at least seven checks — the four `tests/conftest.py` enumerates, plus the three product pins (`TARGETS`, `S_MIN`, `UPLOAD_ENABLED`) that `tests/test_product.py` held for FAO alone. Its first draft asked for a directory containing `__init__.py`. + +**`views_postprocessing/` has no `__init__.py`.** The distribution root is already a PEP 420 namespace package, so the guard applied to its children a test its own parent fails. Verified by building a partner package without one, carrying three real defects — `UPLOAD_ENABLED = True` (ADR-013 §11.4), a wrong `CONSUMER_DOCUMENT_NAME` (§4.1a), and a live `load_dotenv` (þing-01 #134) — and running the full suite: **green**. Adding one empty `__init__.py` to the identical tree made the guard fire. It imported and ran fine at runtime throughout. + +The criterion is now "contains at least one `.py`, and is not `__pycache__`", which is what the suite's eight other tree scans effectively use (`rglob("*.py")`). The same review found `MACHINERY_PACKAGES` was validated against nothing — a stale name there **pre-classifies** any future package that takes it, and `reconciliation` (C-47's phantom) is exactly such a name. Both lists are now checked against disk. + +**One correction to C-47 while here.** That entry's Tier-4 rationale says the phantom directory was *"not importable (no `__init__.py`, no sources)"*. Under PEP 420 that reasoning is wrong: a directory with no `__init__.py` and no sources still imports as a **namespace package** whose `__path__` points at it — only its submodules fail. Reproduced on a copy of the tree. The directory itself was deleted by #177 on 2026-08-01, so nothing is importable today and the tier stands; what does not stand is the reason given for it. The harm C-47 actually recorded — *"misleading tools that inventory the tree"* — is precisely what this entry is about. + +Cross-refs: **C-47** (the phantom directory, and the corrected rationale above), **C-57** (a guard scoped by name missing the second subject — the same disease, one file over), **C-74** (a guard whose declared roots stopped existing), ADR-014 §2, #211. + +--- + ### C-22: No post-delivery correction process for wrong assignments — RESOLVED (procedure written; the partner-facing step is an open OPERATOR decision) | Field | Value | @@ -833,7 +953,7 @@ A second, smaller instance of the same shape: these checks parse TOML with `toml | Tier | 3 | | Source | `manual` (2026-07-31) — review-rr blind-spot analysis, following the þing-01 verdict (`orð_dómr.md`, ratified as amended 2026-07-28) | | Trigger | When views-appwrite amends `coordinate_registry.toml` — renames a coordinate, retires the legacy secret slot in favour of `APPWRITE_{READ,WRITE,PROVISION}_API_KEY`, or adds a target — verify `views_postprocessing/unfao/appwrite_env.py` still matches. Nothing mechanical will tell you: the registry is deliberately **referenced, never copied**, and the two live in different repositories | -| Location | `views_postprocessing/unfao/appwrite_env.py` (`CONNECTION_ENV`, `PROD_FORECASTS_ENV`, `UNFAO_ENV`); views-appwrite `docs/ADRs/platform/coordinate_registry.toml` (the authority); `tests/test_env_declaration.py` (guards this repo's half only); `docs/ADRs/013_sampled_forecast_wire_contract.md` §7(d) (the URL reference) | +| Location | `views_postprocessing/unfao/appwrite_env.py` (`CONNECTION_ENV`, `PROD_FORECASTS_ENV`, `UNFAO_ENV`) and `views_postprocessing/crafd/appwrite_env.py` (`CRAFD_ENV`) — see the 2026-08-03 amendment; views-appwrite `docs/ADRs/platform/coordinate_registry.toml` (the authority); `tests/test_env_declaration.py` (guards this repo's half only); `docs/ADRs/013_sampled_forecast_wire_contract.md` §7(d) (the URL reference) | The þing-01 assembly (D1) settled that the PLATFORM-001 contract is **homed in views-appwrite and referenced by URL, never by copy** — a deliberate and correct choice: copies were the platform's original disease (sáttmál S6, the copy-chain this repo's own `load_dotenv` borrow was the runtime edge of, killed in #134/PR #137). But referencing-not-copying moves the failure mode rather than removing it: **the registry can now change without this repo noticing.** @@ -845,6 +965,15 @@ Two named changes are already anticipated and will fire this trigger: the **reti **⚠ CORRECTED 2026-08-02, then restored the same day.** The word *already* above overclaimed at the time: **C-74** showed that guard scanning one of its five declared roots, four having pointed at paths #153 moved. **C-74 closed later that day (S10 / #192)** — the roots are re-pointed, the scan covers 17 files, and a declared root that does not exist now fails rather than emptying the scan silently. The sentence above is true again, and the episode is left visible because a claim that was false for two days is worth more as a record than as a correction quietly reverted. +**⚠ AMENDED 2026-08-03 (PR #211) — the detector was built for one partner, and the second partner proved it.** Two corrections to the resolution above, and one of them is the same disease in the cure. + +1. **The pin quoted above is stale.** `SEAM_CONTRACT_VERSION = "1.3.0"` / `SEAM_CONTRACT_COMMIT = "47172af"` was accurate when written on 2026-08-02; the registry then moved twice in under twelve hours — to v1.4.0 (`4a5ab1b`, reaching `main` as `20dfd0f`, 2026-08-02 18:45) and to v1.4.1 (`0da2682`, reaching `main` as `5266b90`, 2026-08-03 02:52) — and `unfao/appwrite_env.py` was re-pinned each time. This repo's current pin, `90fc105`, is **neither** of those commits: it is a later views-appwrite merge that does not touch the registry at all. That is correct and intended — a pin names *an edition of `main` that was read*, not the commit that changed the file — but the two must not be written as though they were the same thing. The values are left above as the worked example they were written to be, but they are no longer what the file says. +2. **The detector was scoped to `unfao` by name and did not follow the second partner.** `tests/test_env_declaration.py` imported only `views_postprocessing.unfao.appwrite_env`; a grep for `crafd` in it returned zero. So when PR #211 added `views_postprocessing/crafd/appwrite_env.py` pinned at **`1.3.0` / `47172af`** — an edition at which all four `APPWRITE_CRAFD_*` coordinates were declared with **no value**, and which predates the very views-appwrite PR #38 that the file's own docstring cites as its justification — **nothing failed.** Had this entry's own **version** check covered crafd, that pin would have failed **locally** the moment it was written. Not CI: the check opens with `require_sibling("views-appwrite")` and skips without a checkout, and the workflow checks out only this repository — which is this entry's own standing Residual, below. Note which half does the work: the *reachability* check would have passed, because `47172af` is a perfectly good ancestor of views-appwrite's `main`. Existence and reachability were both satisfied by a pin that was nonetheless two editions out of date — which is precisely why the version check exists alongside them rather than instead of them. + +This is ADR-014 §2 in its narrow form: a guard's *scope* is part of what has to be mutation-proven, not just its matching. The four checks were each proven to fail on the defect they were written for, against `unfao` — and stayed silent on an identical defect one package over. Same shape as **C-74**, one layer up: there the declared scan roots stopped existing; here the declared scope never grew. + +PR #211 re-pins crafd to `1.4.1` / `90fc105` and parameterises **three** of the four checks over both partner declarations — names-and-class, pinned edition, commit reachability. The fourth, the value-copy scan, was never partner-scoped: it walks `_PKG.rglob("*.py")` and so covered `crafd/` from the day it landed. A third partner is now a one-line addition, and an unguarded one is a failure. This entry stays RESOLVED — the mechanism was right, its reach was not — but the residual below now has a companion: a detector that names its subject is a detector that will miss the next subject. + Cross-refs: C-74 (the guard this paragraph vouched for), C-33 (store identity still hardcoded per store — the same env surface, different concern), C-58 (what happens when a coordinate is wrong rather than missing), C-44 (the pipeline-core version coupling that would carry a registry change), issues #134/#135/#138 (this repo's discharged þing-01 obligations), #104 (README env block placeholders). --- diff --git a/tests/conftest.py b/tests/conftest.py index dfee96c..78493ac 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -32,6 +32,34 @@ _REPO = Path(__file__).resolve().parent.parent +#: The partner packages under ``views_postprocessing/``, DECLARED (ADR-003) — and +#: declared **once**, because the alternative is what #211 exposed. +#: +#: **Eight** separate guards named ``"unfao"``, and when ``crafd/`` landed all eight +#: went on passing over a package they did not cover: +#: +#: the import-purity subprocess · the ADR-002 direction check · the manager line +#: budget · the coordinate-drift check · the þing-01 dotenv guard · and the three +#: product pins (``TARGETS``, ``S_MIN``, ``UPLOAD_ENABLED``) +#: +#: So ``contract/`` could import ``crafd``; the second manager had no budget; a registry +#: pin two editions stale failed nothing; the retired ``load_dotenv`` borrow could be +#: reintroduced in the new manager; and nothing asserted the new partner's consumer +#: document name — the one field whose failure mode is a delivery nobody can find. +#: +#: A partner list per guard is eight places to forget the next partner; this is one. +#: (An earlier version of this comment said four. It was written by counting the guards +#: that had already been fixed.) +#: +#: ``tests/test_clone_readiness.py::test_the_declared_partner_list_is_the_real_one`` +#: checks this against the filesystem, so the declaration cannot quietly go stale +#: either — ADR-014 §2: where a guard's inputs are declared, assert they are real. +PARTNER_PACKAGES = ("unfao", "crafd") + +#: The partner-neutral packages. The split is two-sided: a new top-level package is +#: either a partner or machinery, and the same test refuses to let it be neither. +MACHINERY_PACKAGES = ("contract", "delivery") + #: repo name -> the environment variable that overrides its location. #: Declared, never derived from the name: ``views-datafactory`` → ``VIEWS_DATAFACTORY`` #: happens to be mechanical, but a future sibling need not follow the pattern and @@ -39,6 +67,24 @@ SIBLING_ENV = { "views-datafactory": "VIEWS_DATAFACTORY", "views-appwrite": "VIEWS_APPWRITE", + "views-faoapi": "VIEWS_FAOAPI", + "views-crafdapi": "VIEWS_CRAFDAPI", +} + +#: partner package -> the repository that CONSUMES its delivery. +#: +#: Declared, never derived. ``unfao`` → ``views-faoapi`` and ``crafd`` → +#: ``views-crafdapi`` are not a pattern a rule could produce, and the þing records call +#: the second one ``un-crafdapi`` while the repository on disk is ``views-crafdapi`` — +#: exactly the kind of near-miss that makes guessing expensive. +#: +#: This exists so the consumer-document-name pin can be checked **across the seam** +#: rather than asserted locally. A name this repo declares and the consumer filters on +#: is a fact this repo does not own; declaring it here is right, but only the sibling +#: checkout can confirm it still matches (ADR-014 §1 — the guarantee needs a check). +CONSUMER_REPO = { + "unfao": "views-faoapi", + "crafd": "views-crafdapi", } @@ -61,6 +107,30 @@ def sibling_repo(name: str) -> Path | None: return candidate if candidate.exists() else None +def broken_sibling_overrides() -> dict[str, str]: + """Declared sibling variables that are SET but point at nothing. + + A typo'd override is an operator error, not a normal absence: someone set the + variable because they meant to run those checks, and returning ``None`` turns the + typo into permanent, invisible non-coverage of every cross-repo assertion this repo + has. ``test_no_sibling_override_points_at_a_missing_path`` fails on it. + + **This is deliberately not a raise inside ``sibling_repo``.** It was, for about an + hour. Three modules resolve a sibling at import time (``test_delivery_coverage``, + ``test_datafactory_deploy_readiness``, ``test_gaul_lookup_fidelity``), so raising + there turned a one-character typo in ``VIEWS_DATAFACTORY`` into + ``Interrupted: 3 errors during collection`` and **zero tests run** — trading silent + under-coverage for total loss of the suite. One clean failure says the same thing + and lets the other 360 tests report. + """ + return { + var: value + for var in SIBLING_ENV.values() + for value in [os.environ.get(var)] + if value and not Path(value).exists() + } + + def require_sibling(name: str) -> Path: """``sibling_repo`` or skip, with a message naming what to set. diff --git a/tests/test_clone_readiness.py b/tests/test_clone_readiness.py index 27f2de7..549bb77 100644 --- a/tests/test_clone_readiness.py +++ b/tests/test_clone_readiness.py @@ -14,9 +14,16 @@ **Why it exists at all.** #153 separated partner-neutral machinery (``contract/``) from FAO's product (``unfao/``). Nothing then stops the next contributor adding one convenient import back — and the boundary would be gone with no signal, because -everything would still work *for FAO*. It would only break for views-crafdapi and -views-productionapi, in a repository nobody had cut yet. Register C-69's fix is -one-shot; this is what makes it hold. +everything would still work *for FAO*. Register C-69's fix is one-shot; this is what +makes it hold. + +**Amended 2026-08-03 (#211): there are now two partners, and the guard was scoped to +one by name.** ``crafd/`` joined ``unfao/`` and every assertion here hardcoded the +string ``"views_postprocessing.unfao"``, so ``contract/`` could import ``crafd`` and +nothing objected — verified by adding exactly that import and watching the suite stay +green. The partners are now **declared** in ``_PARTNER_PACKAGES`` and checked against +the filesystem, because a guard that names its subject will miss the next subject +(register C-57's 2026-08-03 amendment, which is the same failure one file over). """ from __future__ import annotations @@ -28,7 +35,14 @@ import pytest +# The partner and machinery package lists are declared once, in tests/conftest.py, and +# imported by every guard that needs them. Four guards each carrying their own copy of +# the string "unfao" is how `crafd` arrived unguarded — see that module's comment. +from tests.conftest import MACHINERY_PACKAGES as _MACHINERY_PACKAGES +from tests.conftest import PARTNER_PACKAGES as _PARTNER_PACKAGES + _REPO = Path(__file__).resolve().parent.parent +_PKG = _REPO / "views_postprocessing" #: Every partner-neutral module a clone is expected to reuse as-is. _MACHINERY = ( @@ -50,19 +64,25 @@ "views_postprocessing.contract.historical", "views_postprocessing.contract.gaul_lookup", "views_postprocessing.contract.gaul_schema", + "views_postprocessing.contract.enrichment", "views_postprocessing.contract.launch_config", "views_postprocessing.contract.source_metadata", "views_postprocessing.contract.store_metadata", ) -def _import_in_subprocess(modules: tuple[str, ...], forbidden_prefix: str) -> subprocess.CompletedProcess: - """Import ``modules`` in a fresh interpreter; report any ``forbidden_prefix`` arrivals.""" +def _import_in_subprocess( + modules: tuple[str, ...], forbidden_prefixes: tuple[str, ...] +) -> subprocess.CompletedProcess: + """Import ``modules`` in a fresh interpreter; report any ``forbidden_prefixes`` arrivals.""" script = textwrap.dedent(f""" import sys for name in {list(modules)!r}: __import__(name) - leaked = sorted(m for m in sys.modules if m.startswith({forbidden_prefix!r})) + leaked = sorted( + m for m in sys.modules + if any(m.startswith(p) for p in {list(forbidden_prefixes)!r}) + ) print("LEAKED:" + ",".join(leaked)) """) return subprocess.run( @@ -71,59 +91,204 @@ def _import_in_subprocess(modules: tuple[str, ...], forbidden_prefix: str) -> su ) -def test_the_machinery_imports_without_the_partner(): +#: The subset of the above that is the bottom of the stack. ADR-002's chain is +#: ``/ -> contract/ -> delivery/``, and the arrow points one way: the +#: invariants must be usable without the machinery that calls them. +_INVARIANTS = tuple( + m for m in _MACHINERY if m.startswith("views_postprocessing.delivery.") +) + + +def _partner_prefixes() -> tuple[str, ...]: + return tuple(f"views_postprocessing.{name}" for name in _PARTNER_PACKAGES) + + +def _modules_on_disk(package: str) -> set[str]: + return { + "views_postprocessing." + f.relative_to(_PKG).with_suffix("").as_posix().replace("/", ".") + for f in (_PKG / package).rglob("*.py") + if f.name != "__init__.py" + } + + +def test_the_machinery_list_is_the_whole_machinery(): + """Assert this guard's inputs are real — the clause it kept applying elsewhere. + + ``_MACHINERY`` is a hand-written tuple, and every purity check here iterates it. A + machinery module missing from it is never imported in the subprocess, so it can + depend on a partner and nothing objects. + + **That was not hypothetical.** ``contract/enrichment.py`` was absent, and nothing + else in the package imports it, so it was never dragged in transitively either: + verified 2026-08-03 that ``from ..crafd import product`` in that file left the whole + suite green. This changeset added a completeness assertion to five other declared + lists (ADR-014 §2) and missed the one in the file that argues for them. + """ + declared = set(_MACHINERY) + # Derived from the declared machinery list, not from two names written here. A + # third machinery package added to MACHINERY_PACKAGES and forgotten here would + # otherwise be exempt from every purity check — which is the defect this whole + # changeset moved four other lists into conftest.py to prevent, reproduced inside + # the test written to enforce it. + on_disk = set().union(*(_modules_on_disk(p) for p in _MACHINERY_PACKAGES)) + + assert not (on_disk - declared), ( + f"machinery modules absent from _MACHINERY: {sorted(on_disk - declared)}. Each " + "is exempt from every purity check in this file — it may import a partner, and " + "nothing here will notice." + ) + assert not (declared - on_disk), ( + f"_MACHINERY names modules that no longer exist: {sorted(declared - on_disk)}. " + "A subprocess that imports a vanished module fails confusingly; one that was " + "quietly dropped from the list stops being checked." + ) + + +def test_the_invariants_import_without_the_machinery(): + """ADR-002's lower arrow, proven the way the upper one is. + + ``delivery/`` sits below ``contract/`` and must stay usable on its own — that is + what makes the invariants *representation-free* rather than merely + representation-light, and it is what lets a partner reuse them without the wire. + + **A regex was written for this first and was not enough.** It caught + ``from views_postprocessing.contract import gaul_schema`` — the one form used to + demonstrate the gap — while missing ``from ..contract import gaul_schema`` and + ``from views_postprocessing import contract``, both plain module-level imports that + execute on import. A fresh interpreter sees all of them, which is why this is the + load-bearing half and the regex in ``test_doc_accuracy.py`` is the supplement that + also covers imports hidden in function bodies. + """ + result = _import_in_subprocess( + _INVARIANTS, ("views_postprocessing.contract",) + _partner_prefixes() + ) + assert result.returncode == 0, ( + f"the invariants failed to import on their own:\n{result.stderr}" + ) + leaked = [m for m in result.stdout.split("LEAKED:")[-1].strip().split(",") if m] + assert not leaked, ( + f"importing delivery/ pulled in the machinery above it: {leaked}. ADR-002's " + "dependency arrow points one way; an invariant that needs the wire to load is " + "no longer an invariant about primitives." + ) + + +def test_the_declared_partner_list_is_the_real_one(): + """Assert the guard's inputs are real (ADR-014 §2). + + Every other test here trusts ``_PARTNER_PACKAGES``. A partner package that exists + on disk but is missing from that tuple is unguarded, and the suite stays green — + which is exactly what happened to ``crafd`` on the day it landed. So the tuple is + checked against the filesystem rather than believed. + + A new top-level package under ``views_postprocessing/`` is either a partner or + machinery. If it is neither, this fails and someone decides which it is, on + purpose, rather than by whichever guard happens not to mention it. + + **Why "contains a .py" and not "contains an ``__init__.py``".** The first draft + asked for ``__init__.py`` — a criterion ``views_postprocessing/`` itself fails, since + the distribution root has no ``__init__.py`` and is already a PEP 420 namespace + package. A partner directory without one imports perfectly at runtime and was + invisible here: verified 2026-08-03 by building a namespace partner carrying three + real defects (``UPLOAD_ENABLED = True``, a wrong consumer name, a live + ``load_dotenv``) and watching the full suite stay green, with one empty + ``__init__.py`` the entire difference. Every other tree scan in this suite uses + ``rglob("*.py")`` and would have seen it; this guard was the odd one out. + + **And ``rglob`` is why, not ``glob``.** The first correction used ``glob("*.py")`` + while its own comment claimed parity with the ``rglob`` scans — so a partner whose + modules sit only in ``managers/`` (which is where a partner's manager actually + lives) was *still* invisible. Verified: ``wfp/managers/wfp.py`` carrying a live + ``load_dotenv`` passed this guard. Two drafts, the same mistake, caught the second + time only because someone checked the sentence against the code. + """ + on_disk = { + p.name for p in _PKG.iterdir() + if p.is_dir() and p.name != "__pycache__" and any(p.rglob("*.py")) + } + classified = set(_PARTNER_PACKAGES) | set(_MACHINERY_PACKAGES) + + for label, declared in ( + ("_PARTNER_PACKAGES", _PARTNER_PACKAGES), + ("_MACHINERY_PACKAGES", _MACHINERY_PACKAGES), + ): + missing = sorted(p for p in declared if p not in on_disk) + assert not missing, ( + f"{label} names packages that do not exist: {missing}. A guard whose " + "declared scope has gone missing scans nothing and reports success — and a " + "stale name here is worse than dead, because it pre-classifies any future " + "package that happens to take it (register C-47: `reconciliation` was " + "exactly such a phantom in this tree)." + ) + + unclassified = sorted(on_disk - classified) + assert not unclassified, ( + f"new top-level package(s) {unclassified} are neither declared partners nor " + "declared machinery. Add each to PARTNER_PACKAGES or MACHINERY_PACKAGES in " + "tests/conftest.py — a partner left out of the first is silently exempt from " + "every check below." + ) + + +def test_the_machinery_imports_without_any_partner(): """The load-bearing assertion of the whole epic. A clone must be able to take `delivery/` and `contract/` and get a working - ADR-013 delivery without inheriting FAO's product, FAO's store coordinates, or - FAO's manager. + ADR-013 delivery without inheriting any partner's product, store coordinates, or + manager. "Any" is the operative word since #211: the machinery serves two + partners now, and being neutral toward one of them is not neutrality. """ - result = _import_in_subprocess(_MACHINERY, "views_postprocessing.unfao") + result = _import_in_subprocess(_MACHINERY, _partner_prefixes()) assert result.returncode == 0, ( f"the machinery failed to import on its own:\n{result.stderr}" ) leaked = [m for m in result.stdout.split("LEAKED:")[-1].strip().split(",") if m] assert not leaked, ( - f"importing the partner-neutral machinery pulled in the partner: {leaked}. " - "A clone (views-crafdapi, views-productionapi) would inherit FAO's product " - "through this path — see register C-69 and views_postprocessing/contract/__init__.py." + f"importing the partner-neutral machinery pulled in a partner: {leaked}. " + "The next partner would inherit that one's product through this path — see " + "register C-69 and views_postprocessing/contract/__init__.py." ) def test_the_machinery_does_not_pull_in_pipeline_core(): - """`views_pipeline_core` is imported by exactly one module — the manager. + """`views_pipeline_core` is imported only by the partner managers. - Pinned while it is true. A clone writes its own manager against its own + Pinned while it is true. Each partner's manager is written against its own framework seam; if the machinery started dragging pipeline-core in, that choice - would be made for it, and register C-40's blast radius would widen from one file - to the whole package. + would be made for every partner at once, and register C-40's blast radius would + widen from two files to the whole package. - ADR-013 §11.4-adjacent: þing-02 **S24(5)** makes this binding for the clone — it - must not import `views_pipeline_core.modules.{appwrite,datastore}`, because this - repo's own import of those is how a two-repo defect became three. + ADR-013 §11.4-adjacent: þing-02 **S24(5)** binds *the cloned repositories* — + `un-crafdapi` and `views-productionapi`, cut from views-faoapi — not to import + `views_pipeline_core.modules.{appwrite,datastore}`. It does not reach an in-repo + partner package of the producer, which is why `crafd/managers/crafd.py` may import + them and `docs/CLONING.md` was corrected. What this test defends is the narrower + and repo-owned rule: the *machinery* stays free of them regardless. """ - result = _import_in_subprocess(_MACHINERY, "views_pipeline_core") + result = _import_in_subprocess(_MACHINERY, ("views_pipeline_core",)) assert result.returncode == 0, result.stderr leaked = [m for m in result.stdout.split("LEAKED:")[-1].strip().split(",") if m] assert not leaked, ( - f"the machinery pulled in views_pipeline_core: {leaked}. It is imported by " - "exactly one module (the manager) and that is what keeps C-40 bounded." + f"the machinery pulled in views_pipeline_core: {leaked}. It is imported only " + "by the partner managers, and that is what keeps C-40 bounded." ) -def test_the_guard_would_actually_catch_a_violation(): +@pytest.mark.parametrize("partner", _PARTNER_PACKAGES) +def test_the_guard_would_actually_catch_a_violation(partner): """A purity test that cannot fail is decoration. - Imports the *partner* deliberately and asserts the detector sees it — so a future - reader knows the two tests above are load-bearing rather than vacuously passing - because the subprocess silently did nothing. + Imports each *partner* deliberately and asserts the detector sees it — so a + future reader knows the tests above are load-bearing rather than vacuously + passing because the subprocess silently did nothing. Parametrised, because a + detector proven against one partner is not proven against the other. """ result = _import_in_subprocess( - ("views_postprocessing.unfao.product",), "views_postprocessing.unfao" + (f"views_postprocessing.{partner}.product",), _partner_prefixes() ) assert result.returncode == 0, result.stderr leaked = [m for m in result.stdout.split("LEAKED:")[-1].strip().split(",") if m] - assert leaked, "the detector reported nothing while importing the partner directly" + assert leaked, f"the detector reported nothing while importing {partner} directly" @pytest.mark.parametrize("doc", ["docs/CLONING.md"]) diff --git a/tests/test_datafactory_deploy_readiness.py b/tests/test_datafactory_deploy_readiness.py index f8ffa8d..acadef8 100644 --- a/tests/test_datafactory_deploy_readiness.py +++ b/tests/test_datafactory_deploy_readiness.py @@ -3,9 +3,13 @@ These guard the cross-repo preconditions that the FAO global-delivery plan (umbrella views-postprocessing#20, region flip views-models#127) depends on. -They are written against a local views-datafactory checkout and FAIL BY DESIGN -until the datafactory deploy candidate is actually releasable and the served -artifact matches the branch. +They are written against a local views-datafactory checkout. + +**"FAIL BY DESIGN" no longer describes this module as a whole, and saying so was +misleading.** The release-gate half is satisfied: the land_gaul work reached a tag, and +that is asserted below. What still fails by design is the *served-artifact* half — the +two ``xfail(strict)`` classes, which flip to failures the moment the served artifact +catches up with the branch. Corrected 2026-08-03. Resolution is the repo's one declared way of finding a sibling checkout (``tests/conftest.sibling_repo``): ``$VIEWS_DATAFACTORY``, else the conventional @@ -40,38 +44,40 @@ def _git(*args: str) -> str: class TestReleaseGate: - """P1 (HARD): the deploy gate (ADR-022) checks out an exact git TAG on the - server. land_gaul / Azores / SHDI are all committed AFTER the latest tag - (v1.2.29) and the version in pyproject is unchanged. Deploying 'to serve' - the current state would serve v1.2.29 — which has none of this work — and - views-models#127's REGION='land_gaul' flip would hit a package without it. + """P1: the deploy gate (ADR-022) checks out an exact git TAG on the server, so + anything this repository depends on must be *inside a tag* — not merely committed. + + **The blocker this class was written for is discharged.** When it was written, + land_gaul / Azores / SHDI were all committed after the latest tag (v1.2.29) with + pyproject unchanged, so deploying "to serve" would have served a package without + land_gaul and views-models#127's ``REGION='land_gaul'`` flip would have hit it. + That is no longer the situation: ``bac163e`` is contained in v1.10.0 and v1.11.0, + and datafactory's latest tag is v1.11.0. + + What survives is the standing check — the commit this repository depends on is in a + release tag — which is the durable form of the same question and does not care what + anyone's version number says today. The prose above described a live cross-repo + blocker for a while after it stopped being one; corrected 2026-08-03. """ - # #224 resolved (v1.4.0), then datafactory TAGGED v1.4.0 (2026-06-24) — so - # pyproject version == latest tag, which this over-strict check reads as a - # "stale version". Re-xfail(strict): the gate flaps with datafactory's release - # cadence; it flips back to a failure (demanding promotion) only when - # datafactory bumps past the tag to the next -dev. - @pytest.mark.xfail( - reason="cross-repo deploy gate flaps with datafactory's release cadence: " - "v1.4.0 is now tagged and == pyproject version; xfail(strict) " - "re-promotes when datafactory bumps past the tag.", - strict=True, - ) - def test_version_bumped_past_latest_tag(self): - version_line = (_DF / "pyproject.toml").read_text() - current = next( - ln.split("=")[1].strip().strip('"') - for ln in version_line.splitlines() - if ln.startswith("version") - ) - tags = _git("tag", "-l").splitlines() - assert f"v{current}" not in tags, ( - f"pyproject version {current} is already tagged (v{current}). The " - f"land_gaul/Azores/SHDI work is committed but UNTAGGED; the tag-based " - f"deploy gate would serve v{current}, which lacks land_gaul. Bump the " - f"version and cut a release before merging to main / deploying." - ) + # **`test_version_bumped_past_latest_tag` was RETIRED here on 2026-08-03.** + # + # It asserted `f"v{current}" not in tags` — that datafactory's pyproject version is + # ahead of its latest tag. That is true only in the window between a version bump + # and the tag that follows it, so the check went green or red on another + # repository's release *timing*, not on anything this repository depends on. It had + # been xfail(strict) for that reason, with a written re-promotion trigger; the + # trigger fired (datafactory reached v1.10.0 tagged / 1.11.0 in pyproject), the + # marker turned the unexpected pass into a failure, and re-reading it showed the + # test was the problem rather than the marker. datafactory's HEAD is literally + # "chore: bump to 1.11.0": the moment it cuts v1.11.0 this would have gone red + # again, in a repository doing everything right. + # + # Nothing is lost. Its stated purpose — "the land_gaul work is committed but + # UNTAGGED, so the tag-based deploy gate would serve a release without it" — is + # exactly what the surviving test below asserts, directly and without reference to + # anyone's release cadence. A gate that flaps gets ignored, and then the thing it + # guarded is unguarded (ADR-014 §3). def test_land_gaul_commit_is_in_a_release_tag(self): # bac163e = "feat: add bundled curated region land_gaul" diff --git a/tests/test_doc_accuracy.py b/tests/test_doc_accuracy.py index 869daca..4a29779 100644 --- a/tests/test_doc_accuracy.py +++ b/tests/test_doc_accuracy.py @@ -20,6 +20,10 @@ import re from pathlib import Path +import pytest + +from tests.conftest import PARTNER_PACKAGES as _PARTNER_PACKAGES + _REPO = Path(__file__).resolve().parent.parent _PKG = _REPO / "views_postprocessing" @@ -136,8 +140,19 @@ def test_internal_doc_links_resolve(): # module" while pandas lived in three. Both drifted the same way and neither was # caught by reading. These make the claims fail CI instead. -_MANAGER = _PKG / "unfao" / "managers" / "unfao.py" -_MANAGER_LINE_BUDGET = 450 # epic #148's bound; 406 at close, 636 before #149 +#: Every partner's manager directory, derived from the one declared partner list. +#: +#: **This was a hardcoded 2-tuple until 2026-08-03 and it was the last such list in the +#: suite.** Fifteen guards parametrise over ``PARTNER_PACKAGES``; this one did not, so a +#: third partner with a **906-line** manager passed the budget while failing everything +#: else — and nothing in those failures named the list it was missing from. That is +#: register C-57's shape surviving inside the file that documents C-57's fix. +_MANAGER_DIRS = tuple((_PKG / p / "managers") for p in _PARTNER_PACKAGES) + +#: epic #148's bound; 406 at close, 636 before #149. Applied to the manager *directory*, +#: not the manager file: an 800-line helper module beside a 406-line manager was +#: previously unbudgeted, which is the same regrowth wearing a different filename. +_MANAGER_LINE_BUDGET = 450 def _is_type_checking(test: ast.expr) -> bool: @@ -217,7 +232,7 @@ def test_pandas_is_not_imported_at_runtime_anywhere_in_the_package(): ) -def test_views_pipeline_core_has_exactly_one_importer(): +def test_views_pipeline_core_is_confined_to_the_partner_managers(): """ADR-012's 'Pipeline Manager' row, and C-40's blast-radius claim. This property is what makes the C-40 de-inheritance a bounded job rather than an @@ -236,30 +251,121 @@ def test_views_pipeline_core_has_exactly_one_importer(): ) -def test_the_manager_stays_within_its_line_budget(): - """ADR-012 no longer calls the manager 'thin' — it states a number. Hold it.""" - lines = len(_MANAGER.read_text().splitlines()) +@pytest.mark.parametrize("managers_dir", _MANAGER_DIRS, ids=lambda p: p.parent.name) +def test_the_manager_stays_within_its_line_budget(managers_dir): + """ADR-012 no longer calls the manager 'thin' — it states a number. Hold it. + + Counts every ``.py`` under the partner's ``managers/`` directory. The bound is on + the *seam*, and a seam that stays at 406 lines by moving 800 into a sibling module + has not stayed anywhere. + """ + assert managers_dir.is_dir(), ( + f"no managers/ directory at {managers_dir.relative_to(_PKG.parent)}. A budget " + "over a path that stopped existing counts nothing and reports success." + ) + sources = sorted(managers_dir.rglob("*.py")) + lines = sum(len(f.read_text().splitlines()) for f in sources) assert lines <= _MANAGER_LINE_BUDGET, ( - f"the manager is {lines} lines, over epic #148's {_MANAGER_LINE_BUDGET} bound. " + f"{managers_dir.parent.name}'s managers/ is {lines} lines across " + f"{[f.name for f in sources]}, over epic #148's {_MANAGER_LINE_BUDGET} bound. " "It was 636 before #149 and is the repo's one known dumping ground — growth " "here is the regression that epic existed to reverse." ) -def test_contract_package_does_not_import_the_partner(): - """The dependency direction ADR-002 declares: unfao/ -> contract/ -> delivery/.""" +def _imported_subpackages(source: str, module_path: Path) -> set[str]: + """Every ``views_postprocessing.`` this module imports — all four spellings. + + **Regexes were tried twice here and escaped twice.** A pattern that matched + ``from views_postprocessing.contract import x`` missed ``from ..contract import x``; + widened for that, it still missed ``from views_postprocessing import contract``, + because the optional ``views_postprocessing.`` group requires the dot. Both are + ordinary module-level imports. A third spelling, ``import views_postprocessing.x``, + needed its own alternative. Meanwhile the pattern fired on a docstring that merely + *spelled* a forbidden import — cry-wolf on prose while missing real code, which is + the worst of both (ADR-014 §2, §3). + + The AST knows what an import is. It resolves relative levels, sees the bare + ``from package import subpackage`` form as what it is, and cannot see prose at all. + This module already had the pattern twice (``_classify_pandas_imports``, + and ``_dotenv_use`` in ``test_env_declaration.py``); the import guards simply had + not caught up. + """ + parts = module_path.relative_to(_PKG).with_suffix("").as_posix().split("/") + package = ["views_postprocessing"] + parts[:-1] # the module's own package + + found: set[str] = set() + + def record(dotted: str) -> None: + bits = dotted.split(".") + if len(bits) >= 2 and bits[0] == "views_postprocessing": + found.add(bits[1]) + + for node in ast.walk(ast.parse(source)): + if isinstance(node, ast.Import): + for alias in node.names: + record(alias.name) + elif isinstance(node, ast.ImportFrom): + if node.level: + base = package[: len(package) - (node.level - 1)] + resolved = ".".join(base + ([node.module] if node.module else [])) + else: + resolved = node.module or "" + record(resolved) + # `from import ` — the imported NAME is the subpackage. + for alias in node.names: + record(f"{resolved}.{alias.name}") + return found + + +def test_the_invariants_do_not_import_the_machinery(): + """The lower leg of ADR-002's chain: ``/ -> contract/ -> delivery/``. + + ``delivery/`` is the bottom of the stack and must stay usable on its own — that is + what makes the invariants *representation-free* rather than merely + representation-light. Its module-level counterpart is + ``test_clone_readiness.py::test_the_invariants_import_without_the_machinery``, which + proves the same thing in a fresh interpreter; this one also sees imports hidden + inside function bodies, which a subprocess never executes. + + **Written 2026-08-03 because the claim existed without it.** ADR-012 said the + one-way dependency was "enforced by test, not convention"; only the + partner-to-machinery leg was. + """ + forbidden = {"contract", *_PARTNER_PACKAGES} offenders = [ - f.relative_to(_PKG).as_posix() - for f in (_PKG / "contract").rglob("*.py") - if re.search(r"^\s*(?:from|import)\s+views_postprocessing\.unfao", f.read_text(), re.M) + f"{f.relative_to(_PKG).as_posix()} -> views_postprocessing.{name}" + for f in (_PKG / "delivery").rglob("*.py") + for name in sorted(_imported_subpackages(f.read_text(), f) & forbidden) ] assert not offenders, ( - f"contract/ imports the partner package: {offenders}. The machinery must be " - "reusable by views-crafdapi / views-productionapi without taking FAO (C-69)." + f"delivery/ imports upward: {offenders}. The invariants sit below the machinery " + "and must stay usable without it — that is what makes them representation-free " + "rather than merely representation-light (ADR-002, ADR-012)." ) -# --- 3. retired cross-repo contract name (S3 / #184, finishing #158) -------------------- +def test_contract_package_does_not_import_any_partner(): + """The upper leg of ADR-002's chain, and register C-69's fix made permanent. + + Scoped to the declared partner list rather than to ``unfao`` by name: the + single-name version passed while ``contract/`` was free to import ``crafd``, proven + by adding that import and watching the suite stay green. + + Uses the same AST walk as its sibling above, for the same reason — the regex here + was absolute-only, so ``from ..unfao import product`` inside a function body was + invisible to it *and* to the subprocess check, which never executes a function body. + """ + offenders = [ + f"{f.relative_to(_PKG).as_posix()} -> views_postprocessing.{name}" + for f in (_PKG / "contract").rglob("*.py") + for name in sorted(_imported_subpackages(f.read_text(), f) & set(_PARTNER_PACKAGES)) + ] + assert not offenders, ( + f"contract/ imports a partner package: {offenders}. The machinery must be " + "reusable by the next partner without taking this one's product (C-69)." + ) + #: Retired 2026-07-31. The registry itself records the retirement: #: ``former_contract_name = "PLATFORM-001" # retired 2026-07-31``. diff --git a/tests/test_env_declaration.py b/tests/test_env_declaration.py index 85bae19..ad34e8e 100644 --- a/tests/test_env_declaration.py +++ b/tests/test_env_declaration.py @@ -7,11 +7,20 @@ verdict retired it — the launcher declares its env sourcing (views-models M3, merged), and this package only validates (verdict D6). -Imports `unfao.appwrite_env` and its sibling `contract.launch_config` — both are -dependency-light by design (no views-pipeline-core, no pandas), which is what -lets them be exercised directly here. The manager module is not: it needs -views-pipeline-core, absent in test environments, so the manager-side facts are -pinned by source scan instead — the repo's standing pattern. +Imports **every partner's** `appwrite_env` and their shared sibling +`contract.launch_config` — all dependency-light by design (no views-pipeline-core, no +pandas), which is what lets them be exercised directly here. The manager modules are +not exercised directly: instantiating one needs a full Appwrite environment and a +views-models path manager, so the manager-side facts are pinned by source scan instead — +the repo's standing pattern. (The older reason given here, that views-pipeline-core is +"absent in test environments", stopped being true: it is a declared dependency, +CI installs it, and the manager module imports fine.) + +**Scoped to `unfao` by name until #211**, at which point `crafd/appwrite_env.py` +landed pinned to a registry edition at which its own coordinates had no values, and +every check here went on passing over a module it was not looking at. The partner list +now comes from `tests/conftest.PARTNER_PACKAGES` and is asserted against the +filesystem, so the next partner cannot be silently exempt (register C-57). """ import ast @@ -21,30 +30,165 @@ import pytest -from tests.conftest import commit_is_on_main, git_output, require_sibling, sibling_repo +from tests.conftest import ( + PARTNER_PACKAGES, + broken_sibling_overrides, + commit_is_on_main, + git_output, + require_sibling, + sibling_repo, +) from views_postprocessing.contract import launch_config +from views_postprocessing.crafd import appwrite_env as crafd_env from views_postprocessing.unfao import appwrite_env _PKG = Path(__file__).resolve().parent.parent / "views_postprocessing" -_MANAGER_SOURCE = ( - Path(__file__).resolve().parent.parent - / "views_postprocessing" - / "unfao" - / "managers" - / "unfao.py" -) +def _manager_source(partner: str) -> Path: + """``/managers/.py``, asserted to exist. + + The layout is **declared** — ``docs/CLONING.md`` §3 states it as the shape a new + partner supplies — but a declaration this function merely assumes is one it cannot + notice going stale. Without the check below a renamed manager gives a bare + ``FileNotFoundError`` from ``read_text``, which is loud but says nothing about + which rule was broken (ADR-014 §2: assert that a guard's inputs are real). + """ + path = _PKG / partner / "managers" / f"{partner}.py" + assert path.exists(), ( + f"no manager at {path.relative_to(_PKG.parent)}. Every check in this file that " + f"scans {partner}'s manager silently covers nothing without it. The layout is " + "declared in docs/CLONING.md §3 — either follow it or teach this function the " + "new one; do not leave the scan pointing at a path that stopped existing." + ) + return path + + +_SHARED_CLASS = { + "APPWRITE_ENDPOINT": "connection", + "APPWRITE_DATASTORE_PROJECT_ID": "connection", + "APPWRITE_DATASTORE_API_KEY": "secret", + "APPWRITE_PROD_FORECASTS_BUCKET_ID": "target", + "APPWRITE_PROD_FORECASTS_BUCKET_NAME": "target", + "APPWRITE_PROD_FORECASTS_COLLECTION_ID": "target", + "APPWRITE_PROD_FORECASTS_COLLECTION_NAME": "target", + "APPWRITE_METADATA_DATABASE_ID": "target", + "APPWRITE_METADATA_DATABASE_NAME": "target", +} + +#: Per partner: the module, the tuple naming its own outbound store, and how this test +#: treats every name that partner declares. +#: +#: **Parametrised since #211, and that is the point.** Every check below imported +#: ``unfao.appwrite_env`` by name. When ``crafd/appwrite_env.py`` landed pinned two +#: registry editions stale — at a commit where its own four ``APPWRITE_CRAFD_*`` +#: coordinates still carried no values — not one of them fired, because none of them +#: was looking. A drift detector that names its subject cannot detect drift in the +#: subject it does not name (register C-57, amended 2026-08-03). +_PARTNER_ENV = { + "unfao": ( + appwrite_env, + appwrite_env.UNFAO_ENV, + _SHARED_CLASS | { + "APPWRITE_UNFAO_BUCKET_ID": "target", + "APPWRITE_UNFAO_BUCKET_NAME": "target", + "APPWRITE_UNFAO_COLLECTION_ID": "target", + "APPWRITE_UNFAO_COLLECTION_NAME": "target", + }, + ), + "crafd": ( + crafd_env, + crafd_env.CRAFD_ENV, + _SHARED_CLASS | { + "APPWRITE_CRAFD_BUCKET_ID": "target", + "APPWRITE_CRAFD_BUCKET_NAME": "target", + "APPWRITE_CRAFD_COLLECTION_ID": "target", + "APPWRITE_CRAFD_COLLECTION_NAME": "target", + }, + ), +} + +_PARTNERS = tuple(_PARTNER_ENV) + + +#: Function names that belong to python-dotenv and to essentially nothing else. +#: +#: ``set_key``/``get_key``/``unset_key`` are deliberately **absent**. They are dotenv +#: members, but they are also ordinary method names — a first draft included ``set_key`` +#: and flagged ``draft.set_key("run_id", run_id)`` in ``contract/store_metadata.py`` as +#: *"python-dotenv is back in this package"*. That is ADR-014 §3: a guard that cries +#: wolf gets deleted, and then the rule it carried is unguarded. Reaching them requires +#: importing ``dotenv``, which the import half below catches outright. +_DOTENV_CALLS = {"load_dotenv", "find_dotenv", "dotenv_values"} + + +def _dotenv_use(source: str) -> list[str]: + """Real imports of, and calls into, python-dotenv — parsed, not grepped. + + **Prose is not a violation and must not be treated as one.** ``appwrite_env.py``'s + own docstring explains what the retired borrow was, spelling it exactly; a token + scan over the package flags that sentence and gets deleted, after which the rule it + carried is unguarded (ADR-014 §3 — when a guard cries wolf, the matching is wrong + before the scope is). An AST walk sees imports and calls and never sees a docstring + or a comment, so the guard can cover the whole package without lying about prose. + """ + found = [] + for node in ast.walk(ast.parse(source)): + if isinstance(node, ast.Import): + found += [a.name for a in node.names if a.name.split(".")[0] == "dotenv"] + elif isinstance(node, ast.ImportFrom): + if (node.module or "").split(".")[0] == "dotenv": + found.append(f"from {node.module}") + elif isinstance(node, ast.Call): + fn = node.func + name = fn.id if isinstance(fn, ast.Name) else getattr(fn, "attr", None) + if name in _DOTENV_CALLS: + found.append(f"{name}()") + return sorted(set(found)) + + +def test_no_sibling_override_points_at_a_missing_path(): + """Assert the cross-repo checks' inputs are real (ADR-014 §2), once for all of them. + + Every gated check in this repository resolves a sibling checkout and skips when it + is absent. Absent is normal — CI checks out only this repo. A variable that is + *set* and wrong is not normal, and skipping on it means an operator who asked for + the cross-repo assertions silently got none of them, for as long as the typo lives. + """ + broken = broken_sibling_overrides() + assert not broken, ( + f"sibling override(s) set but pointing at nothing: {broken}. Every check gated " + "on those repositories is skipping — you asked for them and are getting none. " + "Fix the path or unset the variable." + ) def test_the_dotenv_borrow_is_dead(): - # Prose may mention the dead borrow; importing or calling it may not. - text = _MANAGER_SOURCE.read_text() - for token in ("load_dotenv", "from dotenv", "import dotenv", "find_dotenv"): - assert token not in text, ( - f"{token!r} is back in the manager (þing-01 #134 killed the borrow): " - "the launcher declares the environment; this package must never load one." - ) + """þing-01 #134's verdict, held over the whole package rather than two files. + + **This guard has been too narrow twice.** It was scoped to ``unfao``'s manager, and + a live ``load_dotenv`` in ``crafd/managers/crafd.py`` left the suite green. Widened + to both managers, it still covered 2 files of 30: verified 2026-08-03 that + ``load_dotenv(find_dotenv())`` at module scope in ``unfao/appwrite_env.py`` — *the + entry validator whose own docstring says the borrow is dead* — ran on import with + the suite green, as did the same line in ``product.py`` and ``launch_config.py``. + + The rule was never about managers. þing-01 D6 says **this package** validates the + environment and never loads one, so the scan is the package. Register **C-74**'s + shape twice over: a guard narrower than the sentence describing it. + """ + offenders = { + source.relative_to(_PKG).as_posix(): used + for source in sorted(_PKG.rglob("*.py")) + for used in [_dotenv_use(source.read_text())] + if used + } + assert not offenders, ( + f"python-dotenv is back in this package: {offenders}. þing-01 #134 killed the " + "borrow — the launcher declares the environment (verdict D6) and this package " + "validates it fail-loud. A module that loads a .env reintroduces the copy-chain " + "the assembly retired, and does it at import time, before any validation runs." + ) def test_missing_env_raises_naming_every_missing_variable(monkeypatch): @@ -89,18 +233,27 @@ def test_empty_string_counts_as_missing(monkeypatch): ) -def test_declared_names_match_the_manager_reads(): +@pytest.mark.parametrize("partner", _PARTNERS) +def test_declared_names_match_the_manager_reads(partner): # Declaration and use must not drift: every APPWRITE_* name the manager # actually reads is declared, and both store paths validate before building. - text = _MANAGER_SOURCE.read_text() + # + # Parametrised since #211. The crafd manager reads its own four coordinates by + # hardcoded literal (`crafd.py:363-366`) exactly as the FAO one does, so it can + # drift from its own declaration in exactly the same way — and did not have a + # check saying otherwise. + module, own_store, _ = _PARTNER_ENV[partner] + text = _manager_source(partner).read_text() read_names = set(re.findall(r'os\.getenv\("(APPWRITE_[A-Z_]+)"\)', text)) - declared = set( - appwrite_env.CONNECTION_ENV + appwrite_env.PROD_FORECASTS_ENV + appwrite_env.UNFAO_ENV + declared = set(module.CONNECTION_ENV + module.PROD_FORECASTS_ENV + own_store) + assert read_names == declared, ( + f"[{partner}] the manager reads {sorted(read_names - declared)} without " + f"declaring them, and declares {sorted(declared - read_names)} without " + "reading them" ) - assert read_names == declared # One validation per store construction. Was 3 until #149 retired the legacy # `_save`, whose Appwrite config duplicated `_unfao_appwrite_config` verbatim; - # the two survivors are the production_forecasts read and the unfao_bucket write. + # the two survivors are the production_forecasts read and the partner-bucket write. assert text.count("appwrite_env.assert_env_declared(") == 2 @@ -223,36 +376,37 @@ def test_secret_env_names_follow_the_seam_contract_naming_rule(): #: inferred from a name's prefix"), reproduced inside the test written to enforce it. #: It happened to be correct, which is what makes the habit worth breaking rather than #: excusing. Adding a name without classifying it now fails below. -_EXPECTED_CLASS = { - "APPWRITE_ENDPOINT": "connection", - "APPWRITE_DATASTORE_PROJECT_ID": "connection", - "APPWRITE_DATASTORE_API_KEY": "secret", - "APPWRITE_PROD_FORECASTS_BUCKET_ID": "target", - "APPWRITE_PROD_FORECASTS_BUCKET_NAME": "target", - "APPWRITE_PROD_FORECASTS_COLLECTION_ID": "target", - "APPWRITE_PROD_FORECASTS_COLLECTION_NAME": "target", - "APPWRITE_UNFAO_BUCKET_ID": "target", - "APPWRITE_UNFAO_BUCKET_NAME": "target", - "APPWRITE_UNFAO_COLLECTION_ID": "target", - "APPWRITE_UNFAO_COLLECTION_NAME": "target", - "APPWRITE_METADATA_DATABASE_ID": "target", - "APPWRITE_METADATA_DATABASE_NAME": "target", -} +def test_every_partner_package_has_its_environment_checked_here(): + """Assert this module's declared scope is the real one (ADR-014 §2). + + ``_PARTNER_ENV`` is what every parametrised check below iterates. A partner + package missing from it is a partner whose coordinates nobody compares against the + registry — and the suite stays green, which is exactly how ``crafd`` arrived. So + the keys are checked against the repository's single declared partner list rather + than maintained by hand and hoped over. + """ + assert set(_PARTNERS) == set(PARTNER_PACKAGES), ( + f"partner packages without an environment check: " + f"{sorted(set(PARTNER_PACKAGES) - set(_PARTNERS))}; " + f"checked but no longer a partner: {sorted(set(_PARTNERS) - set(PARTNER_PACKAGES))}. " + "Add the partner to _PARTNER_ENV — an unlisted one is silently exempt from " + "every registry comparison in this file." + ) -def test_every_declared_name_is_classified_here(): - """The map above must cover the module exactly — no silent gaps, no strays. +@pytest.mark.parametrize("partner", _PARTNERS) +def test_every_declared_name_is_classified_here(partner): + """The map above must cover each module exactly — no silent gaps, no strays. Without this, adding a name to one of the ENV tuples would simply not be checked against the registry, and the drift test would keep passing while covering less. That is register **C-74**'s shape: a guard quietly narrower than it claims. """ - declared = set( - appwrite_env.CONNECTION_ENV + appwrite_env.PROD_FORECASTS_ENV + appwrite_env.UNFAO_ENV - ) - assert set(_EXPECTED_CLASS) == declared, ( - f"unclassified: {sorted(declared - set(_EXPECTED_CLASS))}; " - f"stale: {sorted(set(_EXPECTED_CLASS) - declared)}" + module, own_store, expected = _PARTNER_ENV[partner] + declared = set(module.CONNECTION_ENV + module.PROD_FORECASTS_ENV + own_store) + assert set(expected) == declared, ( + f"[{partner}] unclassified: {sorted(declared - set(expected))}; " + f"stale: {sorted(set(expected) - declared)}" ) @@ -274,47 +428,55 @@ def _declared_classes(registry: dict) -> dict[str, str]: } -def test_every_declared_name_exists_in_the_registry_with_the_class_we_treat_it_as(): +@pytest.mark.parametrize("partner", _PARTNERS) +def test_every_declared_name_exists_in_the_registry_with_the_class_we_treat_it_as(partner): """C-57: a rename or reclassification upstream must not be silent here.""" repo = require_sibling("views-appwrite") declared = _declared_classes(_load_registry(repo)) + _, _, expected_class = _PARTNER_ENV[partner] - missing = sorted(n for n in _EXPECTED_CLASS if n not in declared) + missing = sorted(n for n in expected_class if n not in declared) assert not missing, ( - f"names this package requires are absent from the Appwrite Seam Contract's " - f"registry: {missing}. Either the registry retired them or this module invented " - "them; the registry is the authority." + f"[{partner}] names this package requires are absent from the Appwrite Seam " + f"Contract's registry: {missing}. Either the registry retired them or this " + "module invented them; the registry is the authority." ) misclassified = { n: (expected, declared[n]) - for n, expected in _EXPECTED_CLASS.items() + for n, expected in expected_class.items() if declared[n] != expected } assert not misclassified, ( - f"class mismatch (expected, registry) {misclassified}. Class is DECLARED by the " - "registry, never inferred from a name's prefix — a coordinate treated as a " - "secret (or the reverse) is a redaction bug waiting to happen." + f"[{partner}] class mismatch (expected, registry) {misclassified}. Class is " + "DECLARED by the registry, never inferred from a name's prefix — a coordinate " + "treated as a secret (or the reverse) is a redaction bug waiting to happen." ) -def test_the_pinned_contract_edition_still_matches_the_registry(): +@pytest.mark.parametrize("partner", _PARTNERS) +def test_the_pinned_contract_edition_still_matches_the_registry(partner): """The check that catches everything the other one cannot — including additions. Names and classes catch a rename. The edition catches **any** other change: a new target this repo ought to adopt, a retired secret slot, a reworded rule. It fails loudly and tells you what to do rather than what broke. + + It is also the check that would have caught #211's stale crafd pin on the day it + was written, had it been looking at crafd. It was not. It is now. """ repo = require_sibling("views-appwrite") + module = _PARTNER_ENV[partner][0] actual = _load_registry(repo)["meta"]["version"] - assert actual == appwrite_env.SEAM_CONTRACT_VERSION, ( - f"the Appwrite Seam Contract's registry moved to v{actual}; this repo declares " - f"v{appwrite_env.SEAM_CONTRACT_VERSION}. Re-verify appwrite_env's declaration " - f"against v{actual}, then bump SEAM_CONTRACT_VERSION and SEAM_CONTRACT_COMMIT " - "together. Do not bump one alone — the pair is the claim." + assert actual == module.SEAM_CONTRACT_VERSION, ( + f"the Appwrite Seam Contract's registry moved to v{actual}; " + f"{partner}/appwrite_env.py declares v{module.SEAM_CONTRACT_VERSION}. Re-verify " + f"that module's declaration against v{actual}, then bump SEAM_CONTRACT_VERSION " + "and SEAM_CONTRACT_COMMIT together. Do not bump one alone — the pair is the claim." ) -def test_the_pinned_commit_is_reachable_from_the_contract_repos_main(): +@pytest.mark.parametrize("partner", _PARTNERS) +def test_the_pinned_commit_is_reachable_from_the_contract_repos_main(partner): """Existence is not reachability, and that distinction cost a merged PR (#196). S3 pinned a commit resolved with ``rev-parse HEAD`` on a views-appwrite checkout @@ -323,44 +485,60 @@ def test_the_pinned_commit_is_reachable_from_the_contract_repos_main(): ``main``, declared a version that was never ratified, and was withdrawn. """ repo = require_sibling("views-appwrite") - commit = appwrite_env.SEAM_CONTRACT_COMMIT + commit = _PARTNER_ENV[partner][0].SEAM_CONTRACT_COMMIT if not git_output(repo, "cat-file", "-t", commit): pytest.skip( f"{commit} is not in the local views-appwrite checkout — run `git fetch` " "there; a stale clone cannot answer whether the pin reached main" ) assert commit_is_on_main(repo, commit), ( - f"the pinned commit {commit!r} is not an ancestor of views-appwrite's main. A " + f"[{partner}] the pinned commit {commit!r} is not an ancestor of " + "views-appwrite's main. A " "pin taken from a working copy's HEAD can land on an unmerged branch — that is " "#196, verbatim. Re-pin from `git rev-parse --short origin/main`." ) -def test_the_drift_check_would_catch_a_rename(tmp_path): +@pytest.mark.parametrize("partner", _PARTNERS) +def test_the_drift_check_would_catch_a_rename(partner): """A gated test that cannot fail is decoration — so prove this one bites in CI. Runs with **no** views-appwrite checkout: a synthetic registry with one name renamed and one reclassified, fed to the same comparison the gated tests use. + + Parametrised over partners for the reason the whole file now is: a detector proven + against one partner's coordinates is not proven against another's, and the gated + checks skip on any machine without a views-appwrite checkout — so this is the only + proof that runs everywhere. """ + module, own_store, expected_class = _PARTNER_ENV[partner] + # The partner's own outbound bucket id — a `target` in the real registry, which is + # what makes reclassifying it to `secret` the meaningful mutation. + canary = own_store[0] + registry = { - "meta": {"version": appwrite_env.SEAM_CONTRACT_VERSION}, + "meta": {"version": module.SEAM_CONTRACT_VERSION}, "connection": {"APPWRITE_ENDPOINT": {"class": "connection"}}, - "target": {"APPWRITE_UNFAO_BUCKET_ID": {"class": "secret"}}, # reclassified + "target": {canary: {"class": "secret"}}, # reclassified "secret": {"APPWRITE_DATASTORE_API_KEY": {"class": "secret"}}, } declared = _declared_classes(registry) + assert expected_class[canary] == "target", ( + f"{canary} is not classified as a target here, so reclassifying it below is " + "not the mutation this test believes it is" + ) assert "APPWRITE_DATASTORE_PROJECT_ID" not in declared, "fixture should omit it" - missing = sorted(n for n in _EXPECTED_CLASS if n not in declared) + missing = sorted(n for n in expected_class if n not in declared) assert missing, "the detector reported no missing names against a registry that omits most" mismatched = [ - n for n, expected in _EXPECTED_CLASS.items() + n for n, expected in expected_class.items() if n in declared and declared[n] != expected ] - assert "APPWRITE_UNFAO_BUCKET_ID" in mismatched, ( - "a target reclassified as a secret went unnoticed — that is the case where " - "getting it wrong leaks or hides a value" + assert canary in mismatched, ( + f"[{partner}] a target reclassified as a secret went unnoticed — that is the " + "case where getting it wrong leaks or hides a value" ) diff --git a/tests/test_frame_extraction.py b/tests/test_frame_extraction.py index 947aae1..210d628 100644 --- a/tests/test_frame_extraction.py +++ b/tests/test_frame_extraction.py @@ -1,4 +1,4 @@ -"""Tests for the representation seam (`unfao/frame_extraction.py`). +"""Tests for the representation seam (`contract/frame_extraction.py`). **These were parity tests until #151.** They proved that the same data expressed as a pandas MultiIndex frame and as a views-frames `PredictionFrame` yielded *identical* diff --git a/tests/test_product.py b/tests/test_product.py index ab4b0a8..8cdab80 100644 --- a/tests/test_product.py +++ b/tests/test_product.py @@ -1,22 +1,169 @@ -"""The declared FAO product (unfao/product.py) — ADR-013 §4.2a/§6/§4.1a/§11.4 pins.""" +"""Each partner's declared product — ADR-013 §4.2a/§6/§4.1a/§11.4 pins. -from views_postprocessing.unfao import product +**Scoped to the FAO product until #211.** When `crafd/product.py` landed, every pin +here still read `unfao.product`: nothing asserted CRAF'd's consumer document name, +its target vocabulary, or that its upload interlock defaulted off. The values were all +correct — but "correct and unchecked" is the state a product declaration is in right +up until the day it is not, and §4.1a's failure mode is silence, not an error. +The consumer-name pin is the one that matters most and the one a partner-scoped test +could never catch: a document written under a name the consumer does not filter for is +delivered, stored, billed, and invisible. That is not hypothetical here — six +`orange_ensemble` forecast documents sat in `unfao_bucket` for months while FAO's +forecast serving read empty (ADR-013 §11.4 post-adoption record, register C-01). -def test_targets_are_the_pinned_wire_vocabulary(): +Parametrised over `tests/conftest.PARTNER_PACKAGES`, so a third partner is caught by +`test_clone_readiness.py::test_the_declared_partner_list_is_the_real_one` rather than +quietly skipped. +""" + +from __future__ import annotations + +import importlib +import re +from pathlib import Path + +import pytest + +from tests.conftest import CONSUMER_REPO, PARTNER_PACKAGES, SIBLING_ENV, require_sibling + +#: partner -> the document name its consumer filters on, DECLARED here rather than +#: read back from the module under test. +#: +#: A test that asserts ``product.CONSUMER_DOCUMENT_NAME == product.CONSUMER_DOCUMENT_NAME`` +#: passes for every possible value, which is the shape of the replica defect epic #181 +#: spent a story retiring. So the expected value is written out. +#: +#: **But a literal transcribed from another repository is a guarantee, and ADR-014 §1 +#: says a guarantee needs a check.** On its own this pin catches drift authored *here* — +#: which was never the danger. If views-crafdapi changed its filter tomorrow, this file +#: would stay green, the delivery would upload, and the document would be invisible: +#: ADR-013 §4.1a, the defect that stranded six `orange_ensemble` documents in +#: `unfao_bucket` while FAO's forecast serving read empty for months. +#: +#: ``test_the_declared_consumer_name_still_matches_the_consumer`` closes that half +#: against the sibling checkout, following the pattern +#: ``tests/test_env_declaration.py`` already uses for the coordinate registry: declared +#: locally, verified across the seam when the other repo is on disk. +_CONSUMER_DOCUMENT_NAME = { + "unfao": "un_fao", + "crafd": "un_crafd", +} + +#: Where the consumer's name lives, and the mechanism that consumes it. Both are +#: pinned: a consumer that kept the string but started filtering on a different field +#: would strand a delivery just as thoroughly as one that renamed it. +_CONSUMER_PATH_MANAGER = re.compile(r'APIPathManager\(\s*"([a-z0-9_]+)"') +_CONSUMER_FILTER = 'filters["name"] = self.model_path.model_name' + + +def _product(partner: str): + return importlib.import_module(f"views_postprocessing.{partner}.product") + + +def _consumer_package(repo: Path) -> Path: + """``src/views_api`` inside a consumer checkout.""" + candidates = sorted((repo / "src").glob("views_*api")) + assert len(candidates) == 1, ( + f"expected exactly one package under {repo}/src, found {candidates}. The " + "consumer's layout changed; this test's assumption about where to look is " + "part of what it asserts." + ) + return candidates[0] + + +def test_every_partner_has_its_consumer_name_pinned(): + """Assert this file's declared scope is the real one (ADR-014 §2).""" + assert set(_CONSUMER_DOCUMENT_NAME) == set(PARTNER_PACKAGES), ( + f"partners with no pinned consumer name: " + f"{sorted(set(PARTNER_PACKAGES) - set(_CONSUMER_DOCUMENT_NAME))}; " + f"pinned but no longer a partner: " + f"{sorted(set(_CONSUMER_DOCUMENT_NAME) - set(PARTNER_PACKAGES))}. An unpinned " + "partner can be renamed into invisibility without a single test failing." + ) + + +@pytest.mark.parametrize("partner", PARTNER_PACKAGES) +def test_targets_are_the_pinned_wire_vocabulary(partner): # §7a: wire names, never internal model names; order stable for manifests. - assert product.TARGETS == ("lr_ged_sb", "lr_ged_ns", "lr_ged_os") + assert _product(partner).TARGETS == ("lr_ged_sb", "lr_ged_ns", "lr_ged_os") -def test_s_min_is_the_walking_skeleton_floor(): - assert product.S_MIN == 2 +@pytest.mark.parametrize("partner", PARTNER_PACKAGES) +def test_s_min_is_the_walking_skeleton_floor(partner): + # §6's walking-skeleton value. The production floor is still an open maintainer + # item in ADR-013's post-adoption record; both partners inherit it, deliberately. + assert _product(partner).S_MIN == 2 -def test_consumer_document_name_is_the_faoapi_pin(): - # §4.1a: any other name is invisible to faoapi's unconditional name filter. - assert product.CONSUMER_DOCUMENT_NAME == "un_fao" +@pytest.mark.parametrize("partner", PARTNER_PACKAGES) +def test_consumer_document_name_is_the_pin_its_consumer_filters_on(partner): + # §4.1a: any other name is invisible to the consumer's unconditional name filter. + assert _product(partner).CONSUMER_DOCUMENT_NAME == _CONSUMER_DOCUMENT_NAME[partner] -def test_upload_interlock_defaults_off(): +@pytest.mark.parametrize("partner", PARTNER_PACKAGES) +def test_upload_interlock_defaults_off(partner): # §11.4: the default configuration must be unable to touch the live bucket. - assert product.UPLOAD_ENABLED is False + assert _product(partner).UPLOAD_ENABLED is False + + +# ── across the seam: the pin above, checked against the repo that owns the fact ── + + +def test_every_partner_has_a_declared_consumer_repository(): + """The gated check below iterates this map; assert it is real (ADR-014 §2).""" + assert set(CONSUMER_REPO) == set(PARTNER_PACKAGES), ( + f"partners with no declared consumer repository: " + f"{sorted(set(PARTNER_PACKAGES) - set(CONSUMER_REPO))}. Without one, that " + "partner's consumer-name pin is never checked against the consumer." + ) + undeclared = sorted(r for r in CONSUMER_REPO.values() if r not in SIBLING_ENV) + assert not undeclared, ( + f"consumer repositories with no SIBLING_ENV entry: {undeclared}. " + "require_sibling() raises KeyError rather than skipping for those, so the " + "check would fail confusingly instead of skipping cleanly." + ) + + +@pytest.mark.parametrize("partner", PARTNER_PACKAGES) +def test_the_declared_consumer_name_still_matches_the_consumer(partner): + """§4.1a across the seam — the half a local literal cannot carry. + + The consumer decides what it filters on. This repository declares what it writes. + Agreement between the two is the whole of §4.1a, and it is not a fact this + repository owns — so it is checked against the consumer's checkout, exactly as the + coordinate registry is checked against views-appwrite's. + + Gated: skips without the sibling, naming the variable to set. The always-on pin + above still runs everywhere, so CI is never left with nothing. + """ + repo = require_sibling(CONSUMER_REPO[partner]) + pkg = _consumer_package(repo) + + api = (pkg / "managers" / "api.py").read_text() + found = _CONSUMER_PATH_MANAGER.findall(api) + assert len(found) == 1, ( + f"expected exactly one APIPathManager(...) construction in " + f"{CONSUMER_REPO[partner]}'s managers/api.py, found {found}. More than one " + "means the consumer serves several document names and this check no longer " + "knows which one is ours." + ) + assert found[0] == _CONSUMER_DOCUMENT_NAME[partner], ( + f"{CONSUMER_REPO[partner]} filters on {found[0]!r}; this repository declares " + f"{_CONSUMER_DOCUMENT_NAME[partner]!r} and writes it as " + f"{partner}/product.py's CONSUMER_DOCUMENT_NAME. A delivery under a name the " + "consumer does not ask for is uploaded, stored, and invisible (ADR-013 §4.1a)." + ) + assert found[0] == _product(partner).CONSUMER_DOCUMENT_NAME, ( + "the declared pin in this file and the module's constant disagree — the " + "always-on test above should have caught this first" + ) + + manager = (pkg / "managers" / "prediction" / "manager.py").read_text() + assert _CONSUMER_FILTER in manager, ( + f"{CONSUMER_REPO[partner]} no longer selects by " + f"{_CONSUMER_FILTER!r}. The name may still match while the consumer filters on " + "something else entirely — same invisibility, different cause. Re-read its " + "selection path before assuming this repository's deliveries are reachable." + ) diff --git a/tests/test_views_frames_conformance.py b/tests/test_views_frames_conformance.py index e17a3fe..5e94c74 100644 --- a/tests/test_views_frames_conformance.py +++ b/tests/test_views_frames_conformance.py @@ -1,4 +1,4 @@ -"""Conformance + parity tests for the views-frames constructors (`unfao/frames.py`). +"""Conformance + parity tests for the views-frames constructors (`contract/frames.py`). `build_prediction_frame` / `build_target_frame` take **declared primitives** — a 2-D `(N, S)` value array + `(time, unit)` arrays — and build a views-frames value object. These diff --git a/views_postprocessing/contract/__init__.py b/views_postprocessing/contract/__init__.py index b1bb6a5..421d0b1 100644 --- a/views_postprocessing/contract/__init__.py +++ b/views_postprocessing/contract/__init__.py @@ -4,7 +4,10 @@ delivery/ what makes a delivery VALID — representation-free invariants contract/ how a delivery is BUILT — this package - unfao/ who a delivery is FOR — one partner's product and manager + unfao/ who a delivery is FOR — the FAO product and its manager + crafd/ who a delivery is FOR — the CRAF'd product and its manager + +There is one package per partner and this package must import none of them. **Why this package exists (register C-69, #153).** Until now ~800 of `unfao/`'s 1,001 lines were partner-neutral: the whole ADR-013 wire, the frame seam, the GAUL @@ -17,9 +20,10 @@ coming UN-agency deliveries reuse these"* and nothing in it names FAO. This package extends that property to the machinery. -**Nothing here may import `unfao`.** That is the whole point, and it is enforced -mechanically rather than by convention — see `tests/test_clone_readiness.py` (#155). +**Nothing here may import a partner package.** That is the whole point, and it is +enforced mechanically rather than by convention — see `tests/test_clone_readiness.py` +(#155), which since #211 checks every declared partner rather than only `unfao`. -What a clone supplies for itself: its product declarations, its store coordinates, +What a partner supplies for itself: its product declarations, its store coordinates, and its manager. See `docs/CLONING.md`. """ diff --git a/views_postprocessing/contract/wire/__init__.py b/views_postprocessing/contract/wire/__init__.py index 1b0b3c6..ed88c67 100644 --- a/views_postprocessing/contract/wire/__init__.py +++ b/views_postprocessing/contract/wire/__init__.py @@ -2,7 +2,8 @@ One closure: this package changes when the wire contract (``docs/ADRs/013_sampled_forecast_wire_contract.md``) changes, and for no other -reason. The FAO *product* declaration lives outside it (``unfao/product.py`` — the partner layer; -different reason to change); representation-free *rules* live below it +reason. Each partner's *product* declaration lives outside it (``unfao/product.py``, +``crafd/product.py`` — the partner layer, a different reason to change); +representation-free *rules* live below it (``delivery/``); the manager above composes it. """ diff --git a/views_postprocessing/crafd/appwrite_env.py b/views_postprocessing/crafd/appwrite_env.py index 4758f12..68c42ac 100644 --- a/views_postprocessing/crafd/appwrite_env.py +++ b/views_postprocessing/crafd/appwrite_env.py @@ -6,16 +6,21 @@ views-appwrite and referenced by pinned URL, never copied — copies were the platform's original failure (þing-01 sáttmál S6): - https://github.com/views-platform/views-appwrite/blob/47172af/docs/ADRs/platform/coordinate_registry.toml - -That pin is registry **v1.3.0** (ratified, þing-02) — declared below as -``SEAM_CONTRACT_VERSION`` / ``SEAM_CONTRACT_COMMIT`` so the pin is a value a test can -check rather than a fact buried in prose. The ``APPWRITE_CRAFD_*`` names are declared -at that edition (reserved slots); their *values* were filled by the operator at -views-crafdapi S9 (views-appwrite PR #38) — but this module declares names, never -reads values. A pinned URL does not rot, but it does go stale, and nothing in this + https://github.com/views-platform/views-appwrite/blob/90fc105/docs/ADRs/platform/coordinate_registry.toml + +That pin is registry **v1.4.1** — declared below as ``SEAM_CONTRACT_VERSION`` / +``SEAM_CONTRACT_COMMIT`` so the pin is a value a test can check rather than a fact +buried in prose. A pinned URL does not rot, but it does go stale, and nothing in this repository could previously tell you it had (register C-57). +**The first pin taken here predated its own coordinates having values.** At commit +``47172af`` the four ``APPWRITE_CRAFD_*`` names were reserved slots with **no values**. +The operator filled them in views-appwrite PR #38 (merged ``12eb6c6``, 2026-08-02), +which landed *within* v1.3.0 without bumping the version; v1.4.0 came two hours later +in PR #40. So the stale pin was stale by commit, not only by version — and no test +could see it, because the drift check was scoped to the FAO package by name (register +C-57, amended 2026-08-03). It is now scoped to both. + **Pin from the tip of `main`, never from a sibling checkout's `HEAD`** (#196). The LAUNCHER assembles the environment @@ -37,8 +42,13 @@ #: this repo cites. **A version string and a sha are not coordinate values** — the #: registry forbids copying its values, and nothing here copies one. What is recorded is #: *which edition was read*, which is exactly what makes drift detectable. -SEAM_CONTRACT_VERSION = "1.3.0" -SEAM_CONTRACT_COMMIT = "47172af" +#: +#: Bumping these is not bookkeeping: it asserts that someone re-checked this module's +#: declaration against that edition of the registry. ``tests/test_env_declaration.py`` +#: enforces the pair — for this package and for ``unfao`` alike — against a local +#: views-appwrite checkout when one is present. +SEAM_CONTRACT_VERSION = "1.4.1" +SEAM_CONTRACT_COMMIT = "90fc105" CONNECTION_ENV = ( "APPWRITE_ENDPOINT", diff --git a/views_postprocessing/crafd/managers/crafd.py b/views_postprocessing/crafd/managers/crafd.py index 8613160..12119f5 100644 --- a/views_postprocessing/crafd/managers/crafd.py +++ b/views_postprocessing/crafd/managers/crafd.py @@ -51,9 +51,13 @@ def upload(self, file_path, *, filename, name, doc_type, category, loa, targets, targets=targets, description=description, ) - # The store module degrades gracefully (its ADR-046 policy) and only LOGS - # metadata failures — which strands an invisible orphan file (run-0 - # historical, 2026-07-27). The delivery fails loud instead. + # On a metadata failure the store logs, then RETURNS success=False with the + # file already uploaded (pipeline-core modules/appwrite/file.py — the file is + # the claim; its line number moves between releases). It never raises, so a + # caller that discards the result ships an invisible orphan: run-0's historical + # artifact, 2026-07-27. This check is the whole mechanism. + # (Until 2026-08-03 this comment said the store "only LOGS": false, and + # self-defeating — if it only logged, `success` would be True.) success = getattr(result, "success", None) if success is None and hasattr(result, "to_dict"): success = result.to_dict().get("success") @@ -316,7 +320,7 @@ def _save_contract(self) -> dict: upload_enabled=upload_enabled, ) # Historical leg (#126): the CRAF'd product ships actuals alongside the wire — - # frame-built, same artifact shape faoapi already ingests, same interlock. + # frame-built, same artifact shape the FAO delivery already ships, same interlock. if self._historical_frame is None: raise ValueError( "contract _save: no historical frame — the un_crafd descriptor must " diff --git a/views_postprocessing/unfao/managers/README.md b/views_postprocessing/unfao/managers/README.md index abbd524..b3aa405 100644 --- a/views_postprocessing/unfao/managers/README.md +++ b/views_postprocessing/unfao/managers/README.md @@ -17,13 +17,16 @@ and uploads the result to the FAO Appwrite store. rules live in `views_postprocessing/delivery/` and are **called** by the manager (via the `contract/frame_extraction.py` seam), never inherited into it. -It is **406 lines**, against a 450-line budget enforced by +It sits just under a **450-line budget** — applied to this whole directory, not to this file alone — enforced by `tests/test_doc_accuracy.py::test_the_manager_stays_within_its_line_budget`. It was 636 before #149. ADR-012 deliberately stopped calling it "thin" and states a number instead — a word nobody can check became a bound a test can. -It is also **the only module in this repository that imports `views_pipeline_core`**, and a -test keeps it that way (register C-40). +It is also one of **only two modules in this repository that import +`views_pipeline_core`** — this one and its CRAF'd counterpart, `crafd/managers/crafd.py`, +added in #211. A test holds that to an explicit allowlist, so a *third* importer fails +CI (register C-40). The two files are near-identical by design; register **C-33** names +what would make it time to stop copying. It does **not** transform prediction values (no collapse, no reconciliation — those are downstream). It joins metadata, guards integrity, and delivers. diff --git a/views_postprocessing/unfao/managers/unfao.py b/views_postprocessing/unfao/managers/unfao.py index 11782d2..bf4e91c 100644 --- a/views_postprocessing/unfao/managers/unfao.py +++ b/views_postprocessing/unfao/managers/unfao.py @@ -51,9 +51,13 @@ def upload(self, file_path, *, filename, name, doc_type, category, loa, targets, targets=targets, description=description, ) - # The store module degrades gracefully (its ADR-046 policy) and only LOGS - # metadata failures — which strands an invisible orphan file (run-0 - # historical, 2026-07-27). The delivery fails loud instead. + # On a metadata failure the store logs, then RETURNS success=False with the + # file already uploaded (pipeline-core modules/appwrite/file.py — the file is + # the claim; its line number moves between releases). It never raises, so a + # caller that discards the result ships an invisible orphan: run-0's historical + # artifact, 2026-07-27. This check is the whole mechanism. + # (Until 2026-08-03 this comment said the store "only LOGS": false, and + # self-defeating — if it only logged, `success` would be True.) success = getattr(result, "success", None) if success is None and hasattr(result, "to_dict"): success = result.to_dict().get("success") @@ -219,7 +223,7 @@ def _transform(self) -> None: """No-op by design. Geography is not joined into either payload: the historical artifact - attaches it at build time (``unfao/historical.py``) and the forecast ships + attaches it at build time (``contract/historical.py``) and the forecast ships it as the §5 GAUL sidecar, built in the sink at ``_save``. The hook stays so the Template Method's phases remain truthful. """ From 1e3bf300a724b54f9bd5b6af24b900b3851db5be Mon Sep 17 00:00:00 2001 From: Polichinl Date: Mon, 3 Aug 2026 13:24:15 +0200 Subject: [PATCH 23/23] =?UTF-8?q?fix(docs):=20pre-main-sync=20=E2=80=94=20?= =?UTF-8?q?the=20runbook=20told=20operators=20there=20is=20no=20rollback?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three review iterations on the development->main sync. Each found defects in the previous iteration's fixes; the pattern held all three times. The one that mattered - docs/operations/correction_procedure.md said withdrawal "has no mechanism today" and "there is no rollback". Both false. views-faoapi ships an operator quarantine (f1a59bf, on its main): add a store document's file-id to APPWRITE_UNFAO_QUARANTINED_FILE_IDS and selection drops it, falling the consumer back to the previous run. Read at selection time, no redeploy, nothing deleted, reversible. The runbook instead sent an operator to rebuild and republish an entire corrected run, for hours, while wrong data stayed live. - Iteration 2 found that fix incomplete: a run has TWO selection entry points. The historical artifact is a separate document selected by category="historical" and never consults the manifest. Quarantining the manifest alone leaves it serving — wrong way round for a geography fault, since lookup_version lives only there. §3 now gives a per-leg table. - The env vars are partner-specific. Setting FAO's name against a CRAF'd deployment is a silent no-op. Said in the runbook and in CLONING.md, which had told a new partner these steps "transfer unchanged". The guard that protected the error - test_the_procedure_distinguishes_intended_policy_from_what_is_implemented had been mutation-proven against the false claim: it asserted the doc must say withdrawal costs "an ADR-013 amendment". ADR-014 §2 asks whether a guard bites, never whether what it bites on is true. Replaced with one that pins the mechanism -- checkable against another repository rather than against our own prose. Other corrections - README's delivered-schema table had the NORMATIVE column order reversed and typed the four *_code columns int where the wire is float64. A consumer building a reader from it ships a broken one. Corrected and now guarded against gaul_schema.COLUMNS. - README copied four real registry VALUES into a public file, two lines below the sentence saying they never are. The value-copy guard scanned only .py; it now scans tracked markdown, matching NAME=value assignments (a substring draft cried wolf on prose that merely names a store). - The UNFAOPostProcessorManager CIC named GaulLookupEnricher as its enrichment collaborator in six places; the manager has zero references and a test asserts its absence. Its sibling CIC said the opposite. Five further CIC claims described deleted code. - Runbook: run_id and lookup_version are on different artifacts and never co-occur; the fidelity count is 24, not 26; region-set equality is gated, not always-on. - ADR-014 said "no test enforces this and none is proposed" while two partial guards shipped in the same branch. - test_no_partner_contact_details_... scanned the repo root, so a CI sibling checkout could turn main red with a message naming another repo's file. Scoped to tracked files -- via a directory pathspec, because git's ** does not match top-level files and the first fix silently narrowed a security-adjacent scan. CI - run_pytest.yml checks out views-crafdapi. Recovers exactly ONE test. views- datafactory was tried and reverted: its tests need raw parquets absent from its git repo. views-appwrite and views-faoapi are private -- C-81. Register: C-80, C-81, C-82 added; C-07, C-15, C-22, C-24, C-57, Cluster H, D-12 corrected. 79/19/60 -> 82/22/60. 402 collected; 362 passed / 40 xfailed locally, 347/17/38 in a bare checkout, 348/16/38 with the crafdapi sibling. ruff clean. Every new guard mutation-proven. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/run_pytest.yml | 42 +++++ README.md | 83 +++++---- ...4_claims_and_the_guards_that_carry_them.md | 40 ++++- docs/CICs/UNFAOPostProcessorManager.md | 45 +++-- docs/CLONING.md | 15 +- docs/operations/correction_procedure.md | 163 ++++++++++++++---- reports/technical_risk_register.md | 126 ++++++++++++-- tests/test_doc_accuracy.py | 91 ++++++++-- tests/test_env_declaration.py | 59 ++++++- views_postprocessing/unfao/managers/README.md | 2 +- 10 files changed, 550 insertions(+), 116 deletions(-) diff --git a/.github/workflows/run_pytest.yml b/.github/workflows/run_pytest.yml index 6f6cea6..b5dc420 100644 --- a/.github/workflows/run_pytest.yml +++ b/.github/workflows/run_pytest.yml @@ -19,6 +19,46 @@ jobs: - name: Checkout repository uses: actions/checkout@v3 + # ── sibling checkouts: what CI can verify that a bare checkout cannot ──────── + # + # `tests/conftest.sibling_repo` resolves $VIEWS_, else the conventional + # `../` directory. In a one-repo CI checkout neither exists, so every + # cross-repo assertion skips — silently and by design, because a missing sibling + # is normal. The consequence was that CI verified strictly LESS than a developer's + # laptop, and precisely on the guards that matter most across a repo boundary. + # + # Measured 2026-08-03 (402 collected in all three runs, so the delta is entirely + # skips): + # local, all siblings present 362 passed / 40 xfailed / 0 skipped + # CI, bare checkout 347 passed / 38 xfailed / 17 skipped + # CI, with the checkout below 348 passed / 38 xfailed / 16 skipped + # + # So this recovers exactly ONE test — the cross-seam consumer-name check for CRAF'd. + # That is worth having (it is the §4.1a pin, whose failure mode is a delivery nobody + # can find) but it is one, not seventeen. The other sixteen are below. + # + # Only views-crafdapi is checked out, and the choice is deliberate: + # + # views-crafdapi PUBLIC, and its check reads source text -> enabled here. + # views-datafactory PUBLIC, but its 8 tests need the producer's raw GAUL + # parquets (`data/raw/gaul_admin/*.parquet`), which are NOT in + # its git repository. Checking it out converts an honest skip + # into a FileNotFoundError. Measured, not assumed — it was tried. + # views-appwrite PRIVATE. The registry-drift detectors live here and are the + # views-faoapi most valuable of the lot; enabling them needs a token, which + # is an operator decision. Register C-46 and C-57 both carry + # that as their standing residual, and it is still open. + # `actions/checkout` refuses a path outside $GITHUB_WORKSPACE, so the siblings go + # inside it and are pointed at by the declared env vars instead of the conventional + # `../` location. `sibling_repo` checks $VIEWS_ first, precisely so the + # location can be declared rather than assumed. + - name: Checkout views-crafdapi (sibling) + uses: actions/checkout@v3 + with: + repository: views-platform/views-crafdapi + path: _siblings/views-crafdapi + fetch-depth: 0 + - name: Set up Python uses: actions/setup-python@v4 with: @@ -33,6 +73,8 @@ jobs: poetry install - name: Run tests + env: + VIEWS_CRAFDAPI: ${{ github.workspace }}/_siblings/views-crafdapi run: | set -e poetry run pytest tests/ diff --git a/README.md b/README.md index f2e071f..29460ae 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ The **post-forecast delivery layer** for the **VIEWS** (Violence Early-Warning S pipeline. It takes finished VIEWS forecasts, enriches them with geographic metadata, guards their integrity, and delivers them to a partner store. -The only live delivery today is the **UN FAO** path — its product in `views_postprocessing/unfao/`, running on the partner-neutral machinery in `contract/`. +Two partner deliveries run on the same partner-neutral machinery in `contract/`: the **UN FAO** path (`views_postprocessing/unfao/`), serving FAO-FSFC since 2026-07-27, and **CRAF'd** (`views_postprocessing/crafd/`), added 2026-08-03 with its upload interlock still closed. > **New here? Read [`docs/architecture/role_and_seams.md`](docs/architecture/role_and_seams.md) first.** > It explains what this repo is, how it relates to pipeline-core / faoapi / datafactory, @@ -50,7 +50,7 @@ Requires **Python 3.11–3.14**. | Package | Version | Why | |---------|---------|-----| | `views-pipeline-core` | `>=2.1.3,<3.0.0` | The framework: lifecycle base classes, data loader, dataset container, Appwrite/datastore tools | -| `views-frames` | `>=1.0,<2` | The frame data contract — **the live delivery representation** since #126. pandas survives only in `contract/enrichment.py` (the build/verification path) | +| `views-frames` | `>=1.10.2,<2` | The frame data contract — **the live delivery representation** since #126. pandas survives only in `contract/enrichment.py` (the build/verification path) | --- @@ -75,7 +75,7 @@ In practice the manager is constructed and run by **views-models** | **Read** | `_read_historical_frame`, `_read_forecast_data_contract` | Historical actuals from views-datafactory arrive **frame-native** (#126); the forecast run is resolved from the Appwrite store by its **run manifest**, with each shard's header verified on load (ADR-013 §4.3). | | **Transform** | `_transform` | Resolution only. Prediction values are **not** transformed — no collapse, no reconciliation. | | **Validate** | `_validate`, `_check_coverage` | Asserts the read resolved, then enforces the region coverage + GAUL-excluded-cell contract (C-34 / C-30). The metadata null-gate fires later, at artifact build (`contract/historical.assert_metadata_complete`). | -| **Save** | `_save` → `_save_contract` | Builds the ADR-013 wire — arrow shards, the §5 GAUL sidecar, the historical artifact — commits the run **manifest last**, and stamps each upload with structured provenance (C-15). | +| **Save** | `_save` → `_save_contract` | Builds the ADR-013 wire — arrow shards, the §5 GAUL sidecar, the historical artifact — commits the run **manifest last**. The historical artifact carries structured provenance in its store-document `description` (C-15); the forecast leg's uploads carry `{name, category, loa, filename, doc_type, targets}` and **no `description`** — a gap, not a design. | The pandas metadata-join and history-clip stages were retired with the legacy delivery path in #149; their rules survive as called invariants under `delivery/`. See the @@ -89,19 +89,28 @@ retraction primitive; a correction is a new complete run, manifest last. ### Output schema (geographic metadata columns) -These 9 columns are the delivered geography contract (declared in `contract/gaul_schema.py`): - -| Column | Type | Description | -|--------|------|-------------| -| `pg_xcoord` | float | PRIO-GRID cell centroid longitude | -| `pg_ycoord` | float | PRIO-GRID cell centroid latitude | -| `country_iso_a3` | str | ISO 3166-1 alpha-3 country code | -| `admin1_gaul0_code` | int | GAUL level-0 (country) code | -| `admin1_gaul0_name` | str | GAUL level-0 (country) name | -| `admin1_gaul1_code` | int | GAUL level-1 (province) code | -| `admin1_gaul1_name` | str | GAUL level-1 (province) name | -| `admin2_gaul2_code` | int | GAUL level-2 (district) code | -| `admin2_gaul2_name` | str | GAUL level-2 (district) name | +These 9 columns are the delivered geography contract, declared in +`contract/gaul_schema.py`. **The order below is normative** (ADR-013 §5.1) and is +byte-pinned by the §10 golden fixture — a reader that reorders them reads the wrong +column. `tests/test_doc_accuracy.py` checks this table against the declaration. + +| Column | Wire type | Description | +|--------|-----------|-------------| +| `pg_xcoord` | float64 | PRIO-GRID cell centroid longitude | +| `pg_ycoord` | float64 | PRIO-GRID cell centroid latitude | +| `country_iso_a3` | string | ISO 3166-1 alpha-3 country code | +| `admin1_gaul1_code` | float64 | GAUL level-1 (province) code | +| `admin1_gaul1_name` | string | GAUL level-1 (province) name | +| `admin1_gaul0_code` | float64 | GAUL level-0 (country) code | +| `admin1_gaul0_name` | string | GAUL level-0 (country) name | +| `admin2_gaul2_code` | float64 | GAUL level-2 (district) code | +| `admin2_gaul2_name` | string | GAUL level-2 (district) name | + +*(Corrected 2026-08-03: this table had `admin1_gaul0_*` before `admin1_gaul1_*` — +the reverse of the normative order — and typed the four `*_code` columns `int`. They +are **float64 on the wire, always**, by the §5.1 ruling: the codes are nullable and +arrow has no nullable int in this contract. Both errors survived because nothing +compared the table to the declaration.)* --- @@ -158,23 +167,27 @@ See [`docs/CLONING.md`](docs/CLONING.md). ## Configuration -The FAO delivery reads Appwrite connection settings from the environment. The required -names are **declared** in `unfao/appwrite_env.py` and validated fail-loud before any store -is constructed — a missing or empty variable raises, naming every one that is absent, -rather than half-configuring a client. Coordinates come from the Appwrite Seam Contract registry -(referenced by URL, never copied); the API key is an operator slot: +Each delivery reads Appwrite connection settings from the environment. The required +names are **declared** per partner — `unfao/appwrite_env.py`, `crafd/appwrite_env.py` — +and validated fail-loud before any store is constructed: a missing or empty variable +raises, naming every one that is absent, rather than half-configuring a client. + +**The names are below; the values are not.** Coordinates live in the Appwrite Seam +Contract's registry, which this repo references by pinned URL and never copies (þing-01 +sáttmál S6 — copies were the platform's original failure). The launcher supplies the +values; the API key is an operator slot. ```bash -# Appwrite connection (secrets) -APPWRITE_ENDPOINT=https://cloud.appwrite.io/v1 +# Appwrite connection +APPWRITE_ENDPOINT=... APPWRITE_DATASTORE_PROJECT_ID=... -APPWRITE_DATASTORE_API_KEY=... +APPWRITE_DATASTORE_API_KEY=... # operator-issued secret -# Production-forecasts store (input) -APPWRITE_PROD_FORECASTS_BUCKET_ID=production_forecasts -APPWRITE_PROD_FORECASTS_BUCKET_NAME=Production Forecasts -APPWRITE_PROD_FORECASTS_COLLECTION_ID=production_forecasts -APPWRITE_PROD_FORECASTS_COLLECTION_NAME=Production Forecasts +# Production-forecasts store (input — shared by every partner) +APPWRITE_PROD_FORECASTS_BUCKET_ID=... +APPWRITE_PROD_FORECASTS_BUCKET_NAME=... +APPWRITE_PROD_FORECASTS_COLLECTION_ID=... +APPWRITE_PROD_FORECASTS_COLLECTION_NAME=... # UN FAO store (output) APPWRITE_UNFAO_BUCKET_ID=... @@ -182,11 +195,21 @@ APPWRITE_UNFAO_BUCKET_NAME=... APPWRITE_UNFAO_COLLECTION_ID=... APPWRITE_UNFAO_COLLECTION_NAME=... -# Metadata database +# CRAF'd store (output) +APPWRITE_CRAFD_BUCKET_ID=... +APPWRITE_CRAFD_BUCKET_NAME=... +APPWRITE_CRAFD_COLLECTION_ID=... +APPWRITE_CRAFD_COLLECTION_NAME=... + +# Metadata database (shared) APPWRITE_METADATA_DATABASE_ID=... APPWRITE_METADATA_DATABASE_NAME=... ``` +*(Corrected 2026-08-03: four production-forecasts coordinate **values** were written out +above, two lines below the sentence saying they never are. The value-copy guard scanned +only `.py`; it now scans markdown too.)* + --- ## Documentation diff --git a/docs/ADRs/014_claims_and_the_guards_that_carry_them.md b/docs/ADRs/014_claims_and_the_guards_that_carry_them.md index 13292df..547c9f3 100644 --- a/docs/ADRs/014_claims_and_the_guards_that_carry_them.md +++ b/docs/ADRs/014_claims_and_the_guards_that_carry_them.md @@ -13,7 +13,8 @@ infrastructure), [ADR-010](010_technical_risk_register.md) (the register) ## Context Over two days in August 2026, this repository found **seven** places where it said one -thing and did another. None was a bug in the delivery. All were claims that had outlived +thing and did another. *(An eighth arrived the day after this ADR was accepted, and the +count is left at seven deliberately — see the note at the end of this section.)* None was a bug in the delivery. All were claims that had outlived what they described: - three register entries filed as open with their own stated closing conditions met @@ -33,6 +34,15 @@ The pattern is not carelessness. Each was written accurately and became false wh something else moved. What they share is that **nothing was attached to them that would notice**. +**The eighth, and why the count above stays at seven.** On 2026-08-03 — the day after +this ADR was accepted — PR #211 added a second partner package and found that *eight* +guards were scoped to the first partner by name, so the new one landed exempt from all of +them, including the þing-01 `load_dotenv` prohibition. Register **C-78** records it. The +count above is deliberately not incremented: an ADR that renumbers itself every time the +pattern recurs becomes a changelog, and the argument does not depend on the number. What +the eighth case adds is §2's sharpest form — *a guard's declared scope is part of what +must be mutation-proven, not just its matching.* + ADR-003 already forbids inferring what should be declared. This ADR is that rule applied one level up: **a declaration that nothing validates is an inference with better grammar.** @@ -105,11 +115,24 @@ If a pull request cites a register entry, an issue, a CIC or an ADR as the thing addresses, it updates that artefact **in the same pull request**. Not in a follow-up, not at closeout, not from memory. -**No test enforces this and none is proposed.** It was attempted: no expression reliably -separates *"this entry describes work that is done"* from prose, and a guard that guesses -is one that gets deleted (§3). This clause is therefore a habit, and it is written down -precisely because it is the one rule here that cannot be mechanised — which makes it the -one most likely to lapse. +**No test enforces this in general, though two partial guards exist and are green.** +`tests/test_register_integrity.py` carries +`test_no_open_entry_names_a_closing_artifact_that_already_exists` and +`test_no_open_entry_claims_its_mitigation_has_landed` — both added by S2 in this same +arc. They catch the two mechanisable shapes: an entry naming a file that now exists, and +one whose body says a mitigation landed while its header says Open. + +What resists mechanisation is the general case: no expression reliably separates *"this +entry describes work that is done"* from prose, and a guard that guesses is one that gets +deleted (§3). So the clause is mostly a habit, and it is written down because it is the +rule here least amenable to a test — which makes it the one most likely to lapse. + +*(Corrected 2026-08-03. This paragraph said "none is proposed" and the Alternatives +section called widening the guard "attempted and abandoned", while the widened guard was +already shipping in the same branch. And the guards' reach is genuinely partial: C-15 sat +Open claiming a mitigation had landed via a method that no longer exists, and escaped +`test_no_open_entry_claims_its_mitigation_has_landed` because it phrased the claim +without the em-dash the guard matches. Partial is worth saying; absent was wrong.)* The evidence that it does lapse is this epic's own record. C-71 was fixed on the morning of 2026-08-02 and sat filed open for the rest of the day while eight further stories shipped. @@ -150,5 +173,6 @@ Rejected as a replacement, not as a supplement: a checklist records what to do a and the reasoning is what survives a contributor deciding the rule does not apply to them. Worth adding later, pointing here. -**Widen `test_register_integrity` to catch §5.** Attempted and abandoned — see §5. Recorded -so the next person does not spend the same afternoon on it. +**Widen `test_register_integrity` to catch §5.** Partially done — see §5. Two shapes are +mechanised and green; the general case resists. Recorded so the next person neither +re-attempts the general case nor assumes there is nothing there. diff --git a/docs/CICs/UNFAOPostProcessorManager.md b/docs/CICs/UNFAOPostProcessorManager.md index b1a4d77..0178739 100644 --- a/docs/CICs/UNFAOPostProcessorManager.md +++ b/docs/CICs/UNFAOPostProcessorManager.md @@ -8,11 +8,26 @@ --- +> **Corrected 2026-08-03 — this document named a collaborator the manager has never +> called.** Six statements described enrichment as delegated to `GaulLookupEnricher`, +> one naming the call `GaulLookupEnricher.enrich_dataframe_with_pg_info()`. The manager +> contains **zero** references to it — `tests/test_gaul_lookup_access.py` actively +> asserts its absence — and the sibling CIC has long said *"the manager does not call +> this class."* Two contract documents asserted opposite things about the same call. +> Geography is attached by `contract/historical.py` and `contract/wire/sidecar.py` from +> a lookup the manager loads once per delivery. See register **C-75**. +> +> Five further claims in this file described deleted code and are corrected below: +> a `dotenv` load that no longer happens, a "known gap" in env validation that +> `assert_env_declared` closed, an upload count and file type that were wrong in three +> ways, a selection precondition weaker than `source_selection` enforces, and two +> "incorrect usage" examples for code paths that no longer exist. + ## 1. Purpose > **What is this class for?** -`UNFAOPostProcessorManager` orchestrates the end-to-end postprocessing pipeline that reads VIEWS conflict predictions, enriches them with geographic metadata via the precomputed GAUL lookup (`GaulLookupEnricher`, ADR-011), validates the output schema, and delivers the enriched data to the UN FAO via Appwrite cloud storage. +`UNFAOPostProcessorManager` orchestrates the end-to-end postprocessing pipeline that reads VIEWS conflict predictions, enriches them with geographic metadata from the precomputed GAUL lookup (ADR-011), validates the output schema, and delivers the enriched data to the UN FAO via Appwrite cloud storage. It is the single entrypoint for producing and delivering UN FAO-formatted prediction data. @@ -20,7 +35,7 @@ It is the single entrypoint for producing and delivering UN FAO-formatted predic ## 2. Non-Goals (Explicit Exclusions) -- This class does **not** perform spatial mapping logic — it delegates enrichment to `GaulLookupEnricher` (a merge against the precomputed GAUL lookup) +- This class does **not** perform spatial mapping logic — it reads the precomputed GAUL lookup (`contract/gaul_lookup.load()`) and the artifact builders attach geography from it - This class does **not** train, evaluate, or modify prediction models - This class does **not** define the spatial assignment algorithm - This class does **not** manage shapefile data or geographic reference assets @@ -33,7 +48,7 @@ It is the single entrypoint for producing and delivering UN FAO-formatted predic - Guarantees a 4-stage pipeline: read → transform → validate → save - Guarantees that historical data is sourced from ViewsER via `ViewsDataLoader` - Guarantees that forecast data is sourced from the Appwrite production forecasts bucket -- Guarantees that geographic metadata is added via `GaulLookupEnricher.enrich_dataframe_with_pg_info()` (a cell-id merge against the precomputed lookup) +- Guarantees that geographic metadata is attached from the precomputed lookup — by `contract/historical.py` for the historical artifact and `contract/wire/sidecar.py` for the §5 GAUL sidecar, each a keyed gather on cell id - Guarantees that required metadata columns are validated before upload - Guarantees that both historical and forecast datasets are uploaded to the UN FAO Appwrite bucket with correct metadata (name, loa, type, category) - Logs structural failures before raising them (ADR-008): the config/`loa` guards, the `_validate` gates, and the dataset/`_save` guard all `logger.error`-then-raise (#13 / C-19 resolved); the `delivery/` invariants raise representation-free, with the manager logging context at each call site @@ -45,11 +60,10 @@ It is the single entrypoint for producing and delivering UN FAO-formatted predic - Requires a `PostprocessorPathManager` at initialization pointing to valid model paths - Requires `configs` dict to contain an `ensemble` key naming the source ensemble - Requires environment variables for Appwrite connectivity (endpoint, project ID, API key, bucket/collection IDs) -- Requires the ensemble's `.env` file to be loadable via `dotenv` -- Requires the Appwrite production forecasts bucket to contain at least one file with `category="forecast"` -- Requires the precomputed GAUL lookup parquet to be present so `GaulLookupEnricher` can load it at construction +- Requires the Appwrite production forecasts bucket to contain a **complete run**: a manifest matching `{category: "forecast", type: "sampled_forecast_manifest"}`, a manifest per declared target, and every shard those manifests name. A bucket holding merely *some* `category="forecast"` file raises `SourceSelectionError` (`contract/wire/source_selection.py`) +- Requires the precomputed GAUL lookup parquet to be present; it is read once per delivery via `contract/gaul_lookup.load()` -Assumptions that are not met **must cause failure**, not fallback behavior. **Known gap:** the Appwrite env vars are read via `os.getenv()` without a startup validation — a missing var yields `None`, which is passed to `AppwriteConfig` unchecked rather than failing loud at the boundary (tracked by **C-19-adjacent / #11**; a fail-loud env check is the cheap fix). +Assumptions that are not met **must cause failure**, not fallback behavior. The environment is validated fail-loud: `appwrite_env.assert_env_declared` runs before **both** `AppwriteConfig` constructions and names every missing variable, and an empty string counts as missing. *(This paragraph previously described that as a "known gap" with `os.getenv()` passing `None` through unchecked; C-19 closed it, and `tests/test_env_declaration.py` pins it.)* --- @@ -63,8 +77,7 @@ Assumptions that are not met **must cause failure**, not fallback behavior. **Kn - Downloads data from ViewsER (network I/O) - Downloads forecast data from Appwrite (network I/O) - Writes timestamped parquet files to `data_generated/` directory -- Uploads two parquet files to the UN FAO Appwrite bucket (network I/O) -- Loads `.env` from ensemble path (modifies process environment) +- Uploads to the UN FAO Appwrite bucket (network I/O) — **only when the §11.4 interlock is open**. `product.UPLOAD_ENABLED` is `False` by default, and the sink then makes **zero** store calls. Enabled, a run uploads one parquet per (target, month) — 108 at run-0 — plus the GAUL sidecar parquet, the historical parquet, and a **JSON** run manifest, committed last. Forecast-leg documents carry `{name, category, loa, filename, doc_type, targets}` and no `description`; only the historical artifact carries structured provenance - Logs pipeline progress at INFO/ERROR levels --- @@ -95,7 +108,7 @@ The following **must never** fail silently: ## 7. Boundaries and Interactions **Allowed interactions:** -- Delegates geographic enrichment to `GaulLookupEnricher` (a merge against the precomputed GAUL lookup) +- Reads the precomputed GAUL lookup once and passes it to the artifact builders, which attach geography - Uses `views-pipeline-core` managers for path resolution, data loading, and Appwrite integration - Reads environment variables for external service configuration - Writes to local filesystem and Appwrite cloud storage @@ -132,10 +145,16 @@ manager._save() ## 9. Examples of Incorrect Usage -- **Calling `_transform()` before `_read()`** — datasets will be None, causing AttributeError - **Calling `_save()` without `_validate()`** — may upload incomplete data to partners -- **Accessing `_enricher` directly to bypass the enrichment pipeline** — violates the orchestration boundary - **Hardcoding Appwrite configuration instead of reading from environment** — violates ADR-009 +- **Reaching past the manager into `contract/` to publish** — the sink is driven through + `_ContractStorePort` so the store is one seam; bypassing it also bypasses the + `result.success` check that turns a partial upload into a refusal + +*(Two entries were removed here on 2026-08-03 because they described code that no longer +exists: "calling `_transform()` before `_read()`" — `_transform` is a documented no-op +that cannot raise — and "accessing `_enricher` directly", an attribute removed in #152 +/ C-66.)* --- @@ -157,7 +176,7 @@ The input-integrity guards (S0–S6, epic #51) are representation-free invariant - Partner-specific output formats are **evolving** — the UN FAO schema may change (see C-24, D-06 for schema divergence investigation) - The source of forecast data (Appwrite bucket/collection) is **evolving** — operational configuration - Null validation is **active** (C-01 resolved 2026-06-02) -- The enrichment source is the **precomputed GAUL lookup table** (`GaulLookupEnricher`, ADR-011), as of the Stage 3 swap; the old runtime mapper was **removed** (C-39 / PR #42) — it no longer exists in the repo +- The enrichment source is the **precomputed GAUL lookup table** (`views_postprocessing/data/gaul_lookup.parquet`, ADR-011), as of the Stage 3 swap; the old runtime mapper was **removed** (C-39 / PR #42) — it no longer exists in the repo --- diff --git a/docs/CLONING.md b/docs/CLONING.md index dd1c20c..79844a7 100644 --- a/docs/CLONING.md +++ b/docs/CLONING.md @@ -157,12 +157,19 @@ used to select a retired code path silently (register C-63). **Answer your partner's correction questions before you ship to them, not after a bad delivery.** `docs/operations/correction_procedure.md` is FAO's, and its steps 1–3 and 5 -transfer unchanged — they are contract mechanics, not partner specifics. **Step 4 does -not:** who contacts your partner, through what channel, how fast, and whether they -expect a retraction or a supersession are answers only your partner can give. +transfer as *mechanics* — but **§3's environment variable names do not.** The quarantine +that withdraws a bad run is read by **your consumer**, from a name of its own: +`APPWRITE_UNFAO_QUARANTINED_FILE_IDS` for FAO, +`APPWRITE_CRAFD_QUARANTINED_FILE_IDS` for CRAF'd. Setting the wrong partner's name is a +**silent no-op** — nothing errors and the data keeps serving. Substitute yours before you +need it, not during an incident. + +**Step 4 does not transfer at all:** through what channel, how fast, and whether they +expect a withdrawal or a supersession are answers only your partner can give. This repo shipped run-0 to the UN FAO on 2026-07-27 with that step undecided (register -C-22), and it is still undecided. Do not inherit that. +C-22). It is now half-decided — a named person makes contact, by direct email — and the +rest is with FAO. Settle yours *before* your first delivery, not after. ## Where the reasoning lives diff --git a/docs/operations/correction_procedure.md b/docs/operations/correction_procedure.md index 90fa179..9ffa53c 100644 --- a/docs/operations/correction_procedure.md +++ b/docs/operations/correction_procedure.md @@ -7,9 +7,18 @@ documents first. Where a step needs background, the link is inline and optional. Register **C-22**. Supersedes issue #15, whose procedure described the pre-2026-07 delivery (disk caches, shapefiles) and no longer applies to anything. -> **Status: steps 1–3 and 5 are executable today. Step 4 is not** — nobody has decided -> who contacts the UN FAO, or whether FAO expects a retraction or a supersession. Those -> are the operator's calls and are stated in §4 exactly as they need to be answered. +> **Status: steps 1–3 and 5 are executable today. Step 4 is executable but incomplete.** +> *Who* contacts the UN FAO is decided — Simon Polichinel von der Maase, by direct email +> (§4). What is **not** decided is whether those are the right recipients, what +> notification period FAO expects, and whether FAO wants a withdrawal or a supersession. +> Those are put to FAO as Pre-Release Note 07, Topic B. **Follow §4 as written rather +> than improvising** — an inconsistent first message to a partner is harder to correct +> than a slow one. +> +> *(Corrected 2026-08-03: this banner said nobody had decided who makes contact, while +> §4 twenty lines below already named the responsible person. The guard in +> `tests/test_doc_accuracy.py` asserts §4's sentence exists and never read the banner +> beside it.)* --- @@ -27,18 +36,35 @@ understand before touching anything. ## 1. Which deliveries are affected? -Two fields identify a delivery, and both are on it by construction: +Two fields identify a delivery — and **they are not on the same artifact.** Know this +before you try to join them, because you cannot. -| field | what it answers | where it lives | -|---|---|---| -| `run_id` | *which run* | the manifest, every shard's header, and each store document's filename | -| `lookup_version` | *which GAUL build produced its geography* | the historical artifact's store-document `description` (a compact JSON provenance record) | +| field | what it answers | where it lives | where it does NOT | +|---|---|---|---| +| `run_id` | *which run* | the manifest, every shard's header, each store document's filename | the historical artifact (its filename is `historical_dataset_.parquet`) | +| `lookup_version` | *which GAUL build produced its geography* | the historical artifact's store-document `description` (a compact JSON provenance record) | the manifest, the shards, **and the GAUL sidecar** | `lookup_version` has the form `land_gaul@f74d3b2b` — region, then the digest of the views-datafactory ingestion the lookup was built from. **It cannot silently be `"unknown"`**: since register C-60 the reader raises rather than degrading, so a stamp you can read is a stamp you can trust. +**Two consequences you will hit immediately.** + +1. **There is no key joining a `run_id` to a `lookup_version`.** The historical artifact + carries the geography stamp and no run id; the manifest carries the run id and no + geography stamp. In practice you correlate them by **upload timestamp proximity** + within the bucket. Say so in your incident notes rather than implying a join. +2. **The forecast leg carries no geography stamp at all.** The §5 GAUL sidecar has no + version field (ADR-014 §4 defers stamping it to the next ADR-013 version bump). So a + *geography* fault can be scoped precisely on the historical leg and only by timestamp + on the forecast leg. If that is the fault you have, quarantine generously — the cost + of withdrawing one run too many is a re-publish; the cost of leaving one is FAO + serving wrong geography. + +*(Corrected 2026-08-03: this section said both fields were "on it by construction", +which reads as though they sit on one artifact. They never have.)* + **To enumerate affected runs:** 1. If the fault is in **geography** (a wrong country, admin unit or coordinate), the @@ -50,8 +76,10 @@ you can read is a stamp you can trust. `expected_cell_count` and the provenance record's `actual_cell_count` bound it without opening a single parquet. -Do this before touching the store. A correction whose scope you have not established -is a second incident. +Establish scope before you *correct* — a correction whose scope you have not +established is a second incident. **But withdraw first** (§3): quarantine is reversible +and costs seconds, and scoping while wrong data serves is the expensive order. Withdraw +generously, then narrow. --- @@ -64,10 +92,20 @@ re-running changes the thing you are diagnosing. pytest -q tests/test_gaul_lookup_fidelity.py ``` -26 tests. The always-on half checks the committed lookup for key uniqueness, region-set -equality, coordinate correctness, and the absence of nulls and `-1` sentinels. With a -views-datafactory checkout present (`VIEWS_DATAFACTORY`), the second half compares all -seven GAUL columns against the producer's own parquets. +24 tests. **What runs without a views-datafactory checkout** — key uniqueness, cell +count, exclusion-set disjointness, coordinate correctness, and the absence of nulls and +`-1` sentinels, all against the committed lookup. + +**What does NOT run without one**, and this matters if you are on a laptop: the +comparison of all seven GAUL columns against the producer's own parquets, region-set +equality, the coordinate-dtype wire-stability check, and the per-cell coordinate-formula +check — four tests in all. The dtype one touches only the committed lookup and is gated +more strictly than it needs to be. Those tests **skip**, they do not fail. Check the pytest summary +line for `skipped` before concluding you have verified the geography — a green run with +skips is a weaker statement than a green run without them. + +*(Corrected 2026-08-03: this said "26 tests" — a count that was never true — and listed +region-set equality in the always-on half when it is gated.)* **If those pass and a value is still wrong, the fault is upstream, not here.** That distinction is register C-43's and it is load-bearing: this repository is verified to @@ -80,21 +118,68 @@ defect; fix the producer and rebuild. ## 3. Correct on the wire -**The intended policy is WITHDRAWAL. What is implemented is SUPERSESSION.** That gap is -deliberate, recorded, and you need to know about it before you act. +### DO THIS FIRST: withdraw the bad run. It takes one environment variable. + +**Corrected 2026-08-03.** This section previously said withdrawal had no mechanism and +there was no rollback. That was wrong, and following it cost hours: it sent you to fix +the cause and republish an entire corrected run while the wrong data stayed live. + +views-faoapi ships an operator quarantine (`f1a59bf`, on its `main`, +`managers/prediction/quarantine.py`). It is a comma-separated list of **store-document +file-ids**; selection drops anything on it *before* choosing. + +**⚠ A run has TWO selection entry points, and quarantining one does not withdraw the other.** (A run is ~112 files — §0. What matters here is that the consumer reaches them through two independent doors.) + +| leg | what to quarantine | how the consumer selects it | +|---|---|---| +| forecast | the run **manifest**'s file-id | newest manifest, then the shards it names | +| historical actuals | the **historical parquet**'s own file-id | newest document with `category="historical"` — **it never looks at the manifest** | + +``` +APPWRITE_UNFAO_QUARANTINED_FILE_IDS=, +``` + +Quarantining only the manifest leaves the historical artifact serving. That is the wrong +way round for a **geography** fault: `lookup_version` lives *only* on the historical +document (§1), so the leg you can identify precisely is the leg you would have left +live. When in doubt, quarantine both — the cost of withdrawing one document too many is +a re-publish. + +Both legs go through the same filter (`prediction/manager.py:145-156`, inside +`get_predictions_by_metadata` — which the forecast path reaches via `get_latest_file_id` +and the historical path via `get_latest_provenance`), so one variable covers both. Read at selection time — **no redeploy**. Nothing is deleted, so it is reversible: +unset the variable and the run is selectable again. Whitespace around entries is +tolerated; quotes are not stripped, so do not quote the ids. + +There is a matching allowlist, `APPWRITE_UNFAO_APPROVED_FILE_IDS`, which restricts +selection to explicitly approved files (unset or empty = unrestricted). + +**These names are FAO's.** Each partner's consumer reads its own — CRAF'd's is +`APPWRITE_CRAFD_QUARANTINED_FILE_IDS`. Setting the wrong partner's variable is a +**silent no-op**: nothing errors, and the data keeps serving. -- **Intended** (operator decision, 2026-08-02): a delivery found to be materially wrong - should be *removed or blocked* so it can no longer be served. Data known to be wrong - should not stay retrievable through the production interface. -- **In force**: supersession. The manifest-last commit ordering means a run is replaced - by publishing a *new complete run*; the old one stays in the bucket and remains - retrievable by a client that asks for it specifically. +**Withdraw first, then diagnose.** Steps 1–4 below are the correction; the quarantine is +the stop-the-bleeding move that precedes them, and it is what makes the rest unhurried. -Supersession is in force **because it is what the wire does**, not because it was chosen. -Withdrawal has no mechanism today: it needs an ADR-013 amendment and matching work in -views-faoapi. Put to FAO as **Pre-Release Note 07, Decision Point B.2**, which also asks -whether they have an audit or reproducibility requirement that would argue *against* -withdrawal — that answer decides whether the amendment is worth building. +### Then correct: withdrawal vs supersession + +- **Withdrawal** — the intended policy (operator decision, 2026-08-02): data known to be + wrong should not stay retrievable. **Available now**, via the quarantine above. +- **Supersession** — what the *wire itself* does: manifest-last ordering means a run is + replaced by publishing a new complete run, and the old one stays in the bucket. + +The two compose: quarantine withdraws immediately, publishing supersedes durably. Put to +FAO as **Pre-Release Note 07, Decision Point B.2**, which asks whether they have an audit +or reproducibility requirement arguing *against* withdrawal. + +> ⚠ **Note 07 Topic B.2 needs correcting before it goes to FAO.** It presents withdrawal +> as *"Not available today. Requires a change to the delivery contract … and +> corresponding work in the API layer."* That is false — see above. The decision FAO is +> being asked to make is real, but the cost framing given to them is not. + +### Publishing the correction + +With the bad run withdrawn, this part is unhurried: 1. Fix the cause — the lookup, the producer, or the code — and land it. 2. Rebuild any affected artifact. For the lookup: @@ -109,13 +194,19 @@ withdrawal — that answer decides whether the amendment is worth building. ### What the consumer will actually pick up — read this before assuming -views-faoapi resolves **the newest manifest matching a broad filter** -(`wire/source_selection.py`). So a corrected run is picked up because it is *newer*, not -because it is *correct*. Two consequences: +views-faoapi resolves **the newest manifest matching a broad filter** — its own +`src/views_faoapi/managers/prediction/manager.py::get_latest_manifest`, minus anything +quarantined. So a corrected run is picked up because it is *newer*, not because it is +*correct*. Two consequences: + +*(Corrected 2026-08-03: this cited `wire/source_selection.py`. That file is **this** +repo's Hop-A **inbound** selector — how we consume pipeline-core forecasts — not how FAO +selects. views-faoapi has no `wire/` package at all.)* - **Publishing the correction is what switches the consumer over.** There is no - "activate" step and no rollback — republishing the old run would mean publishing it - again under a new id. + "activate" step in the *publishing* path — but there IS a rollback: quarantine the + bad run's documents (above) and the consumer falls back to the previous ones. + Republishing an old run, by contrast, would mean publishing it again under a new id. - **Recency-based selection is register C-73**, open, and fixed upstream in views-pipeline-core 3.0.0 but not yet taken here. **#133** would have the manifest declare `{maturity, source, required-schema-version}` so a consumer could select on @@ -158,7 +249,9 @@ an inconsistent first message to a partner is harder to correct than a slow one. ## 5. Preserve the evidence -A retraction that destroys the evidence makes the post-mortem impossible. Keep, at +A withdrawal that destroys the evidence makes the post-mortem impossible — which is +why quarantine is the right instrument: it removes the run from *selection* without +removing it from the bucket. Keep, at minimum: - the superseded run's `run_id`, its manifest, and its provenance record @@ -177,6 +270,8 @@ leaves the hole. `docs/ADRs/013_sampled_forecast_wire_contract.md` §4 (commit ordering), §11 (transition rules) · `views_postprocessing/delivery/provenance.py` (the fields a delivery carries) · -`views_postprocessing/contract/wire/source_selection.py` (how the consumer chooses) · +`views_postprocessing/contract/wire/source_selection.py` (how **this repo** chooses its +Hop-A input — *not* how FAO chooses; that is views-faoapi's +`managers/prediction/manager.py`) · register **C-22**, **C-43**, **C-60**, **C-73** · issues **#15**, **#131**, **#133**, views-datafactory **#387** diff --git a/reports/technical_risk_register.md b/reports/technical_risk_register.md index 787473f..d54c5ce 100644 --- a/reports/technical_risk_register.md +++ b/reports/technical_risk_register.md @@ -5,8 +5,8 @@ | Project | views-postprocessing | | Owner | Dylan Pinheiro / PRIO MD&D Team | | Last Updated | 2026-08-03 | -| Total Concerns | 79 | -| Open Concerns | 19 | +| Total Concerns | 82 | +| Open Concerns | 22 | | Resolved Concerns | 60 | --- @@ -41,7 +41,7 @@ covered a single open entry (see Historical clusters below). **Root cause:** a family of entries whose entire risk statement was "unverified until the first global run" — all keyed to one event, which occurred **2026-07-27**. **Entries:** C-43 (the survivor), C-30 + C-34 (merged, discharged), C-32 (discharged), C-25 (residual), D-12 and D-09 (deferral conditions) **Highest tier:** 2 (C-43) -**Fix strategy:** one post-run-0 verification pass against producer run `rusty_bucket_forecasting_20260727_095355` — issue **#131 q1**. +**Fix strategy:** one post-run-0 verification pass against producer run `rusty_bucket_forecasting_20260727_095355` — issue **#131 q1**, **CLOSED 2026-07-31**. C-43 resolved 2026-08-02; this cluster is discharged. **Resolution scope:** Full for C-30/C-32/C-34. **Partial for C-43 — the finding that matters.** Run-0 discharged the *availability* half of this cluster (the path runs, memory is bounded at 5.6 GB, coverage is proven at 64,742 cells). It discharged **none of the correctness half**, because proving the path *runs* at scale was never what C-43 asked for. **C-43 now stands alone and un-gated, with delivered data in the partner store.** ### Cluster I: Governance-artifact drift @@ -160,13 +160,25 @@ that indexes only deleted code is noise. | Tier | 3 | | Source | `repo-assimilation` (2026-06-02) | | Trigger | When `views-pipeline-core` updates its dependency tree (e.g., drops `geopandas` or `joblib`), verify that this package's imports still resolve | -| Location | `pyproject.toml:11-15`; `views_postprocessing/unfao/managers/unfao.py:14`, `unfao/enrichment.py:26`, `unfao/extraction.py:26` | +| Location | `pyproject.toml` (the dependency block); the partner managers, which are the only modules importing `views_pipeline_core`. *(This row previously cited `unfao/enrichment.py` and `unfao/extraction.py` — both moved or deleted by #151/#153.)* | `mapping.py` directly imports `geopandas`, `shapely`, `numpy`, `pandas`, `joblib`, and `multiprocessing`. `unfao.py` directly imports `pandas`, `polars`, and `python-dotenv`. Only `views-pipeline-core` and `cachetools` are declared in `pyproject.toml`. The undeclared dependencies presumably arrive transitively via `views-pipeline-core`, but this coupling is implicit and fragile. If the upstream package refactors its dependency tree, this package will break with `ImportError` at install time. **Update 2026-06-24 (narrowed):** the `mapping.py` dimension is gone (C-39 — the `geopandas`/`shapely`/`joblib`/`multiprocessing` imports were deleted; `cachetools` dropped from `pyproject.toml`). Residual: `unfao.py` imports `pandas`/`polars`/`python-dotenv` undeclared, arriving transitively via `views-pipeline-core` (which *is* declared). Much smaller surface (Tier 4-ish); consider resolving outright if the transitive-via-pipeline-core guarantee is deemed sufficient. -**Update 2026-08-01 (`falsify`) — a SECOND undeclared dependency, and this one's transitive path is about to disappear.** `appwrite` is used in this repo (`contract/launch_config.py`, `unfao/managers/unfao.py`) and declared in **no** manifest — it arrives transitively via `views-pipeline-core`, exactly as `pandas` does. What makes it different from the pandas residual: **views-pipeline-core is making `appwrite` an optional extra** (their **#345**, on CRP grounds — three repos that never mention Appwrite currently install its SDK). **When that lands, the transitive path disappears and this repo breaks at import.** +**Update 2026-08-01 (`falsify`) — a SECOND undeclared dependency, and this one's transitive path is about to disappear.** `appwrite` is used in this repo (`contract/launch_config.py`, `unfao/managers/unfao.py`) and declared in **no** manifest — it arrives transitively via `views-pipeline-core`, exactly as `pandas` does. What makes it different from the pandas residual: **views-pipeline-core is making `appwrite` an optional extra** (their **#345**, on CRP grounds — three repos that never mention Appwrite currently install its SDK). **When that lands, the transitive path disappears.** + +**⚠ Corrected 2026-08-03 — the trigger has FIRED and the stated failure mode was wrong.** +pipeline-core **#345 is CLOSED**, and 3.0.0 does make `appwrite` an optional extra. But +this repository contains **zero** direct Appwrite SDK imports (`grep -rn '^\s*\(from\|import\) appwrite' views_postprocessing/` → 0); +`contract/launch_config.py` mentions the word once, in a docstring naming its sibling +`appwrite_env`. So "this repo breaks at import" was never true of *our* imports. + +The real and still-live risk is one level out: **pipeline-core's own `DatastoreModule` +imports the SDK unguarded**, so bumping to 3.0.0 without declaring the `appwrite` extra +breaks the delivery at import — inside a dependency, which is harder to diagnose than a +break in our own code. That is a precondition on the C-44 bump, and it belongs there as +much as here. So this entry is no longer "Tier 4-ish, resolve if the transitive guarantee is deemed sufficient" — the guarantee is being **withdrawn upstream, deliberately**. Fix is one line: declare `appwrite` in `pyproject.toml`, or depend on `views-pipeline-core[appwrite]`. Relayed in **#172**; registered here rather than as a new entry because it is the same problem type at a new location. **New trigger: before views-pipeline-core#345 lands.** @@ -196,13 +208,13 @@ So this entry is no longer "Tier 4-ish, resolve if the transitive guarantee is d | Tier | 3 | | Source | `expert-review` (2026-06-02), `falsification-audit` (2026-06-02) | | Trigger | When wiring pipeline-core #245's structured metadata field, or when adding/removing a provenance key — verify the closed keyset in `delivery/provenance.py` and the `DESCRIPTION_MAX` bound still hold, and that the carrier is no longer free-text `description` | -| Location | `views_postprocessing/delivery/provenance.py`; `views_postprocessing/unfao/managers/unfao.py:559-578` (legacy `_save` uploads), `:593-604` (`_historical_frame_description`), `:619` (`_delivery_description`) | +| Location | `views_postprocessing/delivery/provenance.py` (`build_provenance`); `_historical_frame_description` in each partner's manager (the only caller); `contract/wire/sink.py` (the forecast leg, which attaches none). Symbols rather than line numbers — the earlier row's three line ranges were all past end-of-file. | Both `dsm.upload_data()` calls in `_save()` carry metadata: `name`, `loa`, `type`, `targets`, `description`, `category`. The `description` field was updated from a hardcoded test string to an enrichment timestamp (`"Enriched with geographic metadata on {timestamp}"`). However, broader enrichment provenance is still missing: no shapefile version/hash, no enrichment error count, no unmapped cell count. The consumer cannot verify which shapefile version produced their data or whether any errors occurred during enrichment. Tier recalibrated from 4 to 3 during falsification audit (2026-06-02): the missing provenance affects the partner's ability to audit data quality. -**Mitigation landed (S5, 2026-06-26, `sprint/fao-input-integrity`):** a representation-free `delivery/provenance.py` (`build_provenance`) assembles structured provenance — `lookup_version`, `region`, `expected_cell_count`, `actual_cell_count`, `unmapped_count` — sourced from the enricher + S1 coverage + a new `extraction.unmapped_cell_count` seam (nothing hardcoded). Both `_save` uploads now carry it via `_delivery_description`. **Carrier constraint:** pipeline-core's `upload_data` exposes **no structured field** — only free-text `description` — so the dict is JSON-encoded into `description` behind a human prefix for now. A dedicated metadata field is requested upstream (**pipeline-core #245**); when it lands, only the manager's attach step changes (the provenance shape is already representation-free). `fill_count` is omitted until a fabricated-value count is available (cf. C-26). Residual is now just the carrier abuse, tracked by #245. +**Mitigation landed (S5, 2026-06-26, `sprint/fao-input-integrity`):** a representation-free `delivery/provenance.py` (`build_provenance`) assembles structured provenance — `lookup_version`, `region`, `expected_cell_count`, `actual_cell_count`, `unmapped_count` — sourced from the enricher + S1 coverage + a new `extraction.unmapped_cell_count` seam (nothing hardcoded). The historical upload carries it via `_historical_frame_description`. *(This sentence said "Both `_save` uploads now carry it via `_delivery_description`" — that method was deleted with the legacy path in #149, and there is now only one provenance-carrying upload: the forecast leg's shards carry `{name, category, loa}` and no `description` at all, which is the residual below.)* **Carrier constraint:** pipeline-core's `upload_data` exposes **no structured field** — only free-text `description` — so the dict is JSON-encoded into `description` behind a human prefix for now. A dedicated metadata field is requested upstream (**pipeline-core #245**); when it lands, only the manager's attach step changes (the provenance shape is already representation-free). `fill_count` is omitted until a fabricated-value count is available (cf. C-26). Residual is now just the carrier abuse, tracked by #245. See also C-14 (stale cache without version tracking), C-22 (no post-delivery correction process), C-26 (fabricated zeros — the eventual `fill_count` source). @@ -216,7 +228,7 @@ See also C-14 (stale cache without version tracking), C-22 (no post-delivery cor | Tier | 3 | | Source | `falsification-audit` (2026-06-02) | | Trigger | When views-faoapi implements the Release-Note-01 Topic-C renaming layer — verify this repo's column names stay **unchanged** (faoapi's `_METADATA_COLS` validation depends on them) and that the rename lands consumer-side only. Take no action here otherwise. | -| Location | `views_postprocessing/unfao/managers/unfao.py:277` (`filter_cols`), `unfao/gaul_schema.py` (`METADATA_COLS`), FAO Release Note 01 `topic_c.tex` | +| Location | `contract/gaul_schema.py` (`METADATA_COLS`, the declared 9-column contract); `contract/historical.py` and `contract/wire/sidecar.py` (which project it); FAO Release Note 01 `topic_c.tex`. *(This row previously cited `unfao.py:277 (filter_cols)` — no such symbol exists anywhere in the package, and `unfao/gaul_schema.py` moved to `contract/` in #153.)* | The FAO API contract (Release Note 01, Topic C, confirmed and locked) specifies: UN M49 country codes, `ADM1_CODE`/`ADM1_NAME`/`ADM2_CODE`/`ADM2_NAME` for admin fields, and `lat`/`lon` for coordinates. The postprocessor's `filter_cols` uses: `country_iso_a3` (ISO Alpha-3), `admin1_gaul1_code`/`admin1_gaul1_name`/`admin2_gaul2_code`/`admin2_gaul2_name`, and `pg_xcoord`/`pg_ycoord`. Three of four data categories (country ID, admin fields, coordinates) use different naming conventions from the locked contract. @@ -621,6 +633,94 @@ Cross-refs: **C-59** and **C-61** (RESOLVED — the invariant block this sits be +--- + +### C-80: The doc-accuracy scan exempts ADRs and CICs — the two artifact classes that define the contracts + +| Field | Value | +|-------|-------| +| ID | C-80 | +| Tier | 2 — structural, with a demonstrated failure. A CIC is what a contributor reads before changing a class; an ADR is what a consumer reads before building against the wire. Both were free to describe deleted code indefinitely, and did. | +| Source | `code-review max` (2026-08-03) — development→main sync audit | +| Trigger | When the next module is moved or deleted, check whether any ADR or CIC names it. The deleted-symbol regex will not tell you. #153 moved seven modules out of `unfao/` and the ADRs still cite the old paths. | +| Owner | Whoever next extends `tests/test_doc_accuracy.py`. It is a scope change plus a decision about how to exempt genuine history. | +| Location | `tests/test_doc_accuracy.py` — `_living_docs()` and `_link_checked_docs()` | + +`_living_docs()` returns `README.md`, `docs/architecture/*.md`, and package `README.md`s. **`docs/ADRs/` and `docs/CICs/` are outside it**, deliberately — an ADR legitimately records superseded designs, and a scan that fires on history gets deleted (§3). The exemption is right in principle and far too wide in practice. + +**What it cost, measured in this sync.** `docs/CICs/UNFAOPostProcessorManager.md` named `GaulLookupEnricher` as the manager's enrichment collaborator in **six** places, one of them a specific call — while the manager contains zero references and `tests/test_gaul_lookup_access.py` actively asserts its absence. The sibling CIC said the opposite in plain words. Two contract documents contradicted each other about the same call, and nothing could see it. Five further claims in the same file described a `dotenv` load that does not happen, an env-validation "known gap" that C-19 closed, an upload count wrong in three ways, and two "incorrect usage" examples for code deleted in #149/#152. ADR-013 still cites `unfao/wire/`, `unfao/product.py` and `unfao/launch_config.py`, all moved in #153. + +**The exemption is not understood by the people writing under it.** `docs/CICs/UNFAOPostProcessorManager.md` carries a `legacy-ok` marker — the line-scoped opt-out from a scan that never reaches that file. Its author believed they were suppressing a guard that was not looking. + +**A second, narrower hole in the same file.** `test_internal_doc_links_resolve` follows only markdown `](...)` links. Every path written as prose in backticks — which is how this repository writes paths almost everywhere — is unchecked. That is why the stale `unfao/...` references survived a dedicated sweep (S11) and were still being found two epics later. + +*Not proposed as a fix here:* pointing the existing regex at ADRs would fire on every historical passage and be reverted within a day. The shape that works is what §3 already recommends — check the **claim**, not the vocabulary: for CICs, that every collaborator named is actually referenced by the class (the negative form already exists at `test_gaul_lookup_access.py:156`); for backticked paths, that a path-shaped token which looks like a repo path resolves, with an opt-out for history. + +Cross-refs: **C-74** (a guard narrower than its declared surface), **C-78** (a guard whose declared scope missed a package), **C-67** (ADR-012 drift, which *is* covered and was caught), ADR-014 §1–§3, #211. + +--- + +### C-81: What actually gates `main` is weaker than it looks — CI verifies 17 fewer tests than local, and nothing requires it to pass + +| Field | Value | +|-------|-------| +| ID | C-81 | +| Tier | 2 — the guards this arc built to catch cross-repo drift do not run where drift happens, and the branch they protect has no required check. Both halves are structural and both have fired-in-practice evidence. | +| Source | `code-review max` (2026-08-03) — development→main sync audit | +| Trigger | **Coverage half:** when the Appwrite Seam Contract registry next moves — it moved twice on 2026-08-03 alone — nothing in CI will notice; only a maintainer running the suite locally will. **Enforcement half:** the first time someone merges a red PR to `main`. | +| Owner | Simon — both halves need operator action. The coverage half needs a token for two private repositories; the enforcement half is a GitHub console/ruleset change. Neither is engineering work. | +| Location | `.github/workflows/run_pytest.yml`; the `protect_main` ruleset; `tests/conftest.py::sibling_repo` | + +**Coverage.** Measured in an isolated clone, not estimated — **402 collected in every run**, so the whole delta is skips: + +| environment | result | +|---|---| +| local, all siblings present | 362 passed / 40 xfailed / **0 skipped** | +| CI as it was | 347 passed / **17 skipped** / 38 xfailed | +| CI with the views-crafdapi checkout added | 348 passed / **16 skipped** / 38 xfailed | + +*(**This table was wrong twice, and the second time it refuted itself.** Draft one said 361/40 and "same 401" — measured before the same change added a test. Draft two fixed the collected figure to 402 and did not re-derive the rows, so both rows summed to 401 beside an assertion that 402 was collected. The cause of the second error is worth recording: the measurement was taken on a `git clone` of the branch, and a clone carries **committed** state — the new tests were still uncommitted in the working tree. Measuring a claim about your own change requires applying your own change. This is the entry about miscounted tests.)* `sibling_repo` resolves `$VIEWS_` else `../`; in a one-repo checkout neither exists and the tests skip. Skipping is correct behaviour — a missing sibling *is* normal — but the consequence is that **CI verifies strictly less than a developer's laptop, precisely on the assertions that cross a repository boundary.** + +Nine of the seventeen are **new in this arc**, including both registry-drift detectors (pinned edition, commit-reachable-from-`main`) for both partners. Those detectors have a demonstrated drift rate: they fired **twice on 2026-08-03**, hours apart. A detector for a fault that recurs twice in a day, running only on one machine, is most of the way to not existing. + +Where each sibling stands, after trying them: +- **views-crafdapi** — public, its check reads source text. **Now checked out in CI**, recovering **one** test: the cross-seam consumer-document-name pin for CRAF'd. +- **views-datafactory** — public, but its eight tests need the producer's raw GAUL parquets, which are **not in its git repository**. Checking it out converts an honest skip into a `FileNotFoundError`; tried and reverted. +- **views-appwrite**, **views-faoapi** — **private**. The most valuable checks live here. Closing this needs a token in CI. + +**Enforcement.** `main` is **not branch-protected**: `gh api .../branches/main/protection` returns `404 Branch not protected`, and `gh api .../rules/branches/main` returns `[]`. The `protect_main` ruleset exists and is `active`, but its `ref_name` include-list is **empty**, so it matches nothing — and it declares no `required_status_checks` rule in any case. **A red `Run Pytest` would not block a merge to `main`.** This repository's own `tests/test_falsification_campaign_4_1.py` carries the question as an unverifiable xfail probe; it is verifiable through the API, and the answer is no. + +The two compound: a suite that checks less than you think, and no requirement that even that much passes. Neither is caused by this sync — both are pre-existing — but this sync is the first time `main` receives an epic whose value is largely the guards themselves. + +Cross-refs: **C-46** and **C-57** (both RESOLVED; this is the residual each recorded as *"a CI-cost and cross-repo-coupling decision"* and *"worth deciding once for both"* — it now has a live home and a concrete answer per sibling), **C-80** (the other verification gap found in the same audit), #188. + +--- + +### C-82: Governance-artifact prose carries numbers and statuses that nothing checks + +| Field | Value | +|-------|-------| +| ID | C-82 | +| Tier | 3 — no delivery is affected, but these are the artifacts people plan from. One instance materially under-scopes a planned dependency bump. | +| Source | `code-review max` (2026-08-03) — development→main sync audit | +| Trigger | When the pipeline-core 3.0.0 bump (C-44) is scoped from Cluster M's summary rather than from C-72's body, or when anyone counts on a test-count or issue-state stated in the register. | +| Owner | Whoever runs the next `review-rr` pass; this is curation, not engineering. | +| Location | `reports/technical_risk_register.md` (Clusters I, J, M; D-09, D-11); `docs/CICs/*.md` front matter | + +`tests/test_register_integrity.py` checks structure — header counts, section placement, reference resolution — and **no prose at all**. Roughly twenty-five statements drift beneath it. + +**The one that would change a decision.** Cluster M declares resolution *"Full for … C-72 …"* at the pipeline-core 3.0.0 bump, while C-72's own body says its fix is gated on pipeline-core **#280** (open), **changes delivered wire bytes**, and requires a coordinated three-repo re-vendor of the ADR-013 §10 golden fixture. Someone planning that bump from the cluster summary under-scopes it badly. Cluster M's heading also says six entries where its body says five. + +**Self-contradiction elsewhere.** Cluster I still argues that *"there is no equivalent for the register — a small `tests/test_register_integrity.py` … would make this class self-detecting"*; that file exists, has ten green tests, and is cited elsewhere in the same document. Cluster J names issue **#15** as its fix strategy; #15 is closed and superseded by `docs/operations/correction_procedure.md`. D-11 says a branch *"currently has no scheduled deletion PR"* two paragraphs after recording that it was deleted. D-09's `Status` row reads *"Open … after delivery"* directly above prose recording the deferral expired on 2026-07-31. + +**Numbers.** The `test_gaul_lookup_fidelity.py` count appears as **26** twice in the register and as **18** twice more including `test_register_integrity.py`'s own docstring; the actual is **24**, and 26 was never true — it was written when the file held 24. Also *"40 ADR-013 guard tests"* (39) and *"`test_enrichment.py`, 16"* (39). + +**CIC front matter.** `GaulLookupEnricher.md` says *Last reviewed 2026-06-18* and `UNFAOPostProcessorManager.md` *2026-06-02*, while both bodies carry 2026-08 content. A reader calibrating trust from the header calibrates it wrong in the safe direction, which is lucky rather than designed. + +*The general fix is C-80's, not a re-count:* prose that states a number is a claim, and a claim needs a check. Where a number cannot be checked, the honest move is to state the command that produces it — which is what C-33 was forced into after its measurement was wrong five times. + +Cross-refs: **C-80** (the same disease in ADRs and CICs, and the mechanism that would catch both), **C-72** and **C-44** (the bump this mis-scopes), **C-33** (the worked example of publishing the command instead of the result), ADR-014 §1. + --- ### C-79: `_ContractStorePort.upload`'s result check is called "the whole mechanism" and has no test, and it fails open @@ -693,7 +793,7 @@ Three maintainer-raised intents, assessed and **deliberately deferred** — all **Re-open trigger:** Run 0 verified AND retention owner named — then sequence 2→3 (or 2 alone) as an infrastructure epic, and 1 whenever wire churn is calm. See also C-40 (the migration this rides on), ADR-013 §8. -**Status 2026-07-31 (review-rr — trigger HALF fired):** **Run 0 delivered** on 2026-07-27 (first FAO global-land forecast, frame-native, no OOM) — but it is **delivered, not yet verified**: issue #131 q1 (manifest integrity, sidecar/parity, 3 targets × 36 months, coverage gate on both frames) is still open, and #131 also surfaced a liveness dialect gap on the `unfao_delivery` forecast surface. **The retention owner is still unnamed** (ADR-013 §3.5 records the duty as OPEN). Both halves must hold before this re-opens, so it stays deferred — but it is now one open verification away, not one delivery away. Note that intent 2 (move `production_forecasts` off Appwrite) and the unnamed retention owner compound: run-0 added ~110 objects in a single run to a store with no retention policy. +**Status 2026-08-03 (both halves re-checked): #131 is CLOSED (2026-07-31), so the verification half of this trigger HAS fired.** The text below was written the day it closed and was already stale; it is corrected rather than deleted because the deferral it holds shut is a live decision. **Run 0 delivered** on 2026-07-27 (first FAO global-land forecast, frame-native, no OOM) and its integrity verification is closed — and #131 also surfaced a liveness dialect gap on the `unfao_delivery` forecast surface. **The retention owner is still unnamed** (ADR-013 §3.5 records the duty as OPEN), and that is now the *only* thing holding this deferral shut. Both halves must hold before it re-opens; one of the two now does. **Naming a retention owner re-opens D-12** — that is an operator decision, not engineering work. Note that intent 2 (move `production_forecasts` off Appwrite) and the unnamed retention owner compound: run-0 added ~110 objects in a single run to a store with no retention policy. --- @@ -800,14 +900,16 @@ The document states both verbatim and instructs the reader to stop and ask rathe `docs/CLONING.md` carries the same warning forward: a clone should answer its partner's correction questions **before** first delivery. This repo shipped run-0 on 2026-07-27 with that step undecided, and it still is. | | Tier | 3 | | Source | `falsification-audit` (2026-06-02) | -| Trigger | When the run-0 integrity verification (#131 q1) or any FAO/faoapi query surfaces a suspect delivered value — follow the correction procedure; **issue #15 must produce one first.** Re-check at every subsequent delivery until it exists. | +| Trigger | When any FAO/faoapi query surfaces a suspect delivered value — follow `docs/operations/correction_procedure.md`, **withdrawing first** via views-faoapi's quarantine before diagnosing. *(This row previously said "issue #15 must produce one first" and named #131 q1 as a precondition. #15 is CLOSED and superseded by the procedure; #131 closed 2026-07-31. The procedure exists — the trigger is now the incident, not the paperwork.)* | | Location | `views_postprocessing/unfao/managers/unfao.py:442-494` (`_save_contract`), `:518-578` (legacy `_save`); issue #15 (the undocumented procedure) | The delivery chain has four stages beyond the code: Appwrite bucket → UN FAO download → FAO systems → operational decisions. When an error is discovered post-delivery, correction requires clearing cache, re-running, re-uploading, notifying FAO, and FAO retracting old data. Steps 3-5 have no documented procedure. Part of Cluster B (operational impact dimension). See also C-14 (RESOLVED — mapper-era cache), C-15. -**Update 2026-07-31 (review-rr — the conditional is spent):** this entry was written conditionally — "*if* wrong data ever reaches FAO." **Run-0 delivered on 2026-07-27** (108 arrow shards + sidecar + manifest to `unfao_bucket`, plus 28,356,996 historical rows at 64,742 cells), and its integrity verification is still open (#131 q1). There is now delivered, unverified data in the partner's store and still no documented correction/recall procedure. Tier held at 3 (process gap, no code defect), but this is the acute member of Cluster J — **issue #15 is now the blocking artifact, not a nice-to-have.** +**Update 2026-07-31 (review-rr — the conditional is spent):** this entry was written conditionally — "*if* wrong data ever reaches FAO." **Run-0 delivered on 2026-07-27** (108 arrow shards + sidecar + manifest to `unfao_bucket`, plus 28,356,996 historical rows at 64,742 cells). There is delivered data in the partner's store. + +**Corrected 2026-08-03.** The paragraph above ended *"its integrity verification is still open (#131 q1) … still no documented correction/recall procedure … issue #15 is now the blocking artifact."* All three are spent: **#131 closed 2026-07-31**, the procedure landed as `docs/operations/correction_procedure.md`, and **#15 is closed and superseded by it**. What remains genuinely open is narrower and is in the procedure's §4: FAO has not yet answered who else to notify, in what period, and whether they want withdrawal or supersession (Pre-Release Note 07, Topic B). --- @@ -947,7 +1049,7 @@ The right axis was **exact equality on string constants**, not statement shape. **Gated, and honestly so.** The checks need a views-appwrite checkout and skip without one, naming `VIEWS_APPWRITE` and the conventional sibling path so a contributor can run them rather than merely watch them skip. The would-catch-a-rename proof runs in CI with no checkout at all. Resolution helper shared with **C-46** (S7) in `tests/conftest.py` — the second incident, which is this repo's named trigger for extracting. -**Residual:** the gated half does not run in CI, which needs a views-appwrite checkout in the workflow — a CI-cost and cross-repo-coupling decision, not a code fix. Same shape as **C-46**'s residual and worth deciding once for both (S7 / #188). +**Residual — now tracked as C-81.** The gated half does not run in CI, which needs a views-appwrite checkout in the workflow. That was recorded here and in **C-46** as *"a CI-cost and cross-repo-coupling decision, not a code fix … worth deciding once for both"*, and it sat as a residual on two RESOLVED entries, which is where residuals go to be forgotten. It now has a live entry with a measured cost (17 tests, 9 of them new in this arc), a per-sibling answer, and an owner: **C-81**. views-appwrite is private, so it needs a token — an operator decision. A second, smaller instance of the same shape: these checks parse TOML with `tomllib`, stdlib from Python 3.11, and `pyproject` declares `>=3.11`. CI runs 3.11 and executes them. The maintainer's box runs **3.10**, below the declared floor, so they skip there — the local suite is quietly weaker than a green `pytest -q` suggests. Not a repo defect and not worth its own entry; recorded because "a gate that does not run" is exactly what C-46 is open for, and the CI decision should cover both. | | Tier | 3 | diff --git a/tests/test_doc_accuracy.py b/tests/test_doc_accuracy.py index 4a29779..3e295d0 100644 --- a/tests/test_doc_accuracy.py +++ b/tests/test_doc_accuracy.py @@ -18,6 +18,7 @@ import ast import re +import subprocess from pathlib import Path import pytest @@ -118,6 +119,32 @@ def _link_checked_docs() -> list[Path]: return out + +def test_the_readme_geography_table_matches_the_declared_contract(): + """README's 9-column table is a consumer's implementation spec — check it. + + The order is normative (ADR-013 §5.1) and byte-pinned by the §10 golden fixture, so + a reader built from a reordered table reads the wrong column. Verified 2026-08-03 + that the table had `admin1_gaul0_*` before `admin1_gaul1_*` and typed the four + `*_code` columns `int` where the wire dtype is `float64` — both wrong for months, + because nothing compared the prose to `gaul_schema.COLUMNS`. + """ + from views_postprocessing.contract.gaul_schema import COLUMNS + + rows = re.findall(r"^\|\s*`([a-z0-9_]+)`\s*\|\s*([a-z0-9]+)\s*\|", + (_REPO / "README.md").read_text(), re.M) + documented = [(name, dtype) for name, dtype in rows + if name in {c[0] for c in COLUMNS}] + declared = [(name, dtype) for name, _, dtype in COLUMNS] + + assert documented == declared, ( + f"README's geography table does not match contract/gaul_schema.COLUMNS.\n" + f" README: {documented}\n" + f" declared: {declared}\n" + "The order is normative and the dtypes are the wire's. A consumer builds a " + "reader from this table." + ) + def test_internal_doc_links_resolve(): dead = [] for doc in _link_checked_docs(): @@ -548,20 +575,36 @@ def test_the_procedure_names_who_notifies_and_still_flags_what_fao_has_not_answe ) -def test_the_procedure_distinguishes_intended_policy_from_what_is_implemented(): - """The gap that would otherwise be discovered mid-incident. +def test_the_procedure_names_the_mechanism_that_withdraws_a_bad_run(): + """The first move in an incident must be findable in the first minute. - Withdrawal is the decision; supersession is what the wire actually does, and it is - in force only because nothing else exists. An operator reading this at 22:00 must - not believe a bad delivery becomes unretrievable when it does not. + **This guard used to pin the opposite claim, and pinned it faithfully.** It asserted + the procedure said withdrawal was unbuilt and would cost "an ADR-013 amendment" — + mutation-proven, green for a day, and wrong: views-faoapi had already shipped an + operator quarantine (`f1a59bf`, on its `main`) that drops a manifest file-id from + selection and atomically falls the consumer back to the previous run. The runbook + sent an operator to republish an entire corrected run, for hours, while the wrong + data stayed live. + + ADR-014 §2 asks whether a guard bites. It does not ask whether the thing it bites on + is true — and a guard can hold a falsehood in place perfectly. What this now pins is + the mechanism itself, which is checkable against another repository rather than + against our own prose. """ text = _CORRECTION.read_text() - assert "intended policy is WITHDRAWAL" in text.replace("**", "") - assert "implemented is SUPERSESSION" in text.replace("**", "") - assert "ADR-013 amendment" in text, ( - "the procedure must say what withdrawal would COST — otherwise the gap reads " - "as an oversight rather than as unbuilt work with a known price" + assert "APPWRITE_UNFAO_QUARANTINED_FILE_IDS" in text, ( + "the procedure must name the environment variable that withdraws a run. It is " + "the only step that stops the bleeding, and an operator cannot grep for a " + "mechanism the document does not name." ) + assert "quarantine" in text.lower() + lowered = text.lower() + for token in ("withdrawal", "supersession"): + assert token in lowered, ( + f"the procedure must still distinguish {token} from its sibling — they " + "compose (quarantine withdraws now, publishing supersedes durably) and an " + "operator who conflates them will do only half the correction" + ) def test_no_partner_contact_details_are_published_in_this_repository(): @@ -573,7 +616,33 @@ def test_no_partner_contact_details_are_published_in_this_repository(): as a side effect of documenting a runbook. """ offenders = [] - for doc in (*sorted(_REPO.rglob("*.md")), *sorted(_PKG.rglob("*.py"))): + # This repository's OWN tracked files. `rglob` from the repo root also walks any + # sibling checkout CI places inside the workspace (`_siblings/`) — proven 2026-08-03 + # by planting an address in a checked-out views-crafdapi and watching this fail with + # "appear in this public repository", naming a file in a different repository. A + # commit in someone else's repo could turn `main` red here. + # + # The pathspec is the DIRECTORY, with the suffix filtered in Python. Not + # `views_postprocessing/**/*.py`: git's `**` does not match a top-level file, so that + # form silently skips anything added directly to `views_postprocessing/` — verified + # 2026-08-03 with a probe file. A security-adjacent scan on a public repository must + # not narrow by accident. + tracked = subprocess.run( + ["git", "-C", str(_REPO), "ls-files", "-z", "*.md", "views_postprocessing"], + capture_output=True, text=True, check=False, timeout=30, + ).stdout.split("\0") + scanned = [ + _REPO / name for name in tracked + if name and name.endswith((".md", ".py")) and (_REPO / name).exists() + ] + assert scanned, ( + "git ls-files returned nothing — this scan would pass over an empty set and " + "report success, which is register C-74's exact shape. If this repository is " + "not a git checkout, the guard cannot run and must say so rather than pass." + ) + for doc in sorted(scanned): + if doc.resolve() == Path(__file__).resolve(): + continue # this file names the pattern in order to ban it if ".git" in doc.parts: continue for number, line in enumerate(doc.read_text(errors="ignore").splitlines(), 1): diff --git a/tests/test_env_declaration.py b/tests/test_env_declaration.py index ad34e8e..8be6bd1 100644 --- a/tests/test_env_declaration.py +++ b/tests/test_env_declaration.py @@ -26,6 +26,7 @@ import ast import logging import re +import subprocess from pathlib import Path import pytest @@ -42,7 +43,8 @@ from views_postprocessing.crafd import appwrite_env as crafd_env from views_postprocessing.unfao import appwrite_env -_PKG = Path(__file__).resolve().parent.parent / "views_postprocessing" +_REPO = Path(__file__).resolve().parent.parent +_PKG = _REPO / "views_postprocessing" def _manager_source(partner: str) -> Path: @@ -110,6 +112,10 @@ def _manager_source(partner: str) -> Path: _PARTNERS = tuple(_PARTNER_ENV) +#: Every coordinate name any partner declares — the left-hand sides a document could +#: assign a registry value to. +_EXPECTED_NAMES = {name for _, _, expected in _PARTNER_ENV.values() for name in expected} + #: Function names that belong to python-dotenv and to essentially nothing else. #: @@ -603,8 +609,55 @@ def test_no_coordinate_value_is_copied_into_this_repo(): and id(node) not in docstrings ): copied.append(f"{source.relative_to(_PKG)}:{node.lineno} = {node.value!r}") + # Markdown too — the AST half cannot see a fenced ``bash`` block, and that is exactly + # where four production-forecasts values sat: in README.md's Configuration section, + # two lines below the sentence promising they are never copied, in a PUBLIC + # repository. A guard scoped to `.py` while the rule is about the repository is + # register C-74's shape (2026-08-03). + # + # **What counts as a copy, and what does not.** A first draft flagged any line + # containing a registry value and immediately fired on a dozen documents that merely + # *name* a store in prose — "six stranded documents in unfao_bucket". That is not a + # copy; it is a sentence. C-57 recorded the identical false-positive class over `.py` + # and the identical lesson: when a guard cries wolf, the matching is wrong before the + # scope is (ADR-014 §3). + # + # The copy is a value **assigned to its own coordinate name** — `APPWRITE_X=value` — + # which is a reader's instruction to configure with that literal. That is precise + # enough to have caught README.md and to ignore every legitimate mention. + assignment = re.compile( + r"^\s*(?:export\s+)?(" + "|".join(sorted(_EXPECTED_NAMES)) + r")\s*=\s*(.+?)\s*$" + ) + # This repository's OWN tracked markdown — `git ls-files`, not `rglob`. CI checks + # sibling repositories out into the workspace, and their documents are not this + # repo's to police; an rglob would scan them and fail on someone else's prose. + # + # **This half cannot run in CI**, and that is worth stating rather than discovering. + # The registry values come from the views-appwrite checkout, which is private and + # deliberately not checked out — so the whole test skips there. It guards a + # maintainer's commit, not the merge. Closing that is C-81's token decision. + tracked = subprocess.run( + ["git", "-C", str(_REPO), "ls-files", "-z", "*.md"], + capture_output=True, text=True, check=False, timeout=30, + ).stdout.split("\0") + scanned = [_REPO / name for name in tracked if name and (_REPO / name).exists()] + assert scanned, ( + "git ls-files returned no markdown — the scan would pass over an empty set and " + "report success (register C-74's shape). If this is not a git checkout, the " + "guard cannot run and must say so rather than pass." + ) + for doc in sorted(scanned): + for number, line in enumerate(doc.read_text().splitlines(), 1): + match = assignment.match(line) + if match and match.group(2).strip('"\'') in values: + copied.append( + f"{doc.relative_to(_REPO)}:{number} = {match.group(2)!r} " + f"(assigned to {match.group(1)})" + ) + assert not copied, ( - f"coordinate value(s) from the registry are copied into code: {copied}. The " + f"coordinate value(s) from the registry are copied into this repo: {copied}. The " "registry is referenced, never copied — values reach this package through the " - "environment the launcher assembles, validated by assert_env_declared." + "environment the launcher assembles, validated by assert_env_declared. In a " + "document, write the NAME and leave the value to the launcher." ) diff --git a/views_postprocessing/unfao/managers/README.md b/views_postprocessing/unfao/managers/README.md index b3aa405..701f8f1 100644 --- a/views_postprocessing/unfao/managers/README.md +++ b/views_postprocessing/unfao/managers/README.md @@ -91,5 +91,5 @@ Appwrite). - [`role_and_seams.md`](../../../docs/architecture/role_and_seams.md) — role + seams - [CIC: UNFAOPostProcessorManager](../../../docs/CICs/UNFAOPostProcessorManager.md) — class contract -- [CIC: GaulLookupEnricher](../../../docs/CICs/GaulLookupEnricher.md) — the enrichment join +- [CIC: GaulLookupEnricher](../../../docs/CICs/GaulLookupEnricher.md) — the build/verification path for the GAUL lookup. **Not on this manager's call graph** (register C-75); geography is attached by `contract/historical.py` and `contract/wire/sidecar.py` - ADR-011 (mapper → lookup), ADR-012 (current ontology)