Skip to content

feat(crafd): CRAF'd producer package — clone of unfao/ (per CLONING.md) - #211

Merged
Polichinel merged 2 commits into
developmentfrom
feat/crafd-producer
Aug 3, 2026
Merged

feat(crafd): CRAF'd producer package — clone of unfao/ (per CLONING.md)#211
Polichinel merged 2 commits into
developmentfrom
feat/crafd-producer

Conversation

@Polichinel

@Polichinel Polichinel commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

Stands up the views-postprocessing producer for the second consumer, CRAF'd (views-crafdapi) — the thing that will deliver CRAF'd forecasts into crafd_bucket. Per CLONING.md it's "replace three, keep everything else."

What

  • crafd/product.pyTARGETS = the same three GED fatality series (CRAF'd is FAO extended — same forecasts/geography for now; extra targets are a later Amendment A1 edit), CONSUMER_DOCUMENT_NAME = "un_crafd" (ADR-034 §6), S_MIN = 2, UPLOAD_ENABLED = False (§11.4 interlock — staged locally, zero store calls, until the views-crafdapi selection guard is deployed).
  • crafd/appwrite_env.pyCRAFD_ENV (the APPWRITE_CRAFD_* coordinates the operator filled at views-crafdapi S9 / views-appwrite CIC for ReconciliationModule + migration docs #38); connection + prod-forecasts env shared with FAO (same internal shelf source). Contract pinned v1.4.1 / 90fc105.
  • crafd/managers/crafd.pyCRAFDPostProcessorManager: the pipeline-core seam, reads the shared production_forecasts shelf and delivers the same wire to crafd_bucket. Mechanical retarget of the FAO manager; no partner-neutral machinery touched.

Second commit (6d587d1) — what five review passes found

The package above was correct in substance. What it exposed is that eight guards were scoped to unfao by name, so the new partner landed exempt from all of them.

The one real defect in the package. crafd/appwrite_env.py pinned registry v1.3.0 / 47172af — an edition at which its own four APPWRITE_CRAFD_* coordinates were declared with no values (they were filled in v1.4.0), while the file's own docstring cited the PR that filled them as its justification. unfao was already at v1.4.1 on this branch. Re-pinned. (The original description here said "same edition as unfao"; that was not true when written.)

What was unguarded, and now is not:

gap now
contract/ could import crafd — proven, suite stayed green AST import walk over the declared partner list
delivery/ -> contract/ had no test at all, while ADR-012 claimed the arrow was "enforced by test" AST walk + subprocess import-purity check
þing-01 dotenv borrow: scan covered 2 files of 30; a live load_dotenv(find_dotenv()) in unfao/appwrite_env.py ran on import, suite green package-wide AST walk (sees code, not the prose that legitimately describes the retired borrow)
line budget covered one manager; a 906-line third manager passed derived from the partner list, applied to the whole managers/ directory
nothing asserted crafd's CONSUMER_DOCUMENT_NAME pinned, plus a gated cross-seam check reading views-crafdapi's own APIPathManager literal and the field it filters on
_MACHINERY had no completeness assertion — contract/enrichment.py was exempt from both purity guards asserted against disk

The partner list is now declared once in tests/conftest.py and checked against the filesystem, so a third partner cannot be silently exempt.

A production-code correction. _ContractStorePort.upload's comment said the pipeline-core store "only LOGS" metadata failures. It doesn't — it returns OperationResult(success=False, code="PARTIAL_SUCCESS"). The comment was self-defeating: if the store only logged, success would be True and the guard the comment explains could never fire.

Records disposed of in the same change (ADR-014 §5). ADR-012's ontology, CLONING.md, role_and_seams.md, the CIC, unfao/managers/README.md, the README tree and contract/__init__.py all said "the" partner or "the only importer". CLONING.md additionally over-claimed þing-02 S24(5) as a flat prohibition — the verdict binds the repos cut from views-faoapi, not an in-repo partner package of the producer — and dropped #146's named trigger. Register: C-77, C-78, C-79 added; C-33, C-40, C-47, C-57 amended; 76/17/59 → 79/19/60.

Also retired test_version_bumped_past_latest_tag: it asserted a version is not tagged, so it went red whenever views-datafactory was in a clean released state. Its stated purpose is covered by test_land_gaul_commit_is_in_a_release_tag, which passes.

Validation

Corrected from the original description: the suite does run, and it is green. The earlier note said it could not be run on this machine (Python 3.13 vs pinned pyarrow 16.1.0). It runs in the views_pipeline conda env — Python 3.11.14, views-pipeline-core 3.0.0 editable, pyarrow 16.1.0 — where the five test_wire_shard/test_wire_sidecar failures reported earlier do not occur; they were the toolchain mismatch, as suspected.

361 passed, 40 xfailed, 0 failed
ruff check .  ->  All checks passed!

Every new or widened guard is mutation-proven — roughly 25 mutations, each reverted, including the exact defects this PR fixes (the stale v1.3.0 pin; a live load_dotenv in the crafd manager; all four import spellings at module level and inside function bodies). Sibling checkouts left untouched.

Known and registered, not fixed here

  • C-77 — the historical leg names its document from model_path.model_name, not from CONSUMER_DOCUMENT_NAME. Affects both partners and is not introduced here; fixing it changes what lands in the live FAO bucket, which is not a partner-addition PR's business. Note views-models/postprocessors/ has no un_crafd directory yet, so CRAF'd's historical name has never been resolved.
  • C-79_ContractStorePort.upload's result check has zero tests and fails open.
  • C-33 — the two managers are a deliberate WET copy with a now-named extraction trigger: a third in-repo partner, or the first bug hand-patched identically in both.

Stand up the views-postprocessing producer for the second consumer, CRAF'd
(views-crafdapi). Per CLONING.md this is "replace three, keep everything else":

- crafd/product.py — TARGETS = the same three GED fatality series (CRAF'd is FAO
  extended, same forecasts/geography; additional targets are a later Amendment A1
  edit), CONSUMER_DOCUMENT_NAME = "un_crafd" (ADR-034 §6), S_MIN = 2,
  UPLOAD_ENABLED = False (§11.4 interlock — staged locally, zero store calls,
  until the views-crafdapi selection guard is deployed).
- crafd/appwrite_env.py — CRAFD_ENV (APPWRITE_CRAFD_* coordinates, filled by the
  operator at views-crafdapi S9 / views-appwrite #38); connection + prod-forecasts
  env shared with FAO (same internal shelf source). Contract pinned at 1.3.0 /
  47172af (same edition as unfao; the CRAFD names are reserved there).
- crafd/managers/crafd.py — CRAFDPostProcessorManager: the pipeline-core seam,
  reads the shared production_forecasts shelf and delivers the same wire to
  crafd_bucket. Mechanical retarget of the FAO manager; no partner-neutral
  machinery touched.
- test_doc_accuracy: the pipeline-core-importer guard now expects BOTH partner
  managers (unfao + crafd) — a new partner manager is expected, not blast-radius
  creep; anything OUTSIDE the managers still fails.

VALIDATION: ruff-clean and AST-parses locally; the FULL suite is validated by CI,
because this machine (Python 3.13) cannot build the repo's pinned pyarrow 16.1.0
(no 3.13 wheel) — CI runs the correct 3.11/3.12 + 16.1.0 toolchain.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`crafd/` landed exempt from eight guards, because each named `unfao` as a string.
Five review passes; each found defects in the previous pass's fixes.

The defect crafd shipped with
- `crafd/appwrite_env.py` pinned registry v1.3.0/47172af — an edition at which its
  own four APPWRITE_CRAFD_* coordinates had no values, while the file's docstring
  cited the PR that filled them. Re-pinned to v1.4.1/90fc105.

What was unguarded, and is now
- partner list declared ONCE (`tests/conftest.py:PARTNER_PACKAGES`), asserted
  against the filesystem, and consumed by every guard that needs it
- þing-01 dotenv borrow: covered 2 files of 30. A live `load_dotenv(find_dotenv())`
  in `unfao/appwrite_env.py` — the entry validator whose docstring says the borrow
  is dead — ran on import with the suite green. Now a package-wide AST walk, which
  sees imports and calls and cannot see the prose that legitimately describes it
- ADR-002 both legs: `contract/` could import `crafd`, and `delivery/ -> contract/`
  had no test at all. Both are now AST import walks plus a subprocess import-purity
  check. Regexes escaped three times (relative, bare-package, function-body) before
  being replaced
- line budget: the last hardcoded list; a 906-line third manager passed it. Derived,
  and applied to the whole `managers/` directory
- consumer document name: nothing asserted crafd's. Pinned, plus a gated cross-seam
  check reading views-crafdapi's own `APIPathManager` literal and its filter field
- `_MACHINERY` had no completeness assertion, so `contract/enrichment.py` was exempt
  from both purity guards

Production code
- `_ContractStorePort.upload`'s comment claimed the pipeline-core store "only LOGS"
  metadata failures. It returns `success=False, code="PARTIAL_SUCCESS"`. The comment
  was self-defeating: if it only logged, the guard it explains could never fire

Records disposed of in the same change (ADR-014 §5)
- ADR-012 ontology, CLONING.md, role_and_seams, the CIC, unfao/managers/README,
  README tree, `contract/__init__` — all said "the" partner or "only one" importer
- CLONING.md over-claimed þing-02 S24(5) as a flat prohibition; the verdict binds the
  repos cut from views-faoapi, not an in-repo partner package of the producer. It also
  dropped #146's named trigger and asserted a tracker state #146 did not have (commented)
- register: C-77, C-78, C-79 added; C-33, C-40, C-47, C-57 amended; 76/17/59 -> 79/19/60

Also
- retired `test_version_bumped_past_latest_tag`: it asserted a version is NOT tagged,
  so it went red whenever views-datafactory was in a clean released state. Its purpose
  is covered by `test_land_gaul_commit_is_in_a_release_tag`
- `sibling_repo` briefly raised on a bad override and turned a typo into
  "0 tests run". Moved to its own assertion
- C-33's measurement was wrong six times, twice by edits that shifted lines it had
  just counted. No entry states manager line numbers now — they name symbols, and
  the docs state the `diff` command instead of its result

361 passed, 40 xfailed, 0 failed (conda env, pipeline-core 3.0.0 editable, pyarrow
16.1.0); ruff clean. Every guard mutation-proven; ~25 mutations, all reverted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Polichinel
Polichinel merged commit f41c1a2 into development Aug 3, 2026
4 checks passed
@Polichinel
Polichinel deleted the feat/crafd-producer branch August 3, 2026 10:08
@Polichinel

Copy link
Copy Markdown
Collaborator Author

Ready to merge (MERGEABLE, CI CLEAN) — and it's now on the critical path for standing up views-crafdapi (released v0.1.0). Flagging the sequencing so the operator can route it:

  1. Merge this to development, then development → main. The views-models launcher (filed as Add the un_crafd postprocessor launcher (delivers rusty_bucket → CRAF'd bucket) views-models#333) installs views-postprocessing@main and imports views_postprocessing.crafd.managers.CRAFDPostProcessorManager, so the crafd package must reach main, not just development, before the launcher can run.
  2. Keep UPLOAD_ENABLED=False for now (the §11.4 interlock). It flips to True only for the real crafd delivery — after crafdapi is deployed and a dry-run (upload disabled) has verified the producer builds the correct ADR-013 wire.

Read-side contract already verified against this package: crafdapi filters the store on model name un_crafd == this package's CONSUMER_DOCUMENT_NAME="un_crafd"; same APPWRITE_CRAFD_* bucket via the registry; producer ships raw sb/ns/os draws and crafdapi collapses + computes exceedance p_gt{25,100,1000} at serve. No blockers on the read side.

Not merging it myself — this is your repo; over to you (or the owning session) to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant