Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions .github/workflows/run_pytest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,46 @@ jobs:
- name: Checkout repository
uses: actions/checkout@v3

# ── sibling checkouts: what CI can verify that a bare checkout cannot ────────
#
# `tests/conftest.sibling_repo` resolves $VIEWS_<NAME>, else the conventional
# `../<name>` directory. In a one-repo CI checkout neither exists, so every
# cross-repo assertion skips — silently and by design, because a missing sibling
# is normal. The consequence was that CI verified strictly LESS than a developer's
# laptop, and precisely on the guards that matter most across a repo boundary.
#
# Measured 2026-08-03 (402 collected in all three runs, so the delta is entirely
# skips):
# local, all siblings present 362 passed / 40 xfailed / 0 skipped
# CI, bare checkout 347 passed / 38 xfailed / 17 skipped
# CI, with the checkout below 348 passed / 38 xfailed / 16 skipped
#
# So this recovers exactly ONE test — the cross-seam consumer-name check for CRAF'd.
# That is worth having (it is the §4.1a pin, whose failure mode is a delivery nobody
# can find) but it is one, not seventeen. The other sixteen are below.
#
# Only views-crafdapi is checked out, and the choice is deliberate:
#
# views-crafdapi PUBLIC, and its check reads source text -> enabled here.
# views-datafactory PUBLIC, but its 8 tests need the producer's raw GAUL
# parquets (`data/raw/gaul_admin/*.parquet`), which are NOT in
# its git repository. Checking it out converts an honest skip
# into a FileNotFoundError. Measured, not assumed — it was tried.
# views-appwrite PRIVATE. The registry-drift detectors live here and are the
# views-faoapi most valuable of the lot; enabling them needs a token, which
# is an operator decision. Register C-46 and C-57 both carry
# that as their standing residual, and it is still open.
# `actions/checkout` refuses a path outside $GITHUB_WORKSPACE, so the siblings go
# inside it and are pointed at by the declared env vars instead of the conventional
# `../<name>` location. `sibling_repo` checks $VIEWS_<NAME> first, precisely so the
# location can be declared rather than assumed.
- name: Checkout views-crafdapi (sibling)
uses: actions/checkout@v3
with:
repository: views-platform/views-crafdapi
path: _siblings/views-crafdapi
fetch-depth: 0

- name: Set up Python
uses: actions/setup-python@v4
with:
Expand All @@ -33,6 +73,8 @@ jobs:
poetry install

- name: Run tests
env:
VIEWS_CRAFDAPI: ${{ github.workspace }}/_siblings/views-crafdapi
run: |
set -e
poetry run pytest tests/
83 changes: 53 additions & 30 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ The **post-forecast delivery layer** for the **VIEWS** (Violence Early-Warning S
pipeline. It takes finished VIEWS forecasts, enriches them with geographic metadata,
guards their integrity, and delivers them to a partner store.

The only live delivery today is the **UN FAO** path — its product in `views_postprocessing/unfao/`, running on the partner-neutral machinery in `contract/`.
Two partner deliveries run on the same partner-neutral machinery in `contract/`: the **UN FAO** path (`views_postprocessing/unfao/`), serving FAO-FSFC since 2026-07-27, and **CRAF'd** (`views_postprocessing/crafd/`), added 2026-08-03 with its upload interlock still closed.

> **New here? Read [`docs/architecture/role_and_seams.md`](docs/architecture/role_and_seams.md) first.**
> It explains what this repo is, how it relates to pipeline-core / faoapi / datafactory,
Expand Down Expand Up @@ -50,7 +50,7 @@ Requires **Python 3.11–3.14**.
| Package | Version | Why |
|---------|---------|-----|
| `views-pipeline-core` | `>=2.1.3,<3.0.0` | The framework: lifecycle base classes, data loader, dataset container, Appwrite/datastore tools |
| `views-frames` | `>=1.0,<2` | The frame data contract — **the live delivery representation** since #126. pandas survives only in `contract/enrichment.py` (the build/verification path) |
| `views-frames` | `>=1.10.2,<2` | The frame data contract — **the live delivery representation** since #126. pandas survives only in `contract/enrichment.py` (the build/verification path) |

---

Expand All @@ -75,7 +75,7 @@ In practice the manager is constructed and run by **views-models**
| **Read** | `_read_historical_frame`, `_read_forecast_data_contract` | Historical actuals from views-datafactory arrive **frame-native** (#126); the forecast run is resolved from the Appwrite store by its **run manifest**, with each shard's header verified on load (ADR-013 §4.3). |
| **Transform** | `_transform` | Resolution only. Prediction values are **not** transformed — no collapse, no reconciliation. |
| **Validate** | `_validate`, `_check_coverage` | Asserts the read resolved, then enforces the region coverage + GAUL-excluded-cell contract (C-34 / C-30). The metadata null-gate fires later, at artifact build (`contract/historical.assert_metadata_complete`). |
| **Save** | `_save` → `_save_contract` | Builds the ADR-013 wire — arrow shards, the §5 GAUL sidecar, the historical artifact — commits the run **manifest last**, and stamps each upload with structured provenance (C-15). |
| **Save** | `_save` → `_save_contract` | Builds the ADR-013 wire — arrow shards, the §5 GAUL sidecar, the historical artifact — commits the run **manifest last**. The historical artifact carries structured provenance in its store-document `description` (C-15); the forecast leg's uploads carry `{name, category, loa, filename, doc_type, targets}` and **no `description`** — a gap, not a design. |

The pandas metadata-join and history-clip stages were retired with the legacy delivery path
in #149; their rules survive as called invariants under `delivery/`. See the
Expand All @@ -89,19 +89,28 @@ retraction primitive; a correction is a new complete run, manifest last.

### Output schema (geographic metadata columns)

These 9 columns are the delivered geography contract (declared in `contract/gaul_schema.py`):

| Column | Type | Description |
|--------|------|-------------|
| `pg_xcoord` | float | PRIO-GRID cell centroid longitude |
| `pg_ycoord` | float | PRIO-GRID cell centroid latitude |
| `country_iso_a3` | str | ISO 3166-1 alpha-3 country code |
| `admin1_gaul0_code` | int | GAUL level-0 (country) code |
| `admin1_gaul0_name` | str | GAUL level-0 (country) name |
| `admin1_gaul1_code` | int | GAUL level-1 (province) code |
| `admin1_gaul1_name` | str | GAUL level-1 (province) name |
| `admin2_gaul2_code` | int | GAUL level-2 (district) code |
| `admin2_gaul2_name` | str | GAUL level-2 (district) name |
These 9 columns are the delivered geography contract, declared in
`contract/gaul_schema.py`. **The order below is normative** (ADR-013 §5.1) and is
byte-pinned by the §10 golden fixture — a reader that reorders them reads the wrong
column. `tests/test_doc_accuracy.py` checks this table against the declaration.

| Column | Wire type | Description |
|--------|-----------|-------------|
| `pg_xcoord` | float64 | PRIO-GRID cell centroid longitude |
| `pg_ycoord` | float64 | PRIO-GRID cell centroid latitude |
| `country_iso_a3` | string | ISO 3166-1 alpha-3 country code |
| `admin1_gaul1_code` | float64 | GAUL level-1 (province) code |
| `admin1_gaul1_name` | string | GAUL level-1 (province) name |
| `admin1_gaul0_code` | float64 | GAUL level-0 (country) code |
| `admin1_gaul0_name` | string | GAUL level-0 (country) name |
| `admin2_gaul2_code` | float64 | GAUL level-2 (district) code |
| `admin2_gaul2_name` | string | GAUL level-2 (district) name |

*(Corrected 2026-08-03: this table had `admin1_gaul0_*` before `admin1_gaul1_*` —
the reverse of the normative order — and typed the four `*_code` columns `int`. They
are **float64 on the wire, always**, by the §5.1 ruling: the codes are nullable and
arrow has no nullable int in this contract. Both errors survived because nothing
compared the table to the declaration.)*

---

Expand Down Expand Up @@ -158,35 +167,49 @@ See [`docs/CLONING.md`](docs/CLONING.md).

## Configuration

The FAO delivery reads Appwrite connection settings from the environment. The required
names are **declared** in `unfao/appwrite_env.py` and validated fail-loud before any store
is constructed — a missing or empty variable raises, naming every one that is absent,
rather than half-configuring a client. Coordinates come from the Appwrite Seam Contract registry
(referenced by URL, never copied); the API key is an operator slot:
Each delivery reads Appwrite connection settings from the environment. The required
names are **declared** per partner — `unfao/appwrite_env.py`, `crafd/appwrite_env.py` —
and validated fail-loud before any store is constructed: a missing or empty variable
raises, naming every one that is absent, rather than half-configuring a client.

**The names are below; the values are not.** Coordinates live in the Appwrite Seam
Contract's registry, which this repo references by pinned URL and never copies (þing-01
sáttmál S6 — copies were the platform's original failure). The launcher supplies the
values; the API key is an operator slot.

```bash
# Appwrite connection (secrets)
APPWRITE_ENDPOINT=https://cloud.appwrite.io/v1
# Appwrite connection
APPWRITE_ENDPOINT=...
APPWRITE_DATASTORE_PROJECT_ID=...
APPWRITE_DATASTORE_API_KEY=...
APPWRITE_DATASTORE_API_KEY=... # operator-issued secret

# Production-forecasts store (input)
APPWRITE_PROD_FORECASTS_BUCKET_ID=production_forecasts
APPWRITE_PROD_FORECASTS_BUCKET_NAME=Production Forecasts
APPWRITE_PROD_FORECASTS_COLLECTION_ID=production_forecasts
APPWRITE_PROD_FORECASTS_COLLECTION_NAME=Production Forecasts
# Production-forecasts store (input — shared by every partner)
APPWRITE_PROD_FORECASTS_BUCKET_ID=...
APPWRITE_PROD_FORECASTS_BUCKET_NAME=...
APPWRITE_PROD_FORECASTS_COLLECTION_ID=...
APPWRITE_PROD_FORECASTS_COLLECTION_NAME=...

# UN FAO store (output)
APPWRITE_UNFAO_BUCKET_ID=...
APPWRITE_UNFAO_BUCKET_NAME=...
APPWRITE_UNFAO_COLLECTION_ID=...
APPWRITE_UNFAO_COLLECTION_NAME=...

# Metadata database
# CRAF'd store (output)
APPWRITE_CRAFD_BUCKET_ID=...
APPWRITE_CRAFD_BUCKET_NAME=...
APPWRITE_CRAFD_COLLECTION_ID=...
APPWRITE_CRAFD_COLLECTION_NAME=...

# Metadata database (shared)
APPWRITE_METADATA_DATABASE_ID=...
APPWRITE_METADATA_DATABASE_NAME=...
```

*(Corrected 2026-08-03: four production-forecasts coordinate **values** were written out
above, two lines below the sentence saying they never are. The value-copy guard scanned
only `.py`; it now scans markdown too.)*

---

## Documentation
Expand Down
40 changes: 32 additions & 8 deletions docs/ADRs/014_claims_and_the_guards_that_carry_them.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@ infrastructure), [ADR-010](010_technical_risk_register.md) (the register)
## Context

Over two days in August 2026, this repository found **seven** places where it said one
thing and did another. None was a bug in the delivery. All were claims that had outlived
thing and did another. *(An eighth arrived the day after this ADR was accepted, and the
count is left at seven deliberately — see the note at the end of this section.)* None was a bug in the delivery. All were claims that had outlived
what they described:

- three register entries filed as open with their own stated closing conditions met
Expand All @@ -33,6 +34,15 @@ The pattern is not carelessness. Each was written accurately and became false wh
something else moved. What they share is that **nothing was attached to them that would
notice**.

**The eighth, and why the count above stays at seven.** On 2026-08-03 — the day after
this ADR was accepted — PR #211 added a second partner package and found that *eight*
guards were scoped to the first partner by name, so the new one landed exempt from all of
them, including the þing-01 `load_dotenv` prohibition. Register **C-78** records it. The
count above is deliberately not incremented: an ADR that renumbers itself every time the
pattern recurs becomes a changelog, and the argument does not depend on the number. What
the eighth case adds is §2's sharpest form — *a guard's declared scope is part of what
must be mutation-proven, not just its matching.*

ADR-003 already forbids inferring what should be declared. This ADR is that rule applied
one level up: **a declaration that nothing validates is an inference with better
grammar.**
Expand Down Expand Up @@ -105,11 +115,24 @@ If a pull request cites a register entry, an issue, a CIC or an ADR as the thing
addresses, it updates that artefact **in the same pull request**. Not in a follow-up, not
at closeout, not from memory.

**No test enforces this and none is proposed.** It was attempted: no expression reliably
separates *"this entry describes work that is done"* from prose, and a guard that guesses
is one that gets deleted (§3). This clause is therefore a habit, and it is written down
precisely because it is the one rule here that cannot be mechanised — which makes it the
one most likely to lapse.
**No test enforces this in general, though two partial guards exist and are green.**
`tests/test_register_integrity.py` carries
`test_no_open_entry_names_a_closing_artifact_that_already_exists` and
`test_no_open_entry_claims_its_mitigation_has_landed` — both added by S2 in this same
arc. They catch the two mechanisable shapes: an entry naming a file that now exists, and
one whose body says a mitigation landed while its header says Open.

What resists mechanisation is the general case: no expression reliably separates *"this
entry describes work that is done"* from prose, and a guard that guesses is one that gets
deleted (§3). So the clause is mostly a habit, and it is written down because it is the
rule here least amenable to a test — which makes it the one most likely to lapse.

*(Corrected 2026-08-03. This paragraph said "none is proposed" and the Alternatives
section called widening the guard "attempted and abandoned", while the widened guard was
already shipping in the same branch. And the guards' reach is genuinely partial: C-15 sat
Open claiming a mitigation had landed via a method that no longer exists, and escaped
`test_no_open_entry_claims_its_mitigation_has_landed` because it phrased the claim
without the em-dash the guard matches. Partial is worth saying; absent was wrong.)*

The evidence that it does lapse is this epic's own record. C-71 was fixed on the morning of
2026-08-02 and sat filed open for the rest of the day while eight further stories shipped.
Expand Down Expand Up @@ -150,5 +173,6 @@ Rejected as a replacement, not as a supplement: a checklist records what to do a
and the reasoning is what survives a contributor deciding the rule does not apply to them.
Worth adding later, pointing here.

**Widen `test_register_integrity` to catch §5.** Attempted and abandoned — see §5. Recorded
so the next person does not spend the same afternoon on it.
**Widen `test_register_integrity` to catch §5.** Partially done — see §5. Two shapes are
mechanised and green; the general case resists. Recorded so the next person neither
re-attempts the general case nor assumes there is nothing there.
Loading
Loading