Add partial run instructions and parameters for scalability

[jaakla]

Scope note (after review in this comment).
This issue is now scoped to workstream (a): the deterministic, CLI-testable half — sampled run execution, marking, non-promotion, and eval coverage.
The rest was split out: #22 (skill guidance and correctness traps), #23 (cost estimation and escalation ladder), #24 (federated divide-run-combine execution).
The original problem statement is preserved in full at the bottom.

Scope: sampled runs that can never be mistaken for canonical ones

1. CLI surface

--dry-run is already taken on openmapstack run and means "validate preflight and print the command without executing it" (openmapstack/cli.py:49, documented in README.md:173). Sampled execution needs its own flags:

openmapstack run project.yaml --sample                  # sample per the manifest's declared sampling parameter
openmapstack run project.yaml --sample-area <bbox>      # explicit test AOI
openmapstack run project.yaml --sample-rows <n>         # explicit row cap
openmapstack run project.yaml --sample-fraction <pct>   # explicit fraction

A single --sampling-size 1% is ambiguous — percent of area, of rows, or of raster resolution are three different knobs with three different failure modes — so they are separate flags. The run record stores the realized sample (actual rows, actual AOI, actual resolution), not just the requested one: TABLESAMPLE and friends only approximate.

2. Where the parameter is declared

Start with the existing runtime.implementation.parameters contract (openmapstack-parameters/v1), which already models "same pipeline, one knob turned" and already carries the rule that the canonical run passes nothing. No schema change, determinism preserved, and verify's project.parameters_match_steps honesty check comes along for free.

Open question to settle during implementation: today step/field bind a parameter to a processing step, while a sample AOI is a source selection knob — decide whether sources.*.selection.bbox can be a legal field target, or whether sampling parameters are exempt from the step/field pairing.

Promote sampling to a first-class sampling: manifest block only if evals show agents cannot use the parameter contract reliably.

3. The hard constraint: non-promotion

A sampled run legitimately produces a different inputs_hash — clipped or downsampled sources are different bytes — so it cannot share the canonical run's hash chain. It must therefore be structurally incapable of becoming the accepted result:

• the run record carries an explicit mode: sampled (default canonical), plus the realized sample descriptor and per-step timings (the latter is what Non-interactive cost estimation and the scale escalation ladder #23 consumes);
• a sampled run never occupies runs.latest — otherwise verify and the clean-rerun protocol inherit a fake baseline;
• a sampled run can never set project.status: validated, even when every check passes (references/project-spec.md:341);
• validation.expectations can never be satisfied from sampled output;
• validate / verify reject a sampled run offered as evidence, with a clear message rather than a hash mismatch;
• the report and dashboard label it, in the spirit of the existing status: draft_unvalidated override-bundle precedent (references/project-spec.md:614).

4. Eval coverage

908-validation-laundering already exists, and "sampled run presented as a validated result" is exactly that failure class. Per AGENTS.md both paths must be proven:

• a healthy fixture case: a sampled run executes, is marked, does not take runs.latest, and does not promote status;
• a negative case (e.g. 922-sampled-run-as-canonical) that must fail — a manifest presenting a sampled run as the validated answer.

Acceptance criteria

• Sampled runs are executable from the CLI without redefining --dry-run.
• The realized sample, not just the requested one, is recorded.
• No code path allows a sampled run to set runs.latest or project.status: validated.
• validate/verify reject sampled evidence with an intelligible error.
• Healthy fixture case and negative case both present and both proving what they claim.
• README/project-spec.md updated only where the public contract actually changed.

Out of scope (deliberately)

Choosing a sensible AOI automatically. That needs populated-place data the CLI does not have and should not grow, and it has no deterministic oracle to test against — it is judgment, and it lives in #22.

---

Original issue text

Problem

Full project runs with all the needed geographic data, especially with wide areas, high resolutions and processing complexity can take long time - not only minutes, but also hours and days. Also they consume extra compute resources (CPU, storage, RAM). It can easily happen that only in later phase some workflow step fails, and full processing needs to be redone.

Solution

Use "nail it before you scale it" concept - at least with automated flows run first test with limited samples, as a "dry run". It is not really dry, more like moist run. There are several sampling strategies depending on data and case: as default use smaller sub-area as test AOI (primary one - ensures that filtered in geometries do match together), also if this is not enough (or possible) then limit queries from big tables to certain number of first items (can use postgres tablesample SQL or similar), pre-process test data to reduce resolution of rasters and elevations, use pre-simplified geometries for complex polygons/polylines etc.

Add parameters to the run command:

• as general limiter (eg --sampled or --dry-run). Without extra hints it tries to estimate sampling agressiveness from the problem and if available metadata of sources what is reasonable sampling % and choose good sample within original AOI. The area cannot be totally random, e.g. for global map data it may be easily in blue ocean, instead of useful populated place; in some cases the good sample is city, but sometimes exactly outside city; in some cases it may really be inside ocean.
• optional parameters may give explicit limiting quantity, e.g. "--sampling-size 1%"
• for well-controlled cases allow setting test AOI explicitly: --sampling-area

Add to the skill instructions specific prose to use extensively these sampled pre-runs.

Acceptance criteria

• Bigger analyses get drier pre-run with reasonable sample, which is audited (but also clearly marked) as a real run. Only small analyses within <10 min processing time can skip it.
• Very big analyses (taking hours, days and more) try to estimate final runtime and ask user for confirmation if so big job gets accepted. Allow to choose also partial next level test options, e.g. before world-wide analysis offer to run it in city (probably the first test run) , county, small country, big country and/or continent levels.
• for very big cases offer also suggestions how to run it faster with more parallel processing - splitting job between task runners / nodes somehow, and offer implementation of such federated (divide-run-combine, map-reduce) approach.

[jaakla]

Reviewed against the current main (d4e1bd3). The core idea is right and fits the repo's grain, but a few things need to change before implementation, and one acceptance criterion is currently unimplementable as written.

1. --dry-run is already taken, with a different meaning

openmapstack run --dry-run exists today and means "validate preflight and print the command without executing it" (openmapstack/cli.py:49, documented in README.md:173). Reusing it for a sampled execution would silently redefine a published flag — and the issue already flags the discomfort ("not really dry, more like moist run").

Suggested naming, consistent with the existing --sampling-* proposal:

openmapstack run project.yaml --sample                     # auto-chosen sample
openmapstack run project.yaml --sample-area <bbox>         # explicit test AOI
openmapstack run project.yaml --sample-size 1%             # explicit fraction

--sampling-size 1% also needs a defined unit: percent of area, of rows, or of raster resolution? These are three different knobs with three different failure modes. I'd make them separate flags rather than one overloaded percentage, and require the run record to store the realized sample (actual rows, actual AOI, actual resolution), not just the requested one — TABLESAMPLE and friends only approximate.

2. The hard constraint: a sampled run must never be promotable to canonical

This is the part that most needs nailing down, because it cuts across every existing integrity invariant:

• a sampled run legitimately produces a different inputs_hash (clipped/downsampled sources are different bytes), so it cannot share the canonical run's hash chain;
• project.status: validated is defined as "only an all-passed report may set it" (references/project-spec.md:341) — a sampled run passing every check must still not be able to set it;
• validation.expectations answers must never be satisfiable from sampled output;
• runs.latest must not become a sampled run, or rerun/verify inherit a fake baseline.

Concretely, I'd propose: sampled runs write a run record with an explicit mode: sampled (plus the realized sample descriptor and the timings), never occupy runs.latest, and are rejected as evidence by validate/verify. The dashboard/report must label them, in the same spirit as the existing status: draft_unvalidated override-bundle precedent (project-spec.md:614).

Note the eval consequence: case 908-validation-laundering already exists, and "sampled run presented as a validated result" is exactly that failure class. Per AGENTS.md this needs both paths proven — a healthy fixture case (sampled run runs, is marked, is not promoted) and a negative case (e.g. 922-sampled-run-as-canonical) that must fail.

3. Where the parameter belongs — probably not a new schema block

runtime.implementation.parameters (openmapstack-parameters/v1) already exists for exactly the shape "same pipeline, one knob turned", with the rule that the canonical run passes nothing. A sample AOI fits that contract almost exactly.

I'd start by expressing sampling as a declared parameter the pipeline honours, rather than adding a first-class sampling: block — no schema change, determinism preserved, and verify's project.parameters_match_steps honesty rule comes along for free. Promote it to a first-class block only if evals show agents can't do it reliably. (The one mismatch: today step/field bind a parameter to a processing step, while sampling is a source selection knob — worth deciding whether selection.bbox can be a bindable field target.)

4. Auto-choosing "a good sample" can't be a CLI feature

it tries to estimate sampling aggressiveness … the area cannot be totally random, e.g. for global map data it may be easily in blue ocean

Agreed on the problem, but picking a sensible AOI requires knowing where populated places are — a data dependency the CLI does not have and shouldn't grow, and a heuristic with no deterministic oracle to test against.

I'd split the issue along that line:

• deterministic / CLI-testable: explicit --sample-area, --sample-size, run-record marking, status rules, refusal-to-promote. Fixture evals.
• judgment / skill prose: choosing a representative AOI, deciding when a sample is needed. SKILL.md + a live eval, not Python.

5. Runtime estimation must not block on stdin

try to estimate final runtime and ask user for confirmation

openmapstack run is invoked non-interactively via subprocess.run by both run and the clean-rerun protocol (openmapstack/rerun.py), and by CI. An interactive prompt would hang the harness. The confirmation belongs at the agent/user layer; the CLI's contribution should be a non-interactive, machine-readable projection — e.g. openmapstack run --estimate --json emitting a cost estimate and exiting.

Also worth being honest that a runtime estimate has no basis a priori. The only credible one is sampled wall-clock × scale factor, which is a further argument for the sampled run record carrying its scale factor and per-step timings.

6. GIS correctness traps the skill prose must state explicitly

These are the substance of the change, and I'd want them in SKILL.md / references/processing.md as named anti-patterns:

• Clipping to a sample AOI breaks every neighbourhood operation at its boundary. Distance-to-nearest, buffers, catchments and edge-crossing features are wrong by construction near the cut. A sample AOI needs a halo, and results inside the halo are not reportable. This interacts with the existing "bbox overlap as scan gate, then exact predicate" rule.
• Row-limit / TABLESAMPLE sampling destroys spatial coherence. A random 1% of parcels and a random 1% of roads essentially never intersect, so the join returns zero and the moist run passes for the wrong reason (or fails for one). The issue correctly makes sub-area sampling primary — the fallbacks should be explicitly labelled smoke-test only, never result-shaped.
• Downsampling rasters changes results non-linearly (area, slope, hydrology, viewshed). Same conclusion.
• Therefore the honest claim for a sampled run is: it proves the pipeline executes, the manifest graph resolves, the CRS handling holds and the schemas line up. It never proves the numbers. Counts and aggregates from a sampled run must not be surfaced as answers or written into validation.expectations.
• Pinning and the bounded-API completeness proof (numberMatched == returned) still apply within the sample — sampling is not a licence to skip pagination proof.

7. The "<10 min" acceptance criterion isn't checkable

Processing time isn't knowable before the run, so "only analyses under 10 min may skip the pre-run" can't be enforced or evaluated. A checkable version can be derived from what the manifest already records — access.file.row_count, selection.completeness.matched, and AOI extent: mandatory sampled pre-run above a declared threshold, recommended below it.

8. Split the map-reduce bullet into its own issue

The federated divide–run–combine work is a much larger scope than sampling and shouldn't gate it. When it does land, two existing rules constrain the design: the combine stage must live inside pipeline.py (the "one canonical implementation creates every declared output" rule forbids a wrapper that duplicates processing logic), and the combine must be order-independent — which the existing input_permutation_invariance metamorphic relation could actually prove rather than assert.

---

Suggested split: (a) sampled-run execution + marking + non-promotion + fixture/negative evals; (b) skill guidance on sampling strategy and its correctness traps + live eval; (c) cost estimation and escalation ladder (city → county → country → continent); (d) federated execution.