Repository navigation
Define statistically valid interval and stopping policies for adaptive replication #176
Description
Implemented baseline and remaining scope (2026-10-22)
PR #199 delivered the fixed-replicate policy and retained the valid finite-horizon bounded-score method, with detached policies, planned/actual counts, multiplicity and stopping metadata. These capabilities are available now; this issue does not track building them again.
Remaining work: recoverable failed-attempt artifacts, consumer-specific gate power/false-failure qualification and separately justified tighter/unbounded-horizon methods where needed. A bounded caller-owned Fisher screen is being delivered under #216; its outcome counts preserve unavailable/missing attempts without supplying a statistical probability or new stopping guarantee.
Problem
Sequential replication can stop after observing scores or coverage outcomes. Wilson/Clopper–Pearson intervals at a fixed sample size do not automatically retain their coverage interpretation under arbitrary repeated checking and result-dependent stopping.
Scope
Define an uncertainty/stopping contract for run_sequential and replication-based feasibility/selection. Coordinate with #175's fixed-sample proportion helpers.
Acceptance criteria
- Distinguish interval precision, gate power, false-failure rates and stopping policy in metadata.
- Preserve a fixed-replicate policy with prespecified acceptance rules as a supported baseline.
- If sequential inferential claims are made, use a justified confidence-sequence/alpha-spending or other sequentially valid method and state its assumptions.
- Handle paired/common-random-number, clustered and failed replicates explicitly; independent Bernoulli assumptions cannot be silently reused.
- Record planned/max/actual sample counts, stopping reason, multiplicity and unavailable/underpowered outcomes.
- Demonstrate known calibrated synthetic cases and optional statistical qualification over repeated experiments; keep CI bounded and deterministic.
- Do not let missing results become passes or shrink an undeclared denominator.
- Keep NumPy-only core, optional lazy integrations, typed API/docs and Unreleased changes; run required just ci checks.
Reference and scope
https://arxiv.org/abs/1810.08240
The statistical inference layer defines calibration hypotheses; trade-study owns replicate scheduling and its uncertainty/stopping mechanics. Deliver policy/schema and qualified methods separately.
Roadmap placement
Phase 2: Reproducible calibration, artifacts and scoped assurance reporting.
Activity
feat: report how often serosurvey designs meet a precision target (#189…
jc-macdonald commentedon Oct 15, 2026
First bounded #176 increment after the trade-study handoff:
Research: Howard et al. and the authors' confseq implementations provide tighter or unbounded-horizon mixture/stitching/betting alternatives. They require separate tuning/domain/numerical qualification and are not needed to replace this repository's valid bounded-horizon baseline. Do not add a solver or a compiled dependency merely to express the current policy contract.
Keep #176 open after this contract increment for richer recoverable-failure artifacts, gate-specific power/false-failure qualification and optional repeated-experiment comparisons/tighter sequential methods. #173's independent benchmark evidence remains separately open; no release/archive action is included.
feat: record replication policies and support fixed-count plans (#199)
jc-macdonald commentedon Oct 15, 2026
The first bounded contract increment is delivered in PR #199, merged after all eight current-head checks passed. The authorized maintainer override satisfied the review-protection gate only after CI was green. Remote/local feature branches are deleted; clean local main is synchronized.
The existing valid adaptive Hoeffding/finite-horizon union-bound method is retained. An explicit fixed plan collects its complete prespecified count before one final inference look. Summary metadata now records the used detached policy, exact/min/max/actual counts, objectives/constraints, inference family/multiplicity, stopping and observed precision. Gate power and false-failure rates remain unassessed. Independent cluster-score denominators and cross-design CRN scope are explicit; a failed evaluation aborts rather than becoming a passing reduced-denominator result.
Validation: required local
just ci(823 tests, 97.36% coverage), strict docs/types/lint, all 41 focused cases against a clean installed wheel, built wheel/sdist and strict Twine checks. The GitHub matrix includes Python 3.10–3.13, examples and clean-wheel checks. The committed tests and mathematical scope do not replace the separate JOSS benchmark or qualify gate power.Keep #176 open for recoverable failed-attempt artifacts and broader gate-specific/statistical qualification, including any future tighter/unbounded-horizon methods. #173 remains separately required for JOSS evidence. No new release/archive publication was performed.
jc-macdonald commentedon Oct 22, 2026
The bounded local information screen is delivered in trade-study PR #219, closing #216 after all 10 hosted checks passed on final head
a28d9cf8d39d6333e3b8336911eb29e784f09c99.FisherContext,FisherScreenPolicy,screen_fisherand planned-ID counts consume caller-qualified host matrices and existing feasibility constraints. Records preserve coordinates/scales, noise/design/regime, nuisance and numerical/qualification references, actual spectra and matrix digests. Declared directions support singular nominal examples while keeping directional restriction and nuisance profiling explicit. Unreadable, failed and missing attempts retain their separate counts and cannot silently enter score averages.Qualification: 943 clean installed-wheel tests; 41 focused cases on current NumPy and the 1.24 core floor; actual pinned compiler → forward sensitivities → op_infer Gaussian information → trade-study screen with an independent measured-derivative reference; strict typing/lint/docs and distribution checks. Hosted Python 3.10–3.13, clean-wheel, minimum-NumPy and native-artifact jobs all passed. The guide separates computational SBC, effect recovery, predictive checks, simulation support and sampling/stopping assumptions.
The fixed-replicate and bounded-horizon baseline from #199 is reused. General sequential failure recovery and consumer-specific gate operating characteristics remain in #176; broader information/nuisance interfaces remain in ACCIDDA/op_infer#21 and ACCIDDA/op_infer#22. No estimator epoch or whole-process performance claim follows from this integration.
Next numerical work: checked dependency/action structure in ACCIDDA/op_system#296, reusing existing dense symbolic metadata/actions and existing block proofs; consumer-led operator/Krylov/scheduling increments follow. Local stability ACCIDDA/op_engine#256 remains separately open. All further GPU execution and op_infer package publication remain held.
jc-macdonald commentedon Oct 23, 2026
Next bounded engineering increment: preserve an inspectable abort artifact without changing the established stopping methods.
capture_failure=Trueto sequential execution. Ordinary evaluator/score-validation exceptions then raise a typed error carrying a detached failure record and the original exception as its cause. Default exception behavior remains unchanged.Keep #176 open for consumer-specific gate operating characteristics and any separately justified new statistical methods. This increment changes no frozen consumer run or scientific policy.