Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 45 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,14 @@ jobs:
examples/serosurvey_study.py examples/serosurvey_study.ipynb
examples/serosurvey_contexts.py examples/serosurvey_frames.py
examples/replicated_study.py
examples/illustration_artifacts.py examples/illustrate_replicated_study.py examples/frozen_slurm.py
examples/illustration_artifacts.py examples/illustrate_replicated_study.py examples/frozen_slurm.py examples/fisher_op_stack.py
- name: Lint
run: >-
uv run ruff check --preview src tests
examples/serosurvey_study.py examples/serosurvey_study.ipynb
examples/serosurvey_contexts.py examples/serosurvey_frames.py
examples/replicated_study.py
examples/illustration_artifacts.py examples/illustrate_replicated_study.py examples/frozen_slurm.py
examples/illustration_artifacts.py examples/illustrate_replicated_study.py examples/frozen_slurm.py examples/fisher_op_stack.py
- name: Type check (strict)
run: uv run mypy --strict src
- name: Build documentation (strict)
Expand Down Expand Up @@ -124,10 +124,48 @@ jobs:
just build-all
uv run --no-sync python -c 'from pathlib import Path; import trade_study; assert Path(trade_study.__file__).resolve().is_relative_to(Path("src").resolve())'

fisher-numpy-minimum:
name: Fisher screen (NumPy 1.24 CPU)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v4
with:
python-version: '3.11'
- name: Install core at its minimum NumPy version
run: |
uv venv
uv pip install -e . 'numpy==1.24.0' pytest
- name: Qualify the bounded core and complete guide
run: >-
uv run --no-sync python -m pytest tests/test_fisher.py
tests/test_documentation.py::test_complete_fisher_guide_preserves_analytic_designs_and_denominators

fisher-op-stack:
name: Fisher information artifact (NumPy CPU)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v4
with:
python-version: '3.12'
- name: Install reviewed public producers and this consumer
run: |
uv venv
uv pip install -e . \
'op-system @ git+https://github.com/ACCIDDA/op_system@bbde00697428f16f6cf948e7aff207f7b65231c8' \
'op-engine @ git+https://github.com/ACCIDDA/op_engine@5f7dd592e97a849bca5ccad348ea66e935187f18' \
'op-infer @ git+https://github.com/ACCIDDA/op_infer@713a2b76119e174dab75290eaf5bff9ffce63af6'
- name: Qualify actual measured information and declared directions
env:
OPENBLAS_NUM_THREADS: '1'
OMP_NUM_THREADS: '1'
run: uv run --no-sync python examples/fisher_op_stack.py

ci:
name: ci
if: always()
needs: [lint, test, examples, wheel]
needs: [lint, test, examples, wheel, fisher-numpy-minimum, fisher-op-stack]
runs-on: ubuntu-latest
steps:
- name: Require every validation job to succeed
Expand All @@ -136,8 +174,12 @@ jobs:
TEST_RESULT: ${{ needs.test.result }}
EXAMPLES_RESULT: ${{ needs.examples.result }}
WHEEL_RESULT: ${{ needs.wheel.result }}
FISHER_RESULT: ${{ needs.fisher-op-stack.result }}
FISHER_MINIMUM_RESULT: ${{ needs.fisher-numpy-minimum.result }}
run: |
test "$LINT_RESULT" = success
test "$TEST_RESULT" = success
test "$EXAMPLES_RESULT" = success
test "$WHEEL_RESULT" = success
test "$FISHER_RESULT" = success
test "$FISHER_MINIMUM_RESULT" = success
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,18 @@ All notable changes to this project will be documented in this file.

## [Unreleased]

### Added

- Bounded local Fisher-screen adapter for caller-qualified host information
matrices, explicit coordinates/target directions, existing feasibility
constraints, distinct unavailable/failed outcomes and planned-ID counts.
Export primitive JSON context/policy/evidence without new dependencies.

### Documentation

- Separate computational SBC, effect recovery, predictive adequacy and simulation
support. Document PPC reference-law, conditional-denominator and sampling limits.

- Record execution ownership, host-result boundaries, Fisher-screen integration
requirements and separate qualification targets for penalized stacking and EVSI.

Expand Down
13 changes: 13 additions & 0 deletions docs/api/fisher.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Local information screening

See the [complete example and qualification scope](../guide/fisher.md).

::: trade_study.FisherContext

::: trade_study.FisherScreenPolicy

::: trade_study.FisherScreenResult

::: trade_study.screen_fisher

::: trade_study.fisher_screen_counts
1 change: 1 addition & 0 deletions docs/guide/capabilities.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ scoring, optimization and plotting; statistical estimators belong to the caller.
| Submit frozen Python tasks | `slurm.freeze_checkout`, `prepare_slurm_plan`, `SlurmTask` | [Experimental Slurm launch layer](slurm.md); Git/POSIX/shared environment, checked source/imports and task receipts. Runner integration remains explicit. |
| Compare paired designs | `paired_difference`, `paired_rank` | Core bootstrap; t intervals need SciPy. Nonfinite pairs are excluded and `n_pairs` is reported. |
| Allocate replication | `run_sequential`, `ReplicationPolicy` | Fixed or adaptive policy with recorded bounds, counts and assumptions; power is not automatically certified. |
| Screen local information | `screen_fisher`, `FisherScreenPolicy`, `fisher_screen_counts` | Caller-qualified bounded host matrices, declared directions, existing constraints and retained unavailable outcomes; [scope and example](fisher.md). |
| Plan fixed proportion precision | `proportion_interval`, `replicates_for_half_width` | [Proportions guide](proportion-intervals.md); Wilson precision is not a power calculation or sequential interval. |
| Select nondominated alternatives | `extract_front`, `pareto_rank`, `hypervolume`, `igd_plus` | `pareto`; positive objective scaling preserves front membership. |
| Report decisions across preferences | `preference_sweep`, `PreferencePolicy` | Regret, practical equivalence and optional paired uncertainty; preferences are specified by the caller. |
Expand Down
11 changes: 9 additions & 2 deletions docs/guide/execution-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,9 @@ failures and stable task identities regardless of a simulator's backend.

## Fisher feasibility screening

A future Fisher screen can supply a scalar result to existing constraint and
replication machinery. Consume an explicit information artifact or caller-owned
The [bounded Fisher adapter](fisher.md) supplies a scalar result to existing
constraint machinery and preserves unavailable outcomes. It consumes an explicit
information artifact or caller-owned
screen instead of adding a compiler/inference dependency to the core. The
artifact records parameter coordinates/scales, observation noise, design,
nuisance treatment, reference parameter regime and numerical tolerances.
Expand All @@ -35,6 +36,12 @@ Local numerical rank is a screen, not a structural-identifiability proof. Estima
screen uncertainty needs an actual sampling/replication contract before a
probabilistic feasibility claim.

The implemented fixed-replicate allocation and finite-horizon bounded-score
methods remain available. Recoverable failed-attempt artifacts and gate-specific
power/false-failure qualification retain their separate scope. The Fisher guide
also distinguishes SBC, effect recovery, predictive checks and simulation-support
evidence; each needs its own declared reference and denominator.

## Predictive stacking and dimension penalties

The existing Bayesian path delegates to ArviZ predictive stacking. Preserve that
Expand Down
147 changes: 147 additions & 0 deletions docs/guide/fisher.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# Local information feasibility screens

`screen_fisher` consumes a caller-qualified information matrix and exposes a
local minimum-eigenvalue screen through the existing `Constraint` API. Core
dependencies remain NumPy-only. The inference layer or project supplies measured
output sensitivities, the observation/noise model, coordinate transformations and
any nuisance profiling. For example, OP Infer's bounded Gaussian information
adapter can supply its explicitly transferred host matrix; its qualification
scope and evidence references travel with the screen.

The optional `examples/fisher_op_stack.py` exercises the actual compiler,
forward sensitivity solver and Gaussian information adapter against an
independent exponential-decay derivative, including an initial-state parameter
and direct measurement parameter. A separate required NumPy CPU CI job installs
immutable reviewed public producer commits and runs that example. Those packages
remain optional for users of the core screen; no device qualification is implied.

For independent unit-variance measurements of `a + b*x`, a balanced design at
`x=-1,+1` has information `diag(n,n)`. The following complete example compares
two designs against a prespecified threshold. The Hessian of the Gaussian
negative log likelihood gives this reference independently of the screen.

```python
import numpy as np
from trade_study import (
Direction, FisherContext, FisherScreenPolicy, Observable, ResultsTable,
fisher_screen_counts, screen_fisher,
)

def context(n):
return FisherContext(
parameter_names=("a", "b"),
parameter_coordinates="physical a,b; dimensionless",
parameter_scales=(1.0, 1.0),
observation_model="independent Gaussian, known variance=1",
design=f"{n} balanced measurements at x=-1,+1",
reference_regime="linear mean; all a,b",
nuisance_treatment="none; both coefficients targeted",
sensitivity_reference="analytic d(a+b*x)/d(a,b)=(1,x)",
numerical_reference="exact design; host float64 algebra",
qualification_reference="analytic Gaussian likelihood Hessian",
information_reference=f"retained synthetic design n={n}",
)

policy = FisherScreenPolicy(threshold=4.0)
records = {
str(n): screen_fisher(np.eye(2) * n, context=context(n), policy=policy)
for n in (2, 8)
}
table = ResultsTable(
configs=[{"n": n} for n in (2, 8)],
scores=np.array([[records[str(n)].as_scores()["fisher_min_eigenvalue"]]
for n in (2, 8)]),
observable_names=["fisher_min_eigenvalue"],
metadata=[{"fisher_screen": records[str(n)].to_metadata()} for n in (2, 8)],
)
observable = Observable("fisher_min_eigenvalue", Direction.MAXIMIZE)
assert table.feasible([policy.constraint()]).tolist() == [False, True]

# Each unavailable attempt keeps its identity and reason.
records["missing-artifact"] = screen_fisher(None, context=context(8), policy=policy)
records["failed-solve"] = screen_fisher(
None, context=context(8), policy=policy, failure="upstream solve failed"
)
counts = fisher_screen_counts(
records, planned_ids=["2", "8", "missing-artifact", "failed-solve", "pending"]
)
assert counts == dict(planned=5, feasible=1, infeasible=1,
unreadable=1, failed=1, missing=1)
```

`to_metadata()` returns detached primitive JSON for existing result bundles.
Retain the per-attempt records and complete planned IDs alongside any scored
table. The two numeric rows above are not the denominator for all five planned
attempts. `as_scores()` raises for unreadable/failed outcomes, so the generic
runner follows its existing failure/checkpoint policy rather than silently
averaging them away. These outcome counts alone supply no probability or gate.

## Coordinates, targets and numerical limits

The input matrix is already expressed in its declared coordinates and scales.
This adapter never rescales it automatically. Thresholds have meaning in those
coordinates; a change of units requires a correspondingly qualified policy.
Context references declare the observation law/noise, design, parameter regime,
nuisance convention and sensitivity/numerical/qualification evidence. The caller
retains and verifies those artifacts. The recorded matrix digest identifies the
actual float64 host snapshot, independently of the caller's reference labels.

An optional p-by-k `directions` basis has orthonormal columns, checked with
absolute and relative tolerances of `1e-10`. The tested matrix is `B.T @ F @ B`,
with the complementary directions fixed. This is useful for explicitly declared
identifiable perturbations of a singular nominal matrix. For observations of
`a+b`, the normalized `(1,1)` direction has information while `(1,-1)` does not.
The caller must justify which targets and gauge directions this represents.

Directional restriction does not perform nuisance elimination. For example,
`F=[[1,1],[1,2]]` has conditional information 1 for its first coordinate, while
profiling the second coordinate gives `1 - 1*(1/2)*1 = 0.5`. Supply the caller's
profiled matrix and its distinct context when that is the intended calculation.

The default dense dimension limit is 128 parameters. Inputs are real NumPy host
arrays; a device-to-host transfer is an explicit caller boundary. Matrix and
eigensolver storage remain quadratic in dimension; the limit does not bound
whole-process or AD/compiler memory. No sensitivity array is retained here.

Invalid shape/dtype, nonfinite, asymmetric, indefinite or unsupported matrices
produce unreadable records. Upstream/eigensolver failures produce failed records.
Symmetry and positive-semidefiniteness use
`atol + rtol * max(abs(F))`. Accepted near-symmetric matrices are averaged with
their transpose. Actual tested eigenvalues are retained; tiny negative values
are not clipped and comparison with the threshold is inclusive and exact on the
computed value. Numerical tolerances do not relax the decision threshold.

## Qualification beyond a screen

A local information screen helps shortlist designs. A design decision also needs
declared simulation/refitting targets, numerical checks and relevant recovery
and predictive evidence. In particular:

- SBC tests the selected posterior computations under a declared generator and
design. Finite non-rejection supports “no miscalibration detected”. Ordinary
parameter ranks can miss data-ignoring estimators; appropriately chosen
data-dependent quantities improve sensitivity ([Modrák et al.](https://arxiv.org/abs/2211.02383)).
- Effect recovery uses the actual reporting rule and design. Define signed
effects in a null-centred coordinate. Type S and Type M condition on reporting;
zero decisions leave them unavailable. At zero effect, report false-support
rather than an undefined magnitude ratio. Preserve failures and conditional
denominators ([Gelman and Carlin](https://sites.stat.columbia.edu/gelman/research/published/retropower_final.pdf)).
- Predictive checks examine the relevant observed features and held-out units.
Ordinary in-sample posterior predictive probabilities need not be uniform.
Randomized ties address atoms; a rejection gate still needs a qualified
reference distribution and dependence/multiplicity treatment
([Gelman](https://sites.stat.columbia.edu/gelman/research/published/ppc_understand3.pdf)).
- Amortized estimators also need checks of the observed data against their
training-simulation distribution. A simulator-generated SBC bank does not
directly test that observed-data discrepancy.

Repeated information matrices estimate a probability only under a declared
sampling law. A confidence argument on a deterministic matrix creates no such
law. Use [fixed-sample proportions](proportion-intervals.md) for their supported
Bernoulli assumptions, or the [sequential runner](../api/runner.md) for its
bounded-score allocation contract. Keep planned attempts, failed and missing
results, independent/cluster units and the stopping rule explicit. Interval
precision, gate power and false-failure rates each require their own evidence.

General Fisher families, nuisance profiling, global structural-identifiability
analysis, posterior adequacy and EVSI retain their caller/inference-layer scope.
Loading
Loading