fix(convolver): from_gaussian kernel grid must not honour PYAUTO_SMALL_DATASETS cap

[Jammy2211]

Overview

Convolver.from_gaussian is broken under PYAUTO_SMALL_DATASETS=1 for any kernel
larger than 16x16. It builds its Gaussian on a Grid2D.uniform that the smoke-mode
cap silently shrinks to (16, 16), then wraps those 256 values in an Array2D using
the caller's uncapped shape_native — a self-inconsistent construction that raises
ArrayException. This is a real library bug, not the env-config gap the originating
prompt assumed. It currently kills scripts/imaging/data_preparation/manual/mask_irregular.py
in both autolens_workspace and autogalaxy_workspace.

Plan

• Opt the Gaussian kernel grid out of the small-datasets cap — a convolution kernel's
  shape is intrinsic to the operator, not a dataset size, so the cap must not apply.
• Use the existing Grid2D.uniform(respect_small_datasets=False) opt-out rather than
  adding a new lever.
• Add a numpy-only regression test asserting a 31x31 kernel survives PYAUTO_SMALL_DATASETS=1
  and blurs a 16x16 image correctly.
• No workspace edits: both failing scripts go green off the library fix alone.

Detailed implementation plan

Affected Repositories

• PyAutoArray (primary, and only)

Dropped from the prompt header after investigation: autolens_workspace and
autogalaxy_workspace need no change once the library is fixed; HowToGalaxy and
HowToLens do not carry the script.

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoArray	main	clean
./autolens_workspace	main	dirty (27, unrelated)
./autogalaxy_workspace	main	dirty (9, unrelated)

No worktree claims this task.

Suggested branch: feature/convolver-gaussian-small-datasets-cap

Root cause

autoarray/operators/convolver.py:721:

grid = Grid2D.uniform(shape_native=shape_native, pixel_scales=pixel_scales)

Grid2D.uniform shrinks grids >16x16 to (16, 16) at pixel_scales=0.6 when
PYAUTO_SMALL_DATASETS=1 (autoarray/structures/grids/uniform_2d.py:499). Line 746
then calls Array2D.no_mask(values=gaussian, shape_native=shape_native) with the
original (31, 31), so 256 values meet a 961-pixel mask:

ArrayException: array_2d_slim.shape = 256  vs  mask_2d.pixels_in_mask = 961, shape_native (31, 31)

Reproduced from autolens_workspace root:

PYAUTO_SMALL_DATASETS=1 PYAUTO_TEST_MODE=2 PYAUTO_SKIP_VISUALIZATION=1 \
PYAUTO_DISABLE_JAX=1 PYAUTO_FAST_PLOTS=1 \
python scripts/imaging/data_preparation/manual/mask_irregular.py

Fix verified by monkeypatching Grid2D.uniform in a scratch reproduction:
data (16,16) / kernel (31,31) / blurred (16,16) / mask (16,16), 111 pixels — clean.

Implementation Steps

1. autoarray/operators/convolver.py (~L721) — in Convolver.from_gaussian:

   grid = Grid2D.uniform(
       shape_native=shape_native,
       pixel_scales=pixel_scales,
       respect_small_datasets=False,  # kernel shape is intrinsic to the operator, not dataset size
   )
   

   Note this in the shape_native docstring entry: the kernel is built at the full
   requested size even under PYAUTO_SMALL_DATASETS=1.

2. test_autoarray/operators/test_convolver.py — numpy-only regression test using
   monkeypatch.setenv("PYAUTO_SMALL_DATASETS", "1"):

   • Convolver.from_gaussian(shape_native=(31, 31), pixel_scales=0.1, sigma=0.1)
     constructs without raising, kernel.shape_native == (31, 31);
   • it blurs a 16x16 Array2D to a 16x16 result (the mask_irregular geometry).
     Mirror the style of the neighbouring from_gaussian tests.

Verification

1. New test fails on main, passes after the fix.
2. Full PyAutoArray unit suite (shared Grid2D.uniform path).
3. mask_irregular.py runs clean under the smoke env in both workspaces.
4. Same scripts with PYAUTO_SMALL_DATASETS unset still pass (full-res path untouched).
5. Spot-check an 11x11 consumer (autolens_workspace/scripts/imaging/simulator.py) —
   below the cap, no behaviour change.

Blast radius

Any Convolver.from_gaussian(shape_native > 16x16) is currently broken in smoke mode.
A sweep of both workspaces + HowToLens/HowToGalaxy found four such call sites: the two
mask_irregular.py scripts (live, failing) and two (21, 21) sites
(autolens_workspace/scripts/imaging/simulator.py:435, guides/advanced/over_sampling.py:494)
that are latent — both in non-executed fenced prose blocks, and guides/ already unsets
the flag.

Rejected alternatives

• unset: [PYAUTO_SMALL_DATASETS] in each workspace config/build/env_vars.yaml (the
  prompt's suggestion) — papers over the library bug and leaves every >16x16 kernel broken.
• Capping the mask in the script — the mask is data-derived and already consistent once
  the kernel is right.
• Promoting these scripts to smoke_tests.txt — the smoke list stays a curated subset.

Key Files

• autoarray/operators/convolver.py — Convolver.from_gaussian, the one-line fix
• autoarray/structures/grids/uniform_2d.py:471-517 — Grid2D.uniform and the
  respect_small_datasets opt-out being reused
• autoarray/util/dataset_util.py — the SMALL_DATASETS cap constants for context
• test_autoarray/operators/test_convolver.py — new regression test

Original Prompt

Click to expand starting prompt

mask_irregular fails under SMALL_DATASETS: slim array 256 (16^2) vs mask 961 (31^2)

Type: bug
Target: autoarray
Repos:

• autogalaxy_workspace
• autolens_workspace
• HowToGalaxy
• HowToLens
• PyAutoArray
  Difficulty: small
  Autonomy: supervised
  Priority: normal
  Status: formalised

Un-parked by a parallel chat (old "silent failure" resolved) — now the REAL error is a
SMALL_DATASETS cap mismatch, i.e. an env-config gap, not a deep code bug:

imaging/data_preparation/manual/mask_irregular.py
  autoarray/structures/arrays/array_2d_util.py:69 check_array_2d_and_mask_2d
  ArrayException: slim array_2d_slim.shape = 256  vs  mask_2d.pixels_in_mask = 961, shape_native (31,31)

The data array is capped to 16x16 (256) under PYAUTO_SMALL_DATASETS=1 but the manually-drawn
irregular mask is 31x31 (961) — sizes disagree. Fix options (mirror the 2026-07-21 cap/should_simulate
work): add a per-script unset: [PYAUTO_SMALL_DATASETS] override in each repo's config/build/env_vars.yaml
(this script loads pre-committed/real data at full res), OR cap the mask via the same 16x16 lever.
Affected: autogalaxy_workspace + autolens_workspace (+ HowToGalaxy/HowToLens if they carry the script).
Currently un-parked and FAILING — remove/refresh any NEEDS_FIX marker once green.

[Jammy2211]

Smoke tests — all six workspaces

Run against feature/convolver-gaussian-small-datasets-cap (PR #398).

Workspace	Pass	Fail	Skip	Total
autofit_workspace	7	0	1	8
autogalaxy_workspace	6	0	0	6
autolens_workspace	9	0	0	9
autolens_workspace_test	16	1	3	20
euclid_strong_lens_modeling_pipeline	6	0	0	6
HowToLens	6	0	0	6
Total	50	1	4	55

No regressions from this change.

The one failure — pre-existing and unrelated

autolens_workspace_test/scripts/jax_likelihood_functions/point_source/point.py —
AssertionError: point: JAX vmap likelihood mismatch.

Confirmed unrelated by direct A/B: reverting autoarray/operators/convolver.py to its
pre-fix version and re-running produces the identical failure and value
(ACTUAL 16.131221 vs DESIRED -83.380498) in both cases. The script also contains no
Convolver / from_gaussian / PSF usage — point-source likelihoods do not convolve — so
this change cannot reach it.

It is also non-deterministic: an earlier run in the parallel batch produced
ACTUAL -1e+99 (the failed-fit sentinel) rather than 16.131221. Worth a separate look;
note PointSolver has its own PYAUTO_SMALL_DATASETS short-circuit
(autolens/point/solver/point_solver.py:111) that skips the triangle-tiling solve, which
may make this JAX-vs-numpy parity assert unstable under the smoke env. Filed as an
observation only — not touched here.

The euclid workspace initially showed 6 failures, which were an artifact of my ad-hoc
runner not applying args_default from config/build/env_vars.yaml
(--dataset / --sample are required). Re-run with the correct arguments: 6/6 pass.

[Jammy2211]

Shipped

PR #398 merged (402ffba7).

What it was: not the PYAUTO_SMALL_DATASETS env-config gap this issue's original
prompt assumed — the traceback never reached the mask-building code. Convolver.from_gaussian
evaluated its Gaussian on a Grid2D.uniform that the fast-mode cap shrank to 16x16, then
wrapped those 256 values in an Array2D at the caller's uncapped shape_native (961).
Any Gaussian kernel above 16x16 was broken in smoke mode.

Fix: Grid2D.uniform(respect_small_datasets=False) at the kernel call site — a kernel's
shape is intrinsic to the convolution operator, not a dataset size.

Verification:

• Regression test test__from_gaussian__small_datasets__kernel_keeps_full_requested_shape
  verified RED on main with the exact production ArrayException, green with the fix
• PyAutoArray suite: 927 passed
• Smoke, all six workspaces: 50 pass / 4 skip / 1 pre-existing unrelated failure
• mask_irregular.py green in both workspaces, with the flag set and unset

No workspace changes were required. The originating prompt listed 5 repos; 4 dropped out —
autolens_workspace and autogalaxy_workspace go green off the library fix alone, and
HowToGalaxy / HowToLens do not carry the script. The suggested
unset: [PYAUTO_SMALL_DATASETS] override was deliberately not added: it would have masked
a live library bug and left every >16x16 kernel broken.