Skip to content

Release-fidelity validation: 14 workspace script legs fail under mode=release (run 28784914443) #27

Description

@Jammy2211

Summary

The first real end-to-end release-validation run surfaced 14 failing run_scripts legs (of ~28) in workspace-validation.yml mode=release. This is a tracking issue for the workspace/library maintainers.

  • Run: https://github.com/PyAutoLabs/PyAutoHeart/actions/runs/28784914443 (mode=release, 2026-07-06)
  • Rehearsed wheels: autoconf / autoarray / autofit / autogalaxy / autolens == 2026.7.6.1.dev65001 (TestPyPI), built from the current library main HEADs.
  • Not a packaging problem: all 5 wheels installed cleanly from TestPyPI, and verify_install A–E passed. The failures are genuine per-script failures under release fidelity (real sampler / full evaluation, no source on PYTHONPATH).
  • Gate impact: pyauto-heart validate --ingest recorded release_ready=false → readiness is RED (release validation FAILED (stage integrate)). This is the expected, correct behaviour — the run caught real regressions.

Failing legs (project → directory)

Workspace Failing script directories
autofit_workspace cookbooks
autofit_workspace_test database
autogalaxy_workspace multi, guides, imaging, interferometer, ellipse
autogalaxy_workspace_test ellipse, interferometer
autolens_workspace multi, cluster, imaging, interferometer, guides

All other legs (autofit overview/model/plot/searches/simulators/jax_assertions/model_composition, autolens weak/point_source, autogalaxy_test aggregator, …) passed, as did verify_install_release.

Example root cause (autolens_workspace / imaging leg)

The imaging leg ran 102 scripts; all passed except two, which failed the whole leg:

  • scripts/imaging/features/advanced/double_einstein_ring/chaining.pyTIMEOUT (300 s) — a heavy multi-search chaining pipeline exceeding the per-script cap under the release profile.
  • scripts/imaging/features/advanced/shapelets/modeling.pyFAIL (~77 s) — a JAX runtime exception ("JAX has removed its internal frames from the traceback … Set JAX_TRACEBACK_FILTERING=off").

The pattern (a small number of long-running / JAX-path scripts per leg) likely explains most of the other 13 legs, but only the autolens/imaging leg was drilled into for specific script names — the per-script failures for the remaining legs are in the run's per-job logs (results-scripts-<project>-<directory> artifacts + each run_scripts job log).

Suggested next steps

  1. For each failing leg, open its run_scripts job log in run 28784914443 and identify the specific failing/timed-out scripts.
  2. Triage: genuine regression vs. release-profile timeout (the 300 s cap) vs. JAX-path issue on the packaged wheels.
  3. Fix in the owning workspace/library repo; consider splitting per-workspace sub-issues if ownership differs.
  4. Re-run the release-validation once fixed (Brain Release Agent: pyauto-brain release validate) to confirm readiness returns to GREEN.

Provenance

Surfaced by the PyAutoBrain health conductor + release conductor driving the full Stages 0–3 release-validation pipeline end-to-end (rehearsal → integrate → ingest → re-judge). No workspace/library code was modified — this issue only reports the regressions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions