feat: detect adjacent script docstrings in hygiene

[Jammy2211]

Overview

Add a read-only Hygiene mode that finds consecutive module-level documentation strings in user-facing workspace and HowTo scripts. These boundaries can corrupt generated notebook cell structure and should be reported precisely, ranked in the normal Hygiene worklist, and delegated to the refactor workflow for mechanical cleanup.

The scanner must distinguish genuine adjacent documentation expressions from assigned strings, nested docstrings, comments, and blocks separated by executable code.

Plan

• Add a standard-library AST scanner for scripts/**/*.py in *_workspace and HowTo* repositories.
• Expose exact repository, file, and line findings through human and JSON Hygiene output.
• Include confirmed findings in the default ranked Hygiene worklist.
• Preserve the Hygiene Agent's read-only boundary and delegate repairs to /refactor.
• Add hermetic contract tests and validate against the live workspace catalogue.

Detailed implementation plan

Work Classification

Library workflow — PyAutoBrain organ source only.

Affected Repositories

• PyAutoBrain (primary)

Worktree Root

~/Code/PyAutoLabs-wt/hygiene-adjacent-docstrings/

Branch Survey

Repository	Current Branch	Dirty?
./PyAutoBrain	main	Yes — unrelated skills/WORKFLOW.md edit; use isolated worktree

Suggested branch: feature/hygiene-adjacent-docstrings

Implementation Steps

1. Add agents/conductors/hygiene/_hygiene_docstrings.py.

   • Discover repositories matching *_workspace and HowTo*.
   • Scan only scripts/**/*.py.
   • Parse source with the standard-library ast module.
   • Match consecutive module-body Expr(Constant(str)) nodes whose source gap contains whitespace only.
   • Report repository, relative file, and both blocks' start/end lines.
   • Suppress irrelevant invalid-escape SyntaxWarning noise and report parse failures separately.

2. Extend agents/conductors/hygiene/hygiene.sh.

   • Add a first-class docstrings mode.
   • Treat its results as confirmed, rankable findings rather than an advisory audit surface.
   • Include its count in the default Hygiene worklist.
   • Emit detailed findings in explicit human and JSON output.
   • Delegate remediation to /refactor; never edit scanned repositories.

3. Update the Hygiene contract documentation.

   • Document the new mode, scope, finding semantics, and read-only boundary in the conductor and skill documentation.

4. Extend tests/test_hygiene_conductor.py.

   • Cover adjacent double- and single-quoted blocks.
   • Cover chains containing multiple adjacent boundaries.
   • Exclude assigned strings, nested function/class docstrings, comments between blocks, and strings separated by code.
   • Verify default worklist inclusion and detailed human/JSON output.
   • Verify malformed scripts do not corrupt JSON output.

5. Validate:

   • Run the focused Hygiene conductor tests.
   • Run the full PyAutoBrain test suite.
   • Run hygiene docstrings and hygiene docstrings --json on the live tree.
   • Confirm the initial live result of 79 boundaries in 56 files across six repositories, with zero parse errors.
   • Leave source merging to the dependent workspace-cleanup phase.

Key Files

• agents/conductors/hygiene/_hygiene_docstrings.py — new AST scanner.
• agents/conductors/hygiene/hygiene.sh — mode registration, ranking, and output.
• agents/conductors/hygiene/AGENTS.md — authoritative agent contract.
• skills/hygiene/hygiene.md — user-facing workflow.
• tests/test_hygiene_conductor.py — CLI and scanner contracts.

Original Prompt

Click to expand starting prompt

Phase 2: Detect adjacent script docstrings through Hygiene

Type: feature
Target: PyAutoBrain
Difficulty: medium
Autonomy: supervised
Priority: normal
Status: formalised

Depends on: draft/bug/pyautobuild/back_to_back_docstrings_notebook.md

Requested scope

Extend @PyAutoBrain's Hygiene Agent with a script-docstring surface that scans every
user-facing *_workspace and HowTo* repository for consecutive top-level triple-quoted
documentation blocks separated only by whitespace. The Hygiene Agent must preserve its
reasoning-only boundary: report exact findings and delegate safe, mechanical merges through
the development workflow rather than editing repositories during its read-only pre-scan.

Expose the scan as a first-class Hygiene mode in human and JSON output, include repository,
file and line locations, include it in the default ranked worklist, and route findings to
/refactor. Restrict matches to consecutive top-level triple-quoted expression blocks;
ordinary string literals and documentation separated by executable code must not be flagged.

Applying the reported merges is the dependent phase
draft/maintenance/workspaces/merge_adjacent_docstrings.md.

Original user request (verbatim)

Double docstring formatting like this currently breaks ntoebook generate: GPU; not wrong, but slower than jnp.sqrt(...) if you're inside a hot
loop. For one-off analysis code, don't worry about it.

• The .array property of aa.Array2D etc. is the raw backing array — a
  numpy.ndarray on the NumPy path, a jax.Array on the JAX path.

The data_structures.py guide (scripts/guides/data_structures.py) covers
the wrapper-vs-raw-array distinction in detail.
"""

"""
Units
in the corresponding start_here.ipynb it looks like this:

%%

'''
Units

The units used throughout the strong lensing literature vary, therefore lets quickly describe the units used in
PyAutoLens.

The Tracer object and all mass profiles describe their quantities in terms of angles, which are defined in units
of arc-seconds. To convert these to physical units (e.g. kiloparsecs), we use the redshift of the lens and source
galaxies and an input cosmology. A run through of all normal unit conversions is given in guides in the workspace
outlined below.

The use of angles in arc-seconds has an important property, it means that for a two-plane strong lens system
(e.g. a lens galaxy at one redshift and source galaxy at another redshift) lensing calculations are independent of
the galaxies' redshifts and the input cosmology. This has a number of benefits, for example it makes it straight
forward to compare the lensing properties of different strong lens systems even when the redshifts of the galaxies
are unknown.

Multi-plane lensing is when there are more than two planes. The tracer fully supports this, if you input 3+ galaxies
with different redshifts into the tracer it will use their redshifts and its cosmology to perform multi-plane lensing
calculations that depend on them.

Extensibility
, can you make it so that the generate function that builds notebooks in autohands does not let these things break notebook generation and can you make it so the hygeine agent looks for and merges these in the scripts of all worskpaces, HowTos

[Jammy2211]

Hygiene implementation shipped to PR #163 (pending-release). A follow-up audit caught the original root start_here.py surface: commit fe09324 expands the scan from scripts/**/*.py to root *.py entry scripts as well. Final live inventory: 81 adjacent documentation boundaries in 58 files across 6/7 repos, zero parse errors. Validation: 23 focused tests passed; 163 applicable full-suite tests passed with the known unrelated sizing wrapper baseline deselected. Dependent workspace cleanup remains phase 3.

[Jammy2211]

Phase 3 workspace cleanup is shipped to six open pending-release PRs (nothing merged):

• docs: merge adjacent documentation blocks autocti_workspace#9
• docs: merge adjacent documentation blocks autofit_workspace#118
• docs: merge adjacent documentation blocks autogalaxy_workspace#163
• docs: merge adjacent documentation blocks autolens_workspace#342
• docs: merge adjacent documentation blocks HowToGalaxy#45
• docs: merge adjacent documentation blocks HowToLens#55

The cleanup merged 80 adjacent boundaries across 57 scripts; autolens_workspace#340 had already removed the 81st during a concurrent approved task. HowToFit was re-scanned and remains a zero-change target. Final Hygiene result: 0 findings, 0 parse errors across all seven repositories.

Post-generation validation: AutoFit 10/10, AutoGalaxy 8/8, AutoLens 11/11, HowToGalaxy 4/4, HowToLens 6/6, and the changed AutoCTI script passed directly under test mode. Exact ordered-prose transform witnesses and Python compilation passed for all changed scripts. Primary tracking issue: PyAutoLabs/autolens_workspace#341.