Skip to content

fix(v1_compat): stop silently neutralizing check_function — wrap it like process_function, and log swallowed check exceptions - #80

Open
rhoadesScholar wants to merge 2 commits into
v2.0from
jeffr/v1-compat-check-function
Open

fix(v1_compat): stop silently neutralizing check_function — wrap it like process_function, and log swallowed check exceptions#80
rhoadesScholar wants to merge 2 commits into
v2.0from
jeffr/v1-compat-check-function

Conversation

@rhoadesScholar

Copy link
Copy Markdown
Contributor

What breaks today

A daisy-1.x-style check_function — one that does funlib arithmetic on block.write_roi.offset, the documented idiom downstream consumers use for resume checks — is silently neutralized under v2's compat layer, so resumed runs recompute every done block:

  1. Task.__new__ wraps process_function in the native→funlib block converter but not check_function (daisy-py/python/daisy/v1_compat.py:225-228), so the check receives a raw native block.
  2. daisy._daisy.Coordinate has no arithmetic dunders (daisy-py/src/py_roi.rs:46-105), so the v1-style check raises TypeError: unsupported operand type(s) for -: 'daisy._daisy.Coordinate' and 'Coordinate'.
  3. PyCheckBlock::check swallows every Python exception into false with no log line (daisy-py/src/py_callbacks.rs:71-79, .unwrap_or(false)), so the scheduler's precheck (daisy-core/src/scheduler.rs:317-338) sees "not done" for every block, always.

Measured downstream cost: a production resume recomputed 82,125 already-marked blocks with skip count 0 (volara/slabreg on an AWS cluster, 2026-08-11). The failure is invisible by construction — no error, no warning, just full-price recomputation.

The fix (two commits, independently revertable)

  1. v1_compat.py: wrap check_function exactly as process_function is wrapped (kwarg and positional forms; positional index per _CHECK_FN_ARG_INDEX = 5, _task.py:115).
  2. py_callbacks.rs: log the swallowed exception at warn level before returning false — a check that errors should never be indistinguishable from a check that returns false. This class of bug was undetectable; now it announces itself.

Reproduction

New test (tests/test_funlib_interop.py, parametrized kwarg/positional): a v1-idiom check_function doing funlib arithmetic on block.write_roi.offset, with per-block file markers, run_blockwise twice.

On stock v2.0 @ b323d78:

assert 0 == 4  ...  Skipped: 0    # run 2: Completed: 4 — every done block re-ran
TypeError: unsupported operand type(s) for +: 'daisy._daisy.Coordinate' and 'daisy._daisy.Coordinate'

On this branch: the same file — 9 passed (was 2 failed, 7 passed).

The warning fix, demonstrated in isolation (commit 1 temporarily reverted, extension rebuilt): the previously-invisible failure now announces itself, once per task:

WARNING daisy: check_function for task "compat-check" raised; treating block compat-check/0 as
not done, so it will be (re)processed. Further check failures for this task will not be logged:
Traceback (most recent call last): ... TypeError: unsupported operand type(s) for -: ...

A regression test pins the once-per-task rate limit (2 blocks, always-raising check → exactly 1 WARNING, both blocks processed; fail-open semantics deliberately preserved).

Notes

  • Full suite at stock parity on the branch: 2 failed, 242 passed, 2 skipped, 1 xfailed — the 2 are the hardware-parallelism tests failing identically on stock on this 2-vCPU host. Also verified: clean cross-merge with fix(spawn): emit a logdir-complete context on both spawn channels, and make context_with_logdir public #79 (shared file py_callbacks.rs, disjoint regions), rebuilt and both PRs' test sets green together on the merged tree (35 passed). Explicit check_function=None verified unaffected (_wrap_block_fn passes None through). (Baseline on this 2-vCPU host: only the two hardware-parallelism tests fail, identically on stock v2.0.)
  • Lint: touched files ruff check + ruff format --check clean; repo-wide I001 count went 28 → 27 (one pre-existing violation in a touched test file fixed to keep the file clean). ty check daisy-py/python/daisy passes.
  • Opened as a draft per our review flow. Companion context: this is the mechanism behind the "resumes recompute everything" reports downstream (e11bio volara/slabreg); with this fix their existing v1-idiom checks work unchanged.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ELENZ8uo6Pc1qshkep6iXH

Jeff Rhoades and others added 2 commits August 11, 2026 20:13
Task.__new__ on the compat surface wrapped process_function via
_wrap_block_fn (native block -> funlib-typed compat Block) but never
check_function. A v1-style check that does funlib arithmetic on
block.write_roi.offset therefore got a raw native block and raised
TypeError — and PyCheckBlock::check turns every Python exception into
'not done' (.unwrap_or(false)), so every done block silently re-ran on
resume. Wrap check_function identically, in both the kwarg and the
positional (arg index 5) forms.

Test: a compat task whose check_function does funlib Coordinate
arithmetic on the block's write_roi, with a per-block marker file as
the done state; run_blockwise twice must skip all blocks on run 2
(skipped_count == 4). Fails on v2.0 (skipped_count == 0: every block
re-ran), passes with the fix. Parametrized over kwarg and positional
construction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELENZ8uo6Pc1qshkep6iXH
PyCheckBlock::check turned every Python exception into 'not done' via
a bare .unwrap_or(false): a raising check re-ran every done block on
resume with no trace anywhere. Keep the fail-open semantics (a broken
check must not kill the run) but log the exception as a WARNING on the
'daisy' Python logger with the capped formatted traceback (same
formatting as PyProcessBlock::process / PySpawnWorker's wrap_err),
best-effort like PyProgressObserver. Once per task via an AtomicBool
on the struct — PyCheckBlock is built once per task in py_task.rs, and
a broken check raises identically for every block.

Python logging rather than tracing because the extension installs no
tracing subscriber; logging integration is the Python-side carve-out.

Test: an always-raising check_function still processes every block,
and exactly one WARNING carrying the exception text lands on the
'daisy' logger. (Import sort in the touched test file fixed to make
it lint-clean; it was one of the repo's pre-existing I001s.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELENZ8uo6Pc1qshkep6iXH
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant