fix(v1_compat): stop silently neutralizing check_function — wrap it like process_function, and log swallowed check exceptions - #80
Open
rhoadesScholar wants to merge 2 commits into
Conversation
Task.__new__ on the compat surface wrapped process_function via _wrap_block_fn (native block -> funlib-typed compat Block) but never check_function. A v1-style check that does funlib arithmetic on block.write_roi.offset therefore got a raw native block and raised TypeError — and PyCheckBlock::check turns every Python exception into 'not done' (.unwrap_or(false)), so every done block silently re-ran on resume. Wrap check_function identically, in both the kwarg and the positional (arg index 5) forms. Test: a compat task whose check_function does funlib Coordinate arithmetic on the block's write_roi, with a per-block marker file as the done state; run_blockwise twice must skip all blocks on run 2 (skipped_count == 4). Fails on v2.0 (skipped_count == 0: every block re-ran), passes with the fix. Parametrized over kwarg and positional construction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELENZ8uo6Pc1qshkep6iXH
PyCheckBlock::check turned every Python exception into 'not done' via a bare .unwrap_or(false): a raising check re-ran every done block on resume with no trace anywhere. Keep the fail-open semantics (a broken check must not kill the run) but log the exception as a WARNING on the 'daisy' Python logger with the capped formatted traceback (same formatting as PyProcessBlock::process / PySpawnWorker's wrap_err), best-effort like PyProgressObserver. Once per task via an AtomicBool on the struct — PyCheckBlock is built once per task in py_task.rs, and a broken check raises identically for every block. Python logging rather than tracing because the extension installs no tracing subscriber; logging integration is the Python-side carve-out. Test: an always-raising check_function still processes every block, and exactly one WARNING carrying the exception text lands on the 'daisy' logger. (Import sort in the touched test file fixed to make it lint-clean; it was one of the repo's pre-existing I001s.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELENZ8uo6Pc1qshkep6iXH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What breaks today
A daisy-1.x-style
check_function— one that does funlib arithmetic onblock.write_roi.offset, the documented idiom downstream consumers use for resume checks — is silently neutralized under v2's compat layer, so resumed runs recompute every done block:Task.__new__wrapsprocess_functionin the native→funlib block converter but notcheck_function(daisy-py/python/daisy/v1_compat.py:225-228), so the check receives a raw native block.daisy._daisy.Coordinatehas no arithmetic dunders (daisy-py/src/py_roi.rs:46-105), so the v1-style check raisesTypeError: unsupported operand type(s) for -: 'daisy._daisy.Coordinate' and 'Coordinate'.PyCheckBlock::checkswallows every Python exception intofalsewith no log line (daisy-py/src/py_callbacks.rs:71-79,.unwrap_or(false)), so the scheduler's precheck (daisy-core/src/scheduler.rs:317-338) sees "not done" for every block, always.Measured downstream cost: a production resume recomputed 82,125 already-marked blocks with skip count 0 (volara/slabreg on an AWS cluster, 2026-08-11). The failure is invisible by construction — no error, no warning, just full-price recomputation.
The fix (two commits, independently revertable)
v1_compat.py: wrapcheck_functionexactly asprocess_functionis wrapped (kwarg and positional forms; positional index per_CHECK_FN_ARG_INDEX = 5,_task.py:115).py_callbacks.rs: log the swallowed exception at warn level before returningfalse— a check that errors should never be indistinguishable from a check that returns false. This class of bug was undetectable; now it announces itself.Reproduction
New test (
tests/test_funlib_interop.py, parametrized kwarg/positional): a v1-idiomcheck_functiondoing funlib arithmetic onblock.write_roi.offset, with per-block file markers,run_blockwisetwice.On stock
v2.0@b323d78:On this branch: the same file —
9 passed(was 2 failed, 7 passed).The warning fix, demonstrated in isolation (commit 1 temporarily reverted, extension rebuilt): the previously-invisible failure now announces itself, once per task:
A regression test pins the once-per-task rate limit (2 blocks, always-raising check → exactly 1 WARNING, both blocks processed; fail-open semantics deliberately preserved).
Notes
2 failed, 242 passed, 2 skipped, 1 xfailed— the 2 are the hardware-parallelism tests failing identically on stock on this 2-vCPU host. Also verified: clean cross-merge with fix(spawn): emit a logdir-complete context on both spawn channels, and make context_with_logdir public #79 (shared filepy_callbacks.rs, disjoint regions), rebuilt and both PRs' test sets green together on the merged tree (35 passed). Explicitcheck_function=Noneverified unaffected (_wrap_block_fnpasses None through). (Baseline on this 2-vCPU host: only the two hardware-parallelism tests fail, identically on stockv2.0.)ruff check+ruff format --checkclean; repo-wide I001 count went 28 → 27 (one pre-existing violation in a touched test file fixed to keep the file clean).ty check daisy-py/python/daisypasses.🤖 Generated with Claude Code
https://claude.ai/code/session_01ELENZ8uo6Pc1qshkep6iXH