Extensible program lowering through injected operator handlers

[AlekseiChirkovVention]

Goal

Add a framework-owned program-construction seam that performs graph selection, validation, and traversal while delegating concrete operation emission and optional fusion to consumer-provided handlers. A consumer supplies only handlers keyed by concrete operator type; it never reimplements reachability, ordering, or dependency validation.

What changed

tinychain.autodiff.lowering is a new, additive module. It consumes the structured dependency analysis for reachability, dependency order, and selected-output slicing, then dispatches each operation to a handler registered against its concrete operator type. Handler return values are treated as fully opaque — the framework never compares, hashes, iterates, truth-tests, or stringifies them — so a consumer emits its own target representation with no framework dependency on it.

An optional fusion hook is offered a bounded window of unclaimed operations and may collapse several into one instruction.

Three named invariants

The seam's correctness rests on three properties, each enforced in exactly one place rather than by checks spread along the lowering path:

• Exactly-once lowering. A single claim is the only way an operation becomes lowered and the only way a produced value enters the environment, so a repeated claim fails identically whether it comes from a handler or from a fusion. One set comparison over the whole reachable region at the end catches operations no claim covered — the half a per-operation guard cannot see.
• Fusion liveness. Exactly one value may escape a fused region. Kept deliberately separate from exactly-once: conflating two invariants yields a check that appears to cover both and covers neither.
• Operand availability. Every operand of every claimed operation must be already bound or produced within the claim. This is deliberately not an adjacency check — a hook sees the operations it is offered and never their positions, so an adjacency rule would refuse a legal claim for a reason the consumer cannot observe, and would change meaning if the traversal order were revised.

Every reachable operation must be individually lowerable, with no exception when a hook is present: fusion collapses operations that are already supported rather than substituting for support. An operation with no handler therefore fails before any handler emits a value.

Error handling

Adds one category, handler_contract_violation, for a handler that returns no value or fails with an uncategorized exception. Any ordinary exception a consumer callback raises is normalized to it, including a consumer's own exception classes, which the framework cannot enumerate ahead of time. A categorized error raised by a consumer keeps its own category. Interrupts and interpreter exits propagate untouched.

Acceptance criteria and their tests

44 unit tests in py/tests/test_autodiff_lowering.py.

• The framework owns reachability and traversal — selected-output slicing, dependency order with operands already bound, a repeated operand bound once and passed twice, and every reachable operation lowered exactly once.
• Handlers are keyed by concrete operator identity — a distinct operator type whose route name is literally that of a supported operator must not reach that operator's handler; subclass and superclass registrations stay distinct; duplicate registration is rejected.
• A fake consumer emits its own representation — target values that raise on comparison, hashing, iteration, and truth-testing pass through untouched, and the operation context exposes no mutable framework state.
• Fusion is optional, explicit, deterministic, and preserves provenance — the candidate window is bounded by the declared look-ahead, a declined fusion falls back to per-operation handlers, source provenance is recorded in traversal order regardless of claim order, and a fusion is rejected if it claims an operation twice, claims outside the offered window, omits the operation it was offered, strands a value needed elsewhere, or reads an operand that is neither bound nor produced within the claim.
• Unsupported and malformed inputs fail before execution — no registered handler, a handler returning nothing or failing, cycles, unknown selected outputs, duplicate producers, and missing dependencies, each with its own category and with no handler invoked.

The derivative program path is exercised through the identical seam, including seed and forward-capture bindings.

Test evidence

Test level: unit. The seam is an in-process graph transformation whose only external actor is a consumer-supplied handler, for which a fake consumer is the natural double.

Tests were written first and committed before their implementation: adc27ef and b2aedfc are test-only commits, each observed failing for the right reason beforehand. f8f9cf0 corrects one test whose fixture made its assertions unreachable; all four assertions are verbatim and the replaced fixture is left in place for the test that depends on it.

Focused: 48 passed. Full non-integration suite: 717 passed, 1 skipped, with the 4 failures that pre-date this work unchanged — none vanished, no new one. Deterministic across varied hash seeds.

Stack position

Stack position 2 of 4 — based on feat/derivative-dependency-analysis.

Cannot merge into main before feat/derivative-dependency-analysis.

Notes for review

Public exports are deliberately not added here — that belongs to the public-contract branch at the top of the stack, and the tests import the module directly.

Under the current depth-first traversal, a fusion claim that skips an operation always turns out to read an unavailable operand, so operand availability and adjacency coincide in practice; extensive randomized search found no claim separating them. The guard is written as availability regardless, because that rule stays correct if the traversal order ever changes.