Current outcome
Preserve legitimate filtering/reordering while ensuring failed or malformed stages cannot silently erase known errors or run dependent effects on unprocessed input.
Replanned on 2026-09-06 against GoBatch master 63ef757 and ShitQuant's recorder, enrichment, paper/replay, and flow workloads. This is an implementation target, not a claim that the behavior already exists.
Required behavior
Verification and completion
Prove downstream side effects do not run after a stage failure, valid Filter/reordering/drop-all remains supported, and malformed output preserves known failure reporting. Check #82 and #79 integration.
Follow the repository's formatting, race-test, vet/lint, package documentation, example and changelog requirements for the changed surface. Report the actual supported behavior and migration; do not treat a passing coverage percentage as proof of these outcomes.
Scope and relationships
Depends on #79's diagnostic identity/cardinality contract. Request/reply has a separate stricter one-terminal-outcome-per-accepted-request contract; do not impose that on valid stream filtering.
Design history
The earlier report/prototype remains below for provenance; the requirements above supersede conflicting prescriptions. Existing discussion is preserved.
Original issue: Define the Processor truncation contract: nil/short returned slices silently drop items (and their errors)
On master @ 58cee73, doProcessors rebinds the batch to whatever each processor returns (batch/batch.go:416-436). A processor that follows the standard Go idiom return nil, err on failure silently wipes the entire batch from accounting: downstream processors receive nil, the final per-item Error scan sees nothing, and only the single stage-wide error is reported — per-item tracking (a documented core feature) is lost. Intentional truncation (Filter) is indistinguishable from accidental loss, so the engine cannot warn.
PR #66 fixes the Filter-specific instance (errored items passed through), but any custom processor can still do this.
Ask: make the contract explicit and loud on the Processor interface godoc — "return every item you did not deliberately drop; on error, return the input slice, not nil" — and consider having the engine treat a nil return with a non-nil error as "keep previous items" (defensive, zero cost on the happy path).
Relations: #79 (error identity/shape), #77 (processor requirements), PR #66 (Filter instance). From a deep engine review.
Scope addition (from the PR #66 review discussion): the same engine-level guard should also consider returned slices containing nil elements — today a nil item from a buggy processor panics in Transform/Channel/Error and the engine's final error scan. A per-processor nil check was declined on #66 as inconsistent piecemeal defense; if nil tolerance is wanted, it belongs here, in the engine/contract.
Current outcome
Preserve legitimate filtering/reordering while ensuring failed or malformed stages cannot silently erase known errors or run dependent effects on unprocessed input.
Replanned on 2026-09-06 against GoBatch master
63ef757and ShitQuant's recorder, enrichment, paper/replay, and flow workloads. This is an implementation target, not a claim that the behavior already exists.Required behavior
Verification and completion
Prove downstream side effects do not run after a stage failure, valid Filter/reordering/drop-all remains supported, and malformed output preserves known failure reporting. Check #82 and #79 integration.
Follow the repository's formatting, race-test, vet/lint, package documentation, example and changelog requirements for the changed surface. Report the actual supported behavior and migration; do not treat a passing coverage percentage as proof of these outcomes.
Scope and relationships
Depends on #79's diagnostic identity/cardinality contract. Request/reply has a separate stricter one-terminal-outcome-per-accepted-request contract; do not impose that on valid stream filtering.
Design history
The earlier report/prototype remains below for provenance; the requirements above supersede conflicting prescriptions. Existing discussion is preserved.
Original issue: Define the Processor truncation contract: nil/short returned slices silently drop items (and their errors)
On master @ 58cee73,
doProcessorsrebinds the batch to whatever each processor returns (batch/batch.go:416-436). A processor that follows the standard Go idiomreturn nil, erron failure silently wipes the entire batch from accounting: downstream processors receive nil, the final per-itemErrorscan sees nothing, and only the single stage-wide error is reported — per-item tracking (a documented core feature) is lost. Intentional truncation (Filter) is indistinguishable from accidental loss, so the engine cannot warn.PR #66 fixes the Filter-specific instance (errored items passed through), but any custom processor can still do this.
Ask: make the contract explicit and loud on the
Processorinterface godoc — "return every item you did not deliberately drop; on error, return the input slice, not nil" — and consider having the engine treat a nil return with a non-nil error as "keep previous items" (defensive, zero cost on the happy path).Relations: #79 (error identity/shape), #77 (processor requirements), PR #66 (Filter instance). From a deep engine review.
Scope addition (from the PR #66 review discussion): the same engine-level guard should also consider returned slices containing nil elements — today a nil item from a buggy processor panics in Transform/Channel/Error and the engine's final error scan. A per-processor nil check was declined on #66 as inconsistent piecemeal defense; if nil tolerance is wanted, it belongs here, in the engine/contract.