Skip to content

Make Transform StopOnError fail closed without duplicate failure reports #82

Description

@MasterOfBinary

Current outcome

After a transform fails, an application must not receive untransformed remainder as successful downstream output or count one item failure twice.

Replanned on 2026-09-06 against GoBatch master 63ef757 and ShitQuant's recorder, enrichment, paper/replay, and flow workloads. This is an implementation target, not a claim that the behavior already exists.

Required behavior

Verification and completion

Use a downstream effect recorder to prove unprocessed data is not emitted as success; assert prefix/failure/suffix outcomes and no indistinguishable double reporting.

Follow the repository's formatting, race-test, vet/lint, package documentation, example and changelog requirements for the changed surface. Report the actual supported behavior and migration; do not treat a passing coverage percentage as proof of these outcomes.

Scope and relationships

Follows #79 and #87 contracts. Does not move ShitQuant's required-risk gate into a generic processor.

Design history

The earlier report/prototype remains below for provenance; the requirements above supersede conflicting prescriptions. Existing discussion is preserved.

Original issue: processor.Transform with StopOnError double-reports the failing item and lets the rest of the batch flow downstream unmarked

On master @ 58cee73, processor/transform.go:40-46:

newData, err := p.Func(item.Data)
if err != nil {
    item.Error = err
    if p.StopOnError {
        return items, err
    }
}

The same failure is recorded twice: item.Error = err and the identical err returned as the stage error. doProcessors then emits &ProcessorError{Err: err} for the returned error (batch/batch.go:427) and a second &ProcessorError{Err: item.Error} for the same item in the final per-item scan (batch/batch.go:433) — two errors on the channel for one failure, breaking error counts for callers.

Additionally, when StopOnError aborts mid-batch, the items after the failing one are returned untransformed with Error == nil, so downstream processors (e.g. processor.Channel) emit raw, untransformed Data with no indication the stage stopped.

Suggested fix: pick one reporting path (e.g. under StopOnError return the stage error and don't also set item.Error, or mark the remaining items as skipped) and document the contract on the type; add a test pinning exactly one reported error per failure.

Relations: same territory as #79 (error-plane shape); independent of PR #66 (Filter).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions