Preserve item/stage error identity and define reporting cardinality

[MasterOfBinary]

Current outcome

Let applications distinguish source failures, stage failures and individual item failures and correlate an item failure back to its input. The stream error channel remains diagnostics, not a durable completion or eligibility ledger.

Replanned on 2026-09-06 against GoBatch master 63ef757 and ShitQuant's recorder, enrichment, paper/replay, and flow workloads. This is an implementation target, not a claim that the behavior already exists.

Required behavior

• Give stage-wide errors a stage identity and item errors the input Item.ID, plus stage identity where known.
• Treat Item.ID zero as valid; use an explicit discriminator or distinct error type rather than zero meaning no item.
• Preserve errors.Is/As using one documented wrapper form, coordinated with Document and test pointer error wrappers and errors.Is/As #91; avoid retaining generic payloads in diagnostic errors by default.
• Define reporting cardinality: do not report one underlying item failure twice as indistinguishable item and stage errors. A separate stage-abort summary must be distinguishable.
• Define error representation so known causes/identities can be retained. Implementation of malformed/truncated-output preservation belongs to Define stream filtering and fail-closed stage outcomes without silent loss #87, and prefix/failure/suffix handling to Make Transform StopOnError fail closed without duplicate failure reports #82; those follow this issue without blocking its completion.
• Document caller error-draining/backpressure obligations and cancellation loss policy honestly. Error reporting alone does not prevent downstream side effects.

Verification and completion

Test distinguishable source/stage/item representations, valid ID zero, wrapped causes and reporting cardinality on the existing emission path. Follow-up #87/#82 tests prove their new failure/skip boundaries. Reconcile send-site changes with #65.

Follow the repository's formatting, race-test, vet/lint, package documentation, example and changelog requirements for the changed surface. Report the actual supported behavior and migration; do not treat a passing coverage percentage as proof of these outcomes.

Scope and relationships

Does not force RequestBatcher to inherit stream semantics or block its independent result design. #87 owns stream continuation/filter rules; #91 owns pointer-wrapper usage docs and examples.

Design history

The earlier report/prototype remains below for provenance; the requirements above supersede conflicting prescriptions. Existing discussion is preserved.

Original issue: Error channel conflates per-item and processor-wide errors, and discards Item.ID

Summary

When doProcessors reports errors on the error channel, it loses two distinct pieces of information:

1. Per-item errors and processor-wide errors are indistinguishable. Both are wrapped as ProcessorError, so a caller using errors.As(err, &pe) cannot tell whether a whole processing stage failed or whether a single item failed within an otherwise-healthy batch.
2. The failing item's ID is thrown away. A per-item error is wrapped as &ProcessorError{Err: item.Error} with no reference to the Item it came from, so callers can't correlate the error back to the input item.

This contradicts the documented contract in CLAUDE.md ("Items with errors are tracked individually through the Error field") — the per-item tracking exists right up until the error hits the channel, where the ID is dropped and the error is relabeled as a generic stage error.

Where

batch/batch.go, in doProcessors:

var err error
items, err = proc.Process(ctx, items)
if err != nil {
    b.errs <- &ProcessorError{Err: err}        // batch.go:424 — processor-WIDE error
}
// ...
for _, item := range items {
    if item.Error != nil {
        b.errs <- &ProcessorError{Err: item.Error} // batch.go:430 — PER-ITEM error, ID lost
    }
}

Both sends produce the same ProcessorError type, and the second discards item.ID.

Impact

• A caller cannot route/retry/dead-letter a specific failed item, because it has no idea which item (by ID) failed.
• A caller cannot distinguish "stage N blew up" from "item 4,217 had a validation error" — both surface identically as processor error: ....
• Defeats the purpose of Item carrying both a unique ID and a per-item Error field.

Possible directions

(Design decision — listing options, not prescribing one.)

• Introduce a distinct ItemError type that carries the ID (and optionally the offending Item/Data), e.g.:

  type ItemError struct {
      ItemID uint64
      Err    error
  }
  func (e ItemError) Error() string { return fmt.Sprintf("item %d error: %v", e.ItemID, e.Err) }
  func (e ItemError) Unwrap() error { return e.Err }

  and send &ItemError{ItemID: item.ID, Err: item.Error} from the per-item path while keeping ProcessorError for stage-wide failures.
• Alternatively, attach the ID to ProcessorError (optional field, zero when stage-wide) so existing errors.As(err, &ProcessorError{}) callers keep working.

Either way: stage-wide vs per-item should be distinguishable, and the per-item path should preserve Item.ID.

Notes

• This is the one item-error-handling gap from the recent codebase review not already covered by existing issues/PRs: fix: preserve errored items through Filter to prevent silent error loss #66 fixes Filter silently dropping errored items, and fix(batch): panic recovery, deadlock-safe error sends, bounded prealloc, Done() race #65 documents the errors.As pointer-target form — but neither addresses the conflation or the lost ID described here.
• Breaking-change surface depends on the chosen direction: adding ItemError is additive (existing ProcessorError matches break for per-item errors only); adding a field to ProcessorError is fully backwards-compatible. v0, so either is acceptable with a CHANGELOG note.

References

• batch/batch.go — doProcessors per-item vs processor-wide error sends.
• batch/errors.go — current ProcessorError / SourceError definitions.
• Related: fix: preserve errored items through Filter to prevent silent error loss #66 (Filter error passthrough), fix(batch): panic recovery, deadlock-safe error sends, bounded prealloc, Done() race #65 (engine hardening / errors.As guidance), Track consumer API hardening for bounded batch, request and flow workloads #75 (API hardening tracker).

[MasterOfBinary]

Validated against master @ 58cee73 — all claims hold. The two sends are now batch/batch.go:427 (stage-wide) and :433 (per-item, ID discarded); ProcessorError (batch/errors.go:20) still carries only Err. PR #66 covers only the Filter passthrough and PR #65 only wraps these sends in context-aware selects, so neither resolves this. Recommend implementing after #65 merges since it rewrites the same send sites; an additive ItemError type keeps existing errors.As callers working. Related: a new issue tracks the engine-level truncation contract (nil/short processor returns dropping items entirely).