Skip to content

Wake active DynamicConfig waits and define atomic update boundaries #90

Description

@MasterOfBinary

Current outcome

Allow a valid runtime formation update to wake a collector blocked on too few items, even when there are no incoming items or active timers.

Replanned on 2026-09-06 against GoBatch master 63ef757 and ShitQuant's recorder, enrichment, paper/replay, and flow workloads. This is an implementation target, not a claim that the behavior already exists.

Required behavior

  • Introduce an update notification/version contract that wakes idle waits; polling Get only after another receive/timer event does not solve the reported stall.
  • Validate updates atomically; a rejected update retains the prior configuration.
  • Specify effects on a partial batch, threshold decreases, timer shortening/lengthening, cancellation and EOF races.
  • Keep Bound stream execution and validate formation/resource configuration #73 hard safety limits immutable to formation tuning; updates cannot grow work past the configured bounds.
  • Do not change already-dispatched batches. Document boundary-only behavior for legacy/custom Config implementations without notifications.
  • Use a deterministic notification/barrier mechanism with no periodic polling sleeps required for correctness.

Verification and completion

Reproduce the no-new-input/no-timer stuck wait, apply an update and prove it wakes. Test timer/partial-batch changes, invalid updates, cancel/EOF races and safety limits.

Follow the repository's formatting, race-test, vet/lint, package documentation, example and changelog requirements for the changed surface. Report the actual supported behavior and migration; do not treat a passing coverage percentage as proof of these outcomes.

Scope and relationships

Depends on #73's configuration/limit contract. Coordinate #84 docs and #86 behavioral coverage; this is formation tuning, not a provider rate limiter.

Design history

The earlier report/prototype remains below for provenance; the requirements above supersede conflicting prescriptions. Existing discussion is preserved.

Original issue: DynamicConfig updates are never observed by an in-flight batch wait

doProcessing snapshots the config once per batch (batch/batch.go:407):

config := fixConfig(b.config.Get())
batch := b.waitForItems(ctx, config)

waitForItems then runs to completion against that snapshot. With {MinItems: 100, MaxTime: 0} and an idle source, the wait can block indefinitely — and a later DynamicConfig.UpdateBatchSize(1, 0) is never observed, because Config.Get() is not re-polled inside the wait. The pipeline stalls forever, which defeats the runtime-adjustability use case the DynamicConfig godoc advertises.

Suggested fix: re-poll Config.Get() inside waitForItems (e.g. on each timer/receive iteration) so updates can unstick an in-flight wait. Alternatively, document batch-boundary granularity — but that leaves the stalled-pipeline footgun in place, so the code fix is preferred.

Related: #86 covers the missing test, #84 covers the doc gap; neither covers this behavior bug.

Found in the 2026-07-10 full-repo review.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions