Summary
retryWithBackoff (core/src/main/scala/app/softnetwork/elastic/client/package.scala, RetryConfig in client/scroll/package.scala) retries with a deterministic exponential backoff — initialDelay = 1 s, backoffFactor = 2.0, maxDelay = 10 s, no jitter. Every retrying caller that hits the same failure at the same moment retries at exactly the same instants.
Until now that was one caller per stream. Since sliced PIT paging (#238, 0.21.0) an un-ordered extraction runs up to min(primary shards, elastic.scroll.max-slices) readers (default cap 8) against one cluster, each wrapped in its own retryWithBackoff. A 429 es_rejected_execution_exception / 503 burst on the search thread pool therefore fails N readers within the same few milliseconds, and N retries land together 1 s later, then 2 s, 4 s, 8 s — the retry itself becomes a synchronized burst against the pool that is already saturated (classic thundering herd). With K concurrent extractions that is K × N lockstep retries.
Not a regression introduced by #238 (the helper predates it; the benchmark concurrency arm at k = 4 × 6 slices measured 0 rejections on 3 × 2-CPU nodes), but slicing multiplies the exposure and the fix is a one-liner in a shared helper — hence its own issue.
Where
retryWithBackoff — core/.../client/package.scala (attempt(retriesLeft, delay), akka.pattern.after(delay, …); nextDelay = min(delay × factor, maxDelay))RetryConfig — core/.../client/scroll/package.scala- Callers: PIT page fetch in es7
RestHighLevelClientApi.pitSearchAfter, es8/es9 JavaClientApi.pitSearchAfter (one per slice), classic scroll and search_after paths, BulkApi has its own private retryWithBackoff (BulkApi.scala) — check whether it has the same shape.
Proposal
- Add jitter to the shared helper — full jitter (
delay × U(0, 1)) or decorrelated jitter (AWS-style min(maxDelay, U(initialDelay, previous × 3))). Full jitter is the simplest and keeps maxDelay as the hard ceiling.
- Surface it on
RetryConfig as jitter: Double = 1.0 (fraction of the delay randomised; 0.0 = today's deterministic behaviour for tests that pin timings), so existing explicit configs compile unchanged.
- Keep the WARN per retry but include the actual delay chosen (
Retrying in 1.37 s (2 retries left)), so a storm is visible in logs.
- Align
BulkApi's private helper (or make it use the shared one) if it has the same deterministic schedule.
Acceptance
- A unit test (no Docker) that runs N = 8 concurrent
retryWithBackoff calls against an operation failing twice with a retriable IOException and asserts the retry instants are NOT identical across callers (spread > 0) while each stays ≤ maxDelay.
jitter = 0.0 reproduces today's schedule exactly (existing timing-sensitive tests keep passing).- es7/es8/es9
ScrollCompletenessSpec / SlicedScrollCompletenessSpec unchanged and green.
Not in scope
Found during the #238 code review (decision item "retry storm without jitter across N slices").
Summary
retryWithBackoff(core/src/main/scala/app/softnetwork/elastic/client/package.scala,RetryConfiginclient/scroll/package.scala) retries with a deterministic exponential backoff —initialDelay = 1 s,backoffFactor = 2.0,maxDelay = 10 s, no jitter. Every retrying caller that hits the same failure at the same moment retries at exactly the same instants.Until now that was one caller per stream. Since sliced PIT paging (#238, 0.21.0) an un-ordered extraction runs up to
min(primary shards, elastic.scroll.max-slices)readers (default cap 8) against one cluster, each wrapped in its ownretryWithBackoff. A429 es_rejected_execution_exception/503burst on the search thread pool therefore fails N readers within the same few milliseconds, and N retries land together 1 s later, then 2 s, 4 s, 8 s — the retry itself becomes a synchronized burst against the pool that is already saturated (classic thundering herd). With K concurrent extractions that is K × N lockstep retries.Not a regression introduced by #238 (the helper predates it; the benchmark concurrency arm at k = 4 × 6 slices measured 0 rejections on 3 × 2-CPU nodes), but slicing multiplies the exposure and the fix is a one-liner in a shared helper — hence its own issue.
Where
retryWithBackoff—core/.../client/package.scala(attempt(retriesLeft, delay),akka.pattern.after(delay, …);nextDelay = min(delay × factor, maxDelay))RetryConfig—core/.../client/scroll/package.scalaRestHighLevelClientApi.pitSearchAfter, es8/es9JavaClientApi.pitSearchAfter(one per slice), classic scroll and search_after paths,BulkApihas its own privateretryWithBackoff(BulkApi.scala) — check whether it has the same shape.Proposal
delay × U(0, 1)) or decorrelated jitter (AWS-stylemin(maxDelay, U(initialDelay, previous × 3))). Full jitter is the simplest and keepsmaxDelayas the hard ceiling.RetryConfigasjitter: Double = 1.0(fraction of the delay randomised;0.0= today's deterministic behaviour for tests that pin timings), so existing explicit configs compile unchanged.Retrying in 1.37 s (2 retries left)), so a storm is visible in logs.BulkApi's private helper (or make it use the shared one) if it has the same deterministic schedule.Acceptance
retryWithBackoffcalls against an operation failing twice with a retriableIOExceptionand asserts the retry instants are NOT identical across callers (spread > 0) while each stays ≤maxDelay.jitter = 0.0reproduces today's schedule exactly (existing timing-sensitive tests keep passing).ScrollCompletenessSpec/SlicedScrollCompletenessSpecunchanged and green.Not in scope
ElasticsearchException-wrapped 429/503 on es8/es9 (today onlyIOException-class failures are retried there, while es7 retries them) — a separate pre-existing asymmetry inisRetriableError, noted in the Row extraction pages Elasticsearch sequentially and does not use shard parallelism #238 review.Found during the #238 code review (decision item "retry storm without jitter across N slices").