Send tier2 jobs in the order the client reads them

[sduchesneau]

Jobs of a tier1 request race each other for whatever tier2 instance has room. A tier2 at its concurrent-request limit refuses the job without doing any work, the job redials and can land anywhere, so the segment the client reads first is no more likely to land than one it will only read minutes later. When the low segment loses that race a few times, the execout walker stalls on it and the whole request idles behind it.

This adds a per-request LaunchQueue (orchestrator/work/launchqueue.go). A job asks it for a turn before every request it sends to a tier2, first attempt and retries alike.

The queue holds the jobs waiting to get in, ordered by what the client reads first: lowest segment, then highest stage within a segment (that job also produces the partials of the stages under it). A job leaves the queue the moment a tier2 takes it — not when it finishes — so the jobs behind it move up while it runs.

The window. Only the first 20% of the queue may dial at all, at least two jobs and at most 15. The ones behind them send nothing — they are not retrying slowly, they are not dialing. Each admission moves the window up by one job.

No queue, no wait. A job with nothing queued ahead of it dials immediately and never joins, so a fleet with room is paced exactly as before.

One retry path, two delays

derr.RetryContext and its growing 1s-to-5s backoff are gone from the worker. It was only ever contributing the sleep: maxRetries and maxExecutionTimeouts are hand-rolled locals in the loop, and derr was passed math.MaxUint64 retries. Every reason to dial again now goes through the queue, which means retries keep the request's reading order instead of jumping ahead of it. Two delays replace the curve:

• 100ms — the job never got in and nothing ran: refused for capacity, connection refused, or no healthy upstream. Another instance may have room right now. Not charged to any counter, as before.
• 5s, once — the job got in and then failed: an execution timeout or any other retryable error. If a tier2 then turns it away for capacity on the way back in, it is back on the 100ms delay with its failure already counted.

The counters that end a hopeless request are unchanged: five failures, or two execution timeouts.

What it looks like with 10 workers, fleet full

Jobs for segments 1-10 are dispatched. Segment 1 dials, is refused, joins the queue; segments 2-10 join behind it. Segments 1 and 2 redial every 100ms; segments 3-10 send nothing. A tier2 frees a slot and it goes to segment 1 or 2 — it cannot go to segment 7, which isn't dialing. Segment 1 gets in and leaves, and segment 3 starts dialing.

At 50 workers the window is 10 jobs. At 200 it is 15, not 40: past a few tens of workers a wider window only spreads the fleet's free slots over more jobs without getting the client its first segments any sooner.

Because the order is the segment and not the job's age, a job the scheduler re-creates late for a low segment (the re-run after a missing execout file) goes to the front of the queue and dials on its next tick.

Also here

orchestrator/scheduler/scheduler.go now schedules jobs up to 2x the request's worker count ahead of the blocks the client is reading, instead of 1.5x.

Trade-offs worth knowing

• A job dialing, or waiting out its 5s after a failure, still holds its window slot. At 10 workers that leaves one other job dialing meanwhile.
• Under pressure the request fills freed slots at up to the window size per 100ms. Jobs run for seconds, so this doesn't cost throughput, but it is a real change.
• The dial load on the fleet drops at high worker counts (15 per 100ms at 200 workers, against 200 per 300ms before) and barely moves at low ones. SUBSTREAMS_WORKER_LAUNCH_WINDOW_PERCENT (default 20) and SUBSTREAMS_WORKER_LAUNCH_WINDOW_MAX (default 15) are the levers.

Tests

launchqueue_test.go covers the window sizing and its cap, the launch order, no wait while nothing is queued, a job outside the window staying silent until the window reaches it, a late low segment going to the front, the head's redial cadence, and cancellation. worker_overloaded_test.go adds two worker-level tests: a job leaves the queue while it is still running, and a failure waits once without putting the capacity refusals that follow it on the long delay. go test -race ./orchestrator/... is clean.

[dfuse-bot]

🔍 Vulnerabilities of ghcr.io/streamingfast/substreams:6489bbd

📦 Image Reference ghcr.io/streamingfast/substreams:6489bbd

digest	sha256:3c2ec1fd635d9bb05e9548b691a1726f248f756e7a73db46bbd5b0d3e47a1a8d
vulnerabilities
platform	linux/amd64
size	124 MB
packages	380

📦 Base Image ubuntu:24.04

also known as	c1ca75be10a22ea09ff0b7bbe8b82ee03553a4f9b795030ee2ec921e42418fc8 noble noble-20260810
digest	sha256:1e0a86e57d247923571b75e0aaf48a1449cf8c543d51fb3e07a4a7d7bfa79316
vulnerabilities