Evict requests on tier1 when the pod's CPU is saturated

[sduchesneau]

Off by default. Enable with --substreams-tier1-cpu-eviction-mode (needs the companion firehose-core change that registers the flags).

Problem

Request-count limits balance the number of streams per tier1 pod, but a few CPU-heavy requests can land on the same pod and saturate its CPU quota. Every stream on that pod then lags. The hub does not lag, so head_block_drift stays flat and nothing detects the condition. Rollouts make it worse: reconnecting streams all do catchup at once, which is the worst-case CPU profile, on the fewest ready pods.

What this does

A tier1 pod watches its own cgroup CPU. When it is saturated it advertises unready, waits for the load balancer to stop routing to it, then cancels the most expensive and least important streams with Unavailable so those clients reconnect to a less busy pod.

One signal drives the decision: CPU usage as a fraction of the cpu.max quota, from the container's own cgroup v2, so neighbour pods on the same node do not distort it. Per-request CPU comes from the wasm execution timer that already exists (module processing time minus external-call waits), diffed each tick into a burn rate in cores — no new hot-path instrumentation.

One overload state: usage over threshold (0.90) held for sustain (15s). The pod goes unready, waits drain-delay, then cancels one batch, sized so the surviving burn fits under target-ratio (0.75) of quota. It is ready again once usage stays under recover-threshold (0.75) for recover-sustain (30s).

Only requests burning real CPU are candidates: under min-burn-cores (0.05) cancelling one frees nothing and costs a reconnect, and requests younger than min-age (90s) are never selected — startup and store loading look hot. Among the candidates, victims go least important first: dev-mode, then production on live blocks, then production still catching up from files, highest burn first within each class.

Live before catchup is deliberate, and it is the opposite of what most people expect. A live request burning 2 cores keeps burning them for as long as it stays connected, so moving it is the only thing that helps. A catchup request burning 2 cores is spending them on work that ends: it reaches the head and goes cheap on its own, and cancelling it throws away segment progress that has to be redone elsewhere. The cost of the choice is that a cancelled live stream is visible to a human immediately (reconnect plus store load on the new pod) while a cancelled catchup one is not.

Ordering matters: the pod flips unready and waits drain-delay before the first cancellation, so the balancer has stopped routing to it by the time the evicted clients reconnect. Otherwise they land back on the same pod.

Autoscaling

New substreams_tier1_effective_active_requests gauge, meant to replace substreams_active_requests as the horizontal autoscaler input:

max(activeRequests, nominalCapacity * cpuUsageRatio / targetRatio)

Scaling on the plain count breaks once eviction is on, because the count measures what the pod is serving and eviction lowers it — so overload would make the autoscaler shrink the fleet. Since eviction holds CPU at target-ratio, a pod that has evicted its way back down still reports itself at capacity instead of reporting the few requests it has left.

Set nominal-capacity to the autoscaler's per-pod request target. It defaults to the active-requests soft limit, which is usually higher.

Modes

off (default), observe, dev-only, full. observe evaluates, logs every would-be victim and publishes all metrics, but never touches readiness, admission or any stream — there is a test pinning that.

Metrics

substreams_tier1_cpu_quota_cores, _cpu_usage_ratio, _cpu_overloaded, substreams_tier1_evicted_requests_counter{class,action}, substreams_tier1_effective_active_requests. Each eviction also logs a structured line with trace_id, class, burn cores, age, current block and the usage ratio at the time.

Testing

Unit tests cover cgroup parsing, the burn-rate math, victim selection and its class order, the sustain and recovery windows, the observe-mode guard and the autoscaler metric.

tests_e2e/cpu_eviction_test.go drives the whole path against a real tier1: hand-written cpu.max/cpu.stat files under SUBSTREAMS_CGROUP_DIR, with usage_usec fed from the process's own CPU time, advertise a 6-core quota, and eight concurrent dev-mode requests over a 6000-block archive burn about 7.3 cores against it. Two production requests stream live blocks alongside them. From one run:

• usage ratio 1.215 (7.29 cores against the 6-core quota), overload declared after the 3s sustain
• one batch of 4 requests cancelled 2s later, after the pod went unready and the drain delay elapsed. Excess was (1.212 - 0.60) × 6 = 3.67 cores and the four victims summed to 3.925 cores — the smallest set that covers it; three would have been 2.95
• every victim was class=dev, taken highest burn first: 0.9821, 0.9817, 0.9811, 0.9804 cores
• per-request attribution matched reality: eight serial wasm loops measured ~0.98 cores each against 7.29 cores of real process CPU
• the two live production requests were never candidates — under min-burn-cores — and kept streaming throughout
• clients got Unavailable: server CPU overloaded, please reconnect; the four dev survivors ran to completion
• the pod advertised unready before the first cancellation and became ready again once the load stopped

Ran green 8 times in a row. It needs Docker and takes ~30s.

One thing the harness surfaced that is worth knowing before full: when the overload is deep enough that one batch cannot reach the target, a second batch follows after the cooldown. In an earlier tuning (target 0.5, 3-core quota) the first batch cut 3.9 cores of attributed burn but total usage only fell by 1.5 cores, so the evictor fired again 5s later. Attributed wasm burn is not the pod's whole CPU — the Go-side work around it (protobuf, compression, egress) is not attributed to any request — so batch sizing can undershoot. That is the cooldown doing its job rather than a bug, but it is slower than "one batch and done", and the pprof trace_id-label comparison in the plan is the check that says how large the unattributed share is on a real pod.

Deploying

Flag registration is a companion firehose-core PR. The HPA change (switch to effective_active_requests, long scaleDown stabilization) lives in the deploy repo; details are in plans/2026-08-28-cpu-request-cutoff.md, including the trap that Kubernetes picks unready pods first when scaling down.

Suggested rollout: observe first, then dev-only, then full with the HPA already switched over — evicting without adding capacity just moves the pain around.

[dfuse-bot]

🔍 Vulnerabilities of ghcr.io/streamingfast/substreams:62b0e1a

📦 Image Reference ghcr.io/streamingfast/substreams:62b0e1a

digest	sha256:54b6924b6e9e4dcebf240f61f7e6df110b00072be9f0a63bc5042da05734c76a
vulnerabilities
platform	linux/amd64
size	124 MB
packages	380

📦 Base Image ubuntu:24.04

also known as	c1ca75be10a22ea09ff0b7bbe8b82ee03553a4f9b795030ee2ec921e42418fc8 noble noble-20260810
digest	sha256:1e0a86e57d247923571b75e0aaf48a1449cf8c543d51fb3e07a4a7d7bfa79316
vulnerabilities