[nikhilunni]

Problem

A periodic checkpoint uploads its dirty chunks to GCS in a burst (up to 64 chunk PUTs in flight). GCS sometimes refuses a write with HTTP 429: "Your request distribution is too uneven across the key-ranges in your bucket". This is a rate limit: the bucket's write rate grew faster than GCS spreads the load. The blob client classed the 429 as a generic SDK error and retried it after 100 ms and 200 ms. It gave up after about 300 ms in all.

When the give-up happens in a diff capture's sparse re-chunk, Firecracker has already consumed the dirty bitmap. The host-agent drops the checkpoint chain (engram_checkpoint_chain_poisoned_total, which an alert watches), and the next checkpoint is a full snapshot. No data is lost, but one checkpoint is lost, and the full snapshot adds upload load to a bucket that is already throttling.

In production logs, one host retried several hundred PUTs in half a day. About one in six first retries failed again, and the host gave up a few times and poisoned its chain twice. The errors came in bursts during large uploads.

Change

1. core, storage: a refusal for load is its own error. New variant BlobError::Throttled(BoxError). GCS maps HTTP 429 and 503 to it, and S3 maps 429 and 503 (SlowDown). The blob client treats it as transient, and the chunk store's adaptive admission counts it as overload. This commit does not change behaviour: every op retries a throttle on the schedule it had before.

2. blob-client: a throttled PUT backs off for seconds. A PUT that fails with Throttled uses ClientPolicy::put_throttled:

	value
tries	6 (1 + 5 retries)
ceiling before retry n	min(1 s * 2^(n-1), 16 s)
delay	in [ceiling/2, ceiling)
total backoff before give-up	15.5 s to 31 s

Why these values:

• Google's retry strategy says to retry 408, 429 and 5xx with exponential backoff and jitter. Its client libraries start at 1 s and double. They cap one step at 30 s or more (Java: 32 s cap, 50 s total).
• The request-rate guidance says that Cloud Storage adds capacity as the rate grows, which "can take minutes". It also says to pause or reduce the rate when errors occur. A 100 ms retry lands in the same window.
• The total stays near half a minute. A periodic checkpoint holds its sandbox's capture lock while it uploads, so an eviction or a snapshot of that sandbox waits behind it. A throttle that lasts longer than this is a real failure, and it surfaces as before.
• Each delay is at least half its ceiling, so the store always gets a pause. The other half is jitter, so the concurrent PUTs of one upload do not all retry at the same time.

The jitter is a SplitMix64 hash of a per-client call number and the retry number, not a random draw. The crate had no randomness and documented deterministic backoff. A hash keeps a run replayable, and concurrent calls still get different delays.

What stays the same. GET, ranged GET, HEAD, DELETE and LIST keep the 100 ms / 200 ms schedule for every transient error, a throttle included, because a guest can be waiting on a demand GET. A PUT that fails with a non-throttle transient error (a reset connection, a deadline) also keeps the old schedule.

Key layout

Chunk keys are chunks/sha256/<2 hex>/<62 hex>: one fixed prefix, then the content hash. That is the random naming that the naming-convention guidance recommends, and there is no sequential part. Manifests are manifests/<uuid v4>/v<n>.json, which is also random. This PR does not change the layout. The 429 comes from the write rate of a burst, not from a hot key range.

Upload concurrency (follow-up, not in this PR)

Chunk PUTs already go through the host's adaptive admission limit (engram-chunk-store/src/admission.rs), together with fetches. Background-class uploads use at most bulk_cap slots. A PUT holds its permit while it backs off, so a throttled burst sends fewer requests, and other classes keep their priority. The tuner sees the throttle only when a PUT gives up, or as a drop in throughput after slow completions.

A possible follow-up: report each Throttled attempt to the admission tuner (from the chunk store or through a small hook in the blob client), so that the limit falls on the first 429 and not after the retries. This needs a seam between the blob client and the admission, and the evidence does not yet show that it is necessary.

Eviction captures

An eviction capture is less exposed. It persists its finalize record before the uploads start. A failed upload makes the finalizer re-drive the leg with backoff, up to 10 attempts, and it does not poison the chain. The cost of a throttle there is a later eviction finish. With this PR, a throttled PUT in that leg also waits for seconds, so fewer re-drives are necessary.

Test

• engram-blob-client unit tests (paused tokio time, a fake store that answers Throttled N times, no real sleeps):
  • throttled_put_backs_off_for_seconds_then_lands: three throttles, then the PUT lands after 3.5 s to 7 s of virtual time.
  • throttled_put_that_never_clears_gives_up_within_the_budget: 6 tries and 15.5 s to 31 s, and the error is Throttled.
  • throttled_get_keeps_the_short_schedule: 3 tries and exactly 300 ms.
  • throttle_delays_stay_in_range_and_spread_across_calls: every delay is in [c/2, c), delays spread across calls, and the same input gives the same delay.
• engram-storage-gcs: status_codes_map_to_blob_errors (429 and 503 map to Throttled, 404 to NotFound, 500 and 403 to Sdk).
• Local runs: cargo fmt --all --check, cargo clippy -D warnings --all-targets on the six touched crates, and cargo nextest run on engram-blob-client, engram-storage-gcs, engram-storage-s3, engram-core, engram-testkit (192 passed) and engram-chunk-store (212 passed). The CI cargo nextest run --workspace step runs these tests.
• The S3 mapping has no unit test. Building an SdkError with a raw 503 response needs the smithy test types, and the emulator lanes cannot make MinIO throttle.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HNhpmz4QPYaVCE7tFna54D