relay-worker-miniflare tests hang on unbounded `mf.ready` under suite load

[iceteaSA]

relay-worker-miniflare.test.ts intermittently fails with a 30s timeout, locally and in CI. The failing test varies between runs; the accompanying output is always workerd/server/server.c++: error: disconnected: ... Broken pipe and a killed 1 dangling process.

The hang is in Miniflare startup, not in the websocket exchange the test is named for.

Mechanism

Every timeout inside the harness is bounded — socket open at 5s, waitForControl at 5s. The observed failure is 30s, which is bun's default test timeout, so none of those fired. The only unbounded awaits in the file are await mf.ready, in startWorker and again in connectWorkerSocket.

Under full-suite load, workerd startup can be starved badly enough that mf.ready never settles. Four tests each start their own worker, so whichever one loses the race hangs — which is why the failing test name moves around.

Reproduction

Race mf.ready against an 8s timeout in startWorker:

await Promise.race([
  mf.ready,
  new Promise((_, rej) =>
    setTimeout(() => rej(new Error('MFREADY_TIMEOUT_startWorker')), 8000),
  ),
])

Then run the full suite (bun run test, not the file alone):

error: MFREADY_TIMEOUT_startWorker
(fail) relay Worker under Miniflare > websocket hash mismatch returns 409 before upstream fetch [8001.51ms]
Ran 1359 tests across 45 files. [59.71s]

The instrumented timeout fires on exactly the test that otherwise sits for 30s. A second full run passed, and the file alone passes 4/4 repeatedly — it needs the concurrent load to reproduce.

Measured at 99de290, but it reproduces on a near-upstream tree with none of my changes.

Suggested fix

Bound mf.ready and retry startWorker once on timeout. Startup starvation is transient, so a retry after the spike should succeed, and a bounded failure reports the real cause instead of a bare 30s timeout on an unrelated assertion. Happy to open a PR if you want it.

Correction to my own earlier notes

I characterised this failure three times on PRs #174 and #175 and was wrong each time — "always the last of the four tests", "deterministic, not a flake", and "CI passes it consistently". The first two came from measuring without controlling machine load; the third has since been contradicted by a CI failure. All three were about the test's subject rather than its setup, which is where the fault actually is. Anywhere those claims are quoted, this is the accurate version.

[iceteaSA]

Frequency data point: the timeout has now failed check on two consecutive PR #174 CI runs on the shared runner — run 33337752048 (head c781f0b) and run 33343812764 (head 3531998), both websocket hash mismatch returns 409 before upstream fetch at exactly 30001ms, both solo-green locally 4/4 and green on serial reruns. Local parallel full-suite runs reproduce the same signature intermittently (roughly 1 in 2-3 under load). The failing wait is still worker startup (await mf.ready unbounded), not the websocket exchange — consistent with the original bisect. On PR CI we work around it by amend-retriggering, which is noisy; a bounded mf.ready (fail fast with a named startup-timeout error) or serializing that one file in CI would remove the flake without touching the exchange logic.

[iceteaSA]

Frequency update: three hits today on one branch (runs 33495627135, 33508661808, 33509080794) — two consecutive on the same tree (amend-retriggered, byte-identical), failing different tests in the file each time (websocket hash mismatch, websocket full_sync and patch), always the 30s timeout + Broken pipe signature. The flake rate on shared runners has reached the point where stacked-PR CI is a retrigger lottery. PR implementing the bounded-mf.ready remedy incoming.