Skip to content

Miniflare resets the connection when a Worker responds without reading the request body; workerd does this whenever the body crossed a service binding #15819

Description

@michael-huffaker

What versions & operating system are you using?

  • Windows 11 (10.0.26200), Node 24.14.1. Measured on this machine only.
  • miniflare 5.20260921.0-alpha (the latest tag) with its workerd 1.20260921.1, and that workerd run on its own.
  • The newest workerd, 1.20260923.1, gives the same results, on its own and under the same Miniflare (through an npm overrides entry).
  • Where we first saw it: @cloudflare/vite-plugin 1.57.3 and wrangler 4.136.3, with Miniflare's undici 7.29.0.

Please provide a link to a minimal reproduction

Inline below: npm i miniflare, then node repro.mjs. It runs Miniflare, and it runs the workerd that Miniflare installs directly, without Miniflare.

Describe the Bug

When a Worker responds without reading the request body, and the body reached it through a service binding, workerd resets the client's connection a few milliseconds after the response. Miniflare's entry Worker passes each request to the user's Worker through a service binding (await service.fetch(request) in packages/miniflare/src/workers/core/entry.worker.ts), so under Miniflare this happens to any Worker that answers early - for example one that refuses an oversized upload with 413 before reading it. A client that has not yet read the response when the reset arrives loses it: all 20 did when the client started reading 300 ms after sending.

Without the binding, workerd keeps the connection, as Node's HTTP server does. That fits KJ's HttpServer: when a service responds without reading the whole body, it reads and discards up to canceledUploadGraceBytes (64 KiB) within canceledUploadGracePeriod (1 s) before deciding whether to close (HttpServerSettings in kj/compat/http.h). Through a binding, a 5,000-byte body, well inside that grace, is reset as if there were none.

Since workerd alone reproduces it, the fault looks to be in workerd rather than Miniflare. We are filing it here because Miniflare is where it surfaces; it may belong in cloudflare/workerd.

From repro.mjs below, with workerd 1.20260921.1. Each request sends its headers and its whole body in one write, and every server answers 413. Every row is 20 requests with the same outcome. "Kept" means the connection was still open 2 s later, past the 1 s grace; "reset" means the client's socket failed with ECONNRESET.

Server Body Client reads at once Client reads 300 ms later
workerd: Worker does not read the body 5,000 bytes 413, kept 413, kept
workerd: Worker forwards through a service binding to that Worker 5,000 bytes 413, then reset 3.3-4.2 ms later no response, reset
Miniflare: Worker does not read the body 5,000 bytes 413, then reset 3.2-33.9 ms later no response, reset
workerd, control: Worker reads the body 5,000 bytes 413, kept 413, kept
workerd, control: forwards through a service binding to a Worker that reads it 5,000 bytes 413, kept 413, kept
Miniflare, control: Worker reads the body 5,000 bytes 413, kept 413, kept
Node http server, reference: does not read the body 5,000 bytes 413, kept 413, kept
workerd: Worker does not read the body 100,000 bytes, past the grace 413, then reset 0.1-0.2 ms later no response, reset

RFC 9112 §9.6 is why a reset matters: "If a server performs an immediate close of a TCP connection, there is a significant risk that the client will not be able to read the last HTTP response."

How it shows up. We found it as a rare 500 from vite preview, on a POST that a Worker refuses with 413 before reading the body: the Vite plugin's dispatchFetch rejected with TypeError: fetch failed (stack below) and the plugin answered 500. In runs of 300 and 1,000 requests through mf.dispatchFetch, with a streamed 5,011-byte body and 16 threads keeping every core busy, one request in each failed that way, with read ECONNRESET as the cause. On an idle machine all 1,000 succeeded, and so did a later 8,000 with a 5,000-byte body under the same or heavier load. It takes the client being slow to read at the wrong moment; repro.mjs controls when the client reads, so it reproduces every time.

Related. #15709 looks like a relative: a Worker cancels a ~2 MB body part-way under unstable_dev, and the connection is lost. The last row above shows that even served directly, workerd resets once more than the 64 KiB grace is left unread: that is KJ's grace running out, and the staged close below would address it. This report is the case inside the grace, which fails only because the body crossed a binding - and in Miniflare every body has.

What this does not establish.

  • Measured on Windows 11 only. Whether a reset erases a response the client has already received depends on the client's TCP stack; on Windows it does (the 300 ms column).
  • We haven't traced where the drain is lost after a binding. The reset follows the response by a few milliseconds, far sooner than the 1 s grace, so it looks as if the drain is not attempted, or fails at once.
  • Not tried against deployed Workers.

Suggested fix

  • Give an unread body that crossed a service binding the same grace as one that did not: drain it up to canceledUploadGraceBytes within canceledUploadGracePeriod, then keep the connection or close it cleanly.
  • Past the grace, the staged close RFC 9112 §9.6 describes would stop the reset from erasing the response already sent: "First, the server performs a half-close by closing only the write side of the read/write connection. The server then continues to read from the connection until it receives a corresponding close by the client, or until the server is reasonably certain that its own TCP stack has received the client's acknowledgement of the packet(s) containing the server's last response. Finally, the server fully closes the connection." That would cover [Bug]: unstable_dev PUT after oversized-body cancellation returns 500 "Network connection lost" #15709 as well.

Please provide any relevant error logs

The error behind the 500, as vite preview printed it (paths shortened, frames elided):

TypeError: fetch failed
    at Object.processResponse (...node_modules/undici/lib/web/fetch/index.js:237:16)
    ...
    at async fetch4 (...node_modules/miniflare/dist/src/index.js:64836:20)
    at async _Miniflare.dispatchFetch (...node_modules/miniflare/dist/src/index.js:119777:22)
    at async file:///.../node_modules/@cloudflare/vite-plugin/dist/index.mjs:51227:19
repro.mjs
// A request body that crosses a service binding and is not read gets the connection reset.
// npm i miniflare (which brings workerd), then node repro.mjs. Runs workerd on its own with four
// Workers, Miniflare with one, and a Node HTTP server as a reference. Every request goes over a raw
// socket and declares and sends its body with the headers in one write; every server answers 413.
import { spawn } from "node:child_process";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import http from "node:http";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import { convertV4MiniflareOptions, Miniflare } from "miniflare";
import workerd from "workerd";

const REFUSE = `export default { async fetch() { return new Response("too large", { status: 413 }); } };`;
const READ = `export default { async fetch(request) { await request.arrayBuffer(); return new Response("too large", { status: 413 }); } };`;
const FORWARD = `export default { async fetch(request, env) { return env.TARGET.fetch(request); } };`;
const worker = (script, target) =>
  `(modules = [(name = "worker.js", esModule = ${JSON.stringify(script)})], compatibilityDate = "2026-09-01"` +
  (target ? `, bindings = [(name = "TARGET", service = "${target}")])` : ")");

const PORT = { refuse: 18701, read: 18702, forwardToRefuse: 18703, forwardToRead: 18704, node: 18705 };
const config = `using Workerd = import "/workerd/workerd.capnp";
const config :Workerd.Config = (
  services = [
    (name = "refuse", worker = ${worker(REFUSE)}),
    (name = "read", worker = ${worker(READ)}),
    (name = "forwardToRefuse", worker = ${worker(FORWARD, "refuse")}),
    (name = "forwardToRead", worker = ${worker(FORWARD, "read")}),
  ],
  sockets = [
    (name = "refuse", address = "127.0.0.1:${PORT.refuse}", http = (), service = "refuse"),
    (name = "read", address = "127.0.0.1:${PORT.read}", http = (), service = "read"),
    (name = "forwardToRefuse", address = "127.0.0.1:${PORT.forwardToRefuse}", http = (), service = "forwardToRefuse"),
    (name = "forwardToRead", address = "127.0.0.1:${PORT.forwardToRead}", http = (), service = "forwardToRead"),
  ],
);
`;

// One request with a body of `bytes`. The client starts reading the response `readAfterMs` after
// sending, as a busy client would.
function probe(port, bytes, readAfterMs) {
  return new Promise((resolve) => {
    const socket = net.connect(port, "127.0.0.1");
    const chunks = [];
    let respondedAt;
    let settled = false;
    const finish = (ending) => {
      if (settled) return;
      settled = true;
      const status = /^HTTP\/1\.1 (\d{3})/.exec(Buffer.concat(chunks).toString("latin1"))?.[1];
      const after = respondedAt === undefined ? undefined : performance.now() - respondedAt;
      resolve({ outcome: `${status ?? "no response"}, then ${ending}`, after });
      socket.destroy();
    };
    socket.on("connect", () => {
      socket.write(`POST / HTTP/1.1\r\nHost: localhost\r\nContent-Length: ${bytes}\r\n\r\n${"x".repeat(bytes)}`);
      if (readAfterMs > 0) {
        socket.pause();
        setTimeout(() => socket.resume(), readAfterMs);
      }
    });
    socket.on("data", (chunk) => {
      respondedAt ??= performance.now();
      chunks.push(chunk);
    });
    socket.on("end", () => finish("FIN"));
    socket.on("error", (error) => finish(error.code));
    socket.setTimeout(2000, () => finish("connection still open 2 s later"));
  });
}

async function listening(port) {
  for (let attempt = 0; attempt < 100; attempt++) {
    const open = await new Promise((resolve) => {
      const socket = net.connect(port, "127.0.0.1", () => {
        socket.destroy();
        resolve(true);
      });
      socket.on("error", () => resolve(false));
    });
    if (open) return;
    await new Promise((resolve) => setTimeout(resolve, 100));
  }
  throw new Error(`nothing is listening on ${port}`);
}

const folder = mkdtempSync(path.join(os.tmpdir(), "workerd-repro-"));
writeFileSync(path.join(folder, "config.capnp"), config);
const child = spawn(workerd.default, ["serve", path.join(folder, "config.capnp")], {
  stdio: ["ignore", "inherit", "inherit"],
  windowsHide: true,
});
const reference = http.createServer((request, response) => response.writeHead(413).end("too large"));
await new Promise((resolve) => reference.listen(PORT.node, "127.0.0.1", resolve));
const miniflare = (script) =>
  new Miniflare(convertV4MiniflareOptions({ modules: true, compatibilityDate: "2026-09-01", script }));
const miniflares = { refuse: miniflare(REFUSE), read: miniflare(READ) };

try {
  await Promise.all(Object.values(PORT).map(listening));
  const miniflarePort = async (mf) => Number((await mf.ready).port);
  console.log(`workerd ${workerd.version}, Node ${process.version}, ${os.type()} ${os.release()}; 20 requests per row`);
  const rows = [
    ["workerd: Worker does not read the body", PORT.refuse, 5000],
    ["workerd: Worker forwards through a service binding to that Worker", PORT.forwardToRefuse, 5000],
    ["workerd, control: Worker reads the body", PORT.read, 5000],
    ["workerd, control: Worker forwards through a service binding to the reading Worker", PORT.forwardToRead, 5000],
    ["Miniflare: Worker does not read the body", await miniflarePort(miniflares.refuse), 5000],
    ["Miniflare, control: Worker reads the body", await miniflarePort(miniflares.read), 5000],
    ["Node http server, reference: does not read the body", PORT.node, 5000],
    ["workerd: Worker does not read the body", PORT.refuse, 100000],
  ];
  for (const [name, port, bytes] of rows) {
    for (const readAfterMs of [0, 300]) {
      const outcomes = new Map();
      const after = [];
      for (let i = 0; i < 20; i++) {
        const result = await probe(port, bytes, readAfterMs);
        outcomes.set(result.outcome, (outcomes.get(result.outcome) ?? 0) + 1);
        if (readAfterMs === 0 && result.after !== undefined && !result.outcome.includes("still open")) {
          after.push(result.after);
        }
      }
      const counts = [...outcomes].map(([outcome, n]) => `${n} x ${outcome}`).join("; ");
      const timing = after.length === 0
        ? ""
        : ` (closed ${Math.min(...after).toFixed(1)}-${Math.max(...after).toFixed(1)} ms after the response arrived)`;
      console.log(`${name}, ${bytes}-byte body, client reads after ${readAfterMs} ms: ${counts}${timing}`);
    }
  }
} finally {
  reference.close();
  await Promise.all(Object.values(miniflares).map((mf) => mf.dispose()));
  child.kill();
  await new Promise((resolve) => (child.exitCode !== null ? resolve() : child.once("exit", resolve)));
  rmSync(folder, { recursive: true, force: true });
}
Its output with workerd 1.20260921.1, and with 1.20260923.1
workerd 1.20260921.1, Node v24.14.1, Windows_NT 10.0.26200; 20 requests per row
workerd: Worker does not read the body, 5000-byte body, client reads after 0 ms: 20 x 413, then connection still open 2 s later
workerd: Worker does not read the body, 5000-byte body, client reads after 300 ms: 20 x 413, then connection still open 2 s later
workerd: Worker forwards through a service binding to that Worker, 5000-byte body, client reads after 0 ms: 20 x 413, then ECONNRESET (closed 3.3-4.2 ms after the response arrived)
workerd: Worker forwards through a service binding to that Worker, 5000-byte body, client reads after 300 ms: 20 x no response, then ECONNRESET
workerd, control: Worker reads the body, 5000-byte body, client reads after 0 ms: 20 x 413, then connection still open 2 s later
workerd, control: Worker reads the body, 5000-byte body, client reads after 300 ms: 20 x 413, then connection still open 2 s later
workerd, control: Worker forwards through a service binding to the reading Worker, 5000-byte body, client reads after 0 ms: 20 x 413, then connection still open 2 s later
workerd, control: Worker forwards through a service binding to the reading Worker, 5000-byte body, client reads after 300 ms: 20 x 413, then connection still open 2 s later
Miniflare: Worker does not read the body, 5000-byte body, client reads after 0 ms: 20 x 413, then ECONNRESET (closed 3.2-33.9 ms after the response arrived)
Miniflare: Worker does not read the body, 5000-byte body, client reads after 300 ms: 20 x no response, then ECONNRESET
Miniflare, control: Worker reads the body, 5000-byte body, client reads after 0 ms: 20 x 413, then connection still open 2 s later
Miniflare, control: Worker reads the body, 5000-byte body, client reads after 300 ms: 20 x 413, then connection still open 2 s later
Node http server, reference: does not read the body, 5000-byte body, client reads after 0 ms: 20 x 413, then connection still open 2 s later
Node http server, reference: does not read the body, 5000-byte body, client reads after 300 ms: 20 x 413, then connection still open 2 s later
workerd: Worker does not read the body, 100000-byte body, client reads after 0 ms: 20 x 413, then ECONNRESET (closed 0.1-0.2 ms after the response arrived)
workerd: Worker does not read the body, 100000-byte body, client reads after 300 ms: 20 x no response, then ECONNRESET
workerd 1.20260923.1, Node v24.14.1, Windows_NT 10.0.26200; 20 requests per row
workerd: Worker does not read the body, 5000-byte body, client reads after 0 ms: 20 x 413, then connection still open 2 s later
workerd: Worker does not read the body, 5000-byte body, client reads after 300 ms: 20 x 413, then connection still open 2 s later
workerd: Worker forwards through a service binding to that Worker, 5000-byte body, client reads after 0 ms: 20 x 413, then ECONNRESET (closed 5.1-6.5 ms after the response arrived)
workerd: Worker forwards through a service binding to that Worker, 5000-byte body, client reads after 300 ms: 20 x no response, then ECONNRESET
workerd, control: Worker reads the body, 5000-byte body, client reads after 0 ms: 20 x 413, then connection still open 2 s later
workerd, control: Worker reads the body, 5000-byte body, client reads after 300 ms: 20 x 413, then connection still open 2 s later
workerd, control: Worker forwards through a service binding to the reading Worker, 5000-byte body, client reads after 0 ms: 20 x 413, then connection still open 2 s later
workerd, control: Worker forwards through a service binding to the reading Worker, 5000-byte body, client reads after 300 ms: 20 x 413, then connection still open 2 s later
Miniflare: Worker does not read the body, 5000-byte body, client reads after 0 ms: 20 x 413, then ECONNRESET (closed 5.5-42.3 ms after the response arrived)
Miniflare: Worker does not read the body, 5000-byte body, client reads after 300 ms: 20 x no response, then ECONNRESET
Miniflare, control: Worker reads the body, 5000-byte body, client reads after 0 ms: 20 x 413, then connection still open 2 s later
Miniflare, control: Worker reads the body, 5000-byte body, client reads after 300 ms: 20 x 413, then connection still open 2 s later
Node http server, reference: does not read the body, 5000-byte body, client reads after 0 ms: 20 x 413, then connection still open 2 s later
Node http server, reference: does not read the body, 5000-byte body, client reads after 300 ms: 20 x 413, then connection still open 2 s later
workerd: Worker does not read the body, 100000-byte body, client reads after 0 ms: 20 x 413, then ECONNRESET (closed 0.1-0.2 ms after the response arrived)
workerd: Worker does not read the body, 100000-byte body, client reads after 300 ms: 20 x no response, then ECONNRESET

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    package:miniflareRelating to Miniflareupstream:workerdRoot cause is in the workerd runtime: https://github.com/cloudflare/workerd

    Type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions