[Bug][internal] Streaming threshold handler not invoked; large bodies are buffered

[gsmlg]

Requesting repo: gsmlg-dev/http_fetch on branch main

The library advertises that responses larger than 5MB (or with unknown Content-Length) are streamed into a separate process and exposed via response.stream (with body: nil). In practice, large responses are buffered into the body field and stream is nil. As a consequence, the [:http_fetch, :streaming, :start | :chunk | :stop] telemetry events never fire for these responses.

Root cause (observed)

In lib/http.ex, the streaming logic depends on :httpc's stream: :self client option, which causes :httpc to send a series of {:http, {request_id, :stream, chunk}} messages to the calling process. HTTP.fetch/2 sets this option in handle_httpc_request/4:

httpc_client_opts =
  client_options
  |> Keyword.put(:body_format, :binary)
  |> Keyword.put(:stream, :self)

But the request is also made with the default sync: false (async) client option. When the response is large enough to cross the streaming threshold, the response message is delivered as a complete response tuple (status + headers + body) before the streaming path can take over. The streaming branch in handle_response/2 (matching {:http, {^request_id, :stream_start, _}} or the streaming header sequence) is never reached, so the buffered-body branch runs and body is fully populated.

This breaks two advertised behaviours:

• HTTP.Response.write_to/2 is documented to write the streaming pid's chunks to a file; for large responses, it reads the buffered body instead.
• HTTP.Telemetry events under [:http_fetch, :streaming, ...] never fire for any response, because the streaming handler is never entered.

Reproduction

Using the vendored Go test server (priv/test_server/main.go), which has a /stream-large route that returns a 6MB binary body (crosses the 5MB threshold):

resp =
  "http://127.0.0.1:<port>/stream-large"
  |> HTTP.fetch()
  |> HTTP.Promise.await()

# Expected: %HTTP.Response{body: nil, stream: stream_pid, status: 200}
#            where stream_pid is alive and emits :stream_chunk / :stream_end
# Actual:   %HTTP.Response{body: <<137, 155, 133, 173, ...>>,  # 6MB binary in memory
#            stream: nil, status: 200}

# And telemetry:
:telemetry.attach("test", [:http_fetch, :streaming, :start], &IO.inspect/2, nil)
HTTP.fetch("http://127.0.0.1:<port>/stream-large") |> HTTP.Promise.await()
# Expected: at least one [:http_fetch, :streaming, :start] event delivered
# Actual:   no event delivered

Impact

• HTTP.Response.write_to/2 still "works" (because it falls back to writing body when stream is nil), but it does so by buffering the entire response into the caller's process — defeating the point of streaming.
• Memory usage: a 6MB response stays resident in the calling process for the lifetime of the %HTTP.Response{} struct. For larger responses this is a real risk.
• Telemetry users cannot observe streaming events, which the docstring lists as a feature.

Suggested direction (not requesting a specific implementation)

1. Investigate whether :httpc requires a different combination of sync, stream, and body_format to actually deliver the streaming message sequence (vs. a complete response tuple) on the async path.
2. If :httpc will not cooperate, add a transport-level fallback for large bodies: read the response in chunks from :httpc's async API and forward them to a streaming pid, regardless of the threshold.
3. Make the threshold itself configurable (today it lives in HTTP.Config and is hard-coded to 5MB) so callers can tune it.

Environment

• Elixir 1.18+
• Erlang/OTP 28

Severity

needed (the streaming path is a documented feature and the body: nil / stream: pid contract is a load-bearing part of the public API; current behaviour silently buffers instead)