Skip to content

bug(router): relay circuit limits are hardcoded to go-libp2p defaults, and a relay reset surfaces as a bare 502 #483

Description

@kaisoz

Problem

Nodes behind NAT reach each other through circuit-relay-v2 on the router. The
router sets up the relay with go-libp2p's default resources
(relay.New(hostNode, relay.WithACL(...)), internal/router/router.go:410),
so every relayed connection is reset after 2 minutes or 128 KiB per
direction
, whichever comes first (go-libp2p v0.49.0,
p2p/protocol/circuitv2/relay/resources.go:63-66). Both limits count over the
lifetime of the relayed connection, not per request.

Nothing in SAM exposes these limits, so on a mesh where peers only talk
through the relay (every node behind NAT, e.g. a sam-one on Cloud Run with
laptop, phone and VM nodes):

  • Any request that stays open longer than 2 minutes dies mid-flight. A
    blocking A2A SendMessage to an agent that runs an LLM and delegates to
    another agent routinely exceeds this.
  • Any exchange that moves more than 128 KiB over one relayed connection dies,
    e.g. file artifacts or a long MCP session.

What the caller sees

The egress proxy (createEgressProxy, internal/node/sidecar.go:690) is an
httputil.ReverseProxy with no ErrorHandler. When the relay resets the
stream, the transport returns network.ErrReset and the caller gets a plain
502 Bad Gateway with an empty body. The callee keeps working: the remote
agent carries on, delegates to a third agent and finishes, while the caller
has already reported failure. Nothing in the 502 points at the
relay, so the natural reading ("the remote service crashed") is wrong.

Repro

  1. Run the router (or sam-one) with two nodes that cannot connect directly,
    so traffic between them is relayed.
  2. Register a service on node B whose backend sleeps 150 s before responding.
  3. Call it from node A through the sidecar:
    curl -H 'X-Sam-Authentication: Bearer <token>' http://127.0.0.1:<port>/sam/<peer-B>/<type>/<name>/
  4. After ~120 s the call returns 502, while B's backend is still running.

Proposal

  1. Configurable relay limits on sam-router and sam-one: a duration
    and a data limit for relayed connections, passed to
    relay.WithResources. Keep the go-libp2p defaults when unset. Allow 0
    to mean "no limit" (a nil Resources.Limit makes the relay unlimited,
    relay.go:473). Operators of a relay-only mesh can then size it for
    their workloads.
  2. Explicit error on relay reset: give the egress proxy an
    ErrorHandler that recognises network.ErrReset on a limited (relayed)
    connection and answers with a body that names the cause, e.g.
    502 relayed connection to <peer> was reset by the relay (limit: 2m / 128KiB),
    and logs it at warn level on the caller's node. The status can stay 502;
    the point is a body and a log line that name the relay limit.

Out of scope: clients that avoid long-lived requests (A2A ReturnImmediately
plus polling GetTask) already survive the resets; that is a per-client
choice and does not remove the 128 KiB cap.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions