Problem
Nodes behind NAT reach each other through circuit-relay-v2 on the router. The
router sets up the relay with go-libp2p's default resources
(relay.New(hostNode, relay.WithACL(...)), internal/router/router.go:410),
so every relayed connection is reset after 2 minutes or 128 KiB per
direction, whichever comes first (go-libp2p v0.49.0,
p2p/protocol/circuitv2/relay/resources.go:63-66). Both limits count over the
lifetime of the relayed connection, not per request.
Nothing in SAM exposes these limits, so on a mesh where peers only talk
through the relay (every node behind NAT, e.g. a sam-one on Cloud Run with
laptop, phone and VM nodes):
- Any request that stays open longer than 2 minutes dies mid-flight. A
blocking A2A SendMessage to an agent that runs an LLM and delegates to
another agent routinely exceeds this.
- Any exchange that moves more than 128 KiB over one relayed connection dies,
e.g. file artifacts or a long MCP session.
What the caller sees
The egress proxy (createEgressProxy, internal/node/sidecar.go:690) is an
httputil.ReverseProxy with no ErrorHandler. When the relay resets the
stream, the transport returns network.ErrReset and the caller gets a plain
502 Bad Gateway with an empty body. The callee keeps working: the remote
agent carries on, delegates to a third agent and finishes, while the caller
has already reported failure. Nothing in the 502 points at the
relay, so the natural reading ("the remote service crashed") is wrong.
Repro
- Run the router (or
sam-one) with two nodes that cannot connect directly,
so traffic between them is relayed.
- Register a service on node B whose backend sleeps 150 s before responding.
- Call it from node A through the sidecar:
curl -H 'X-Sam-Authentication: Bearer <token>' http://127.0.0.1:<port>/sam/<peer-B>/<type>/<name>/
- After ~120 s the call returns
502, while B's backend is still running.
Proposal
- Configurable relay limits on
sam-router and sam-one: a duration
and a data limit for relayed connections, passed to
relay.WithResources. Keep the go-libp2p defaults when unset. Allow 0
to mean "no limit" (a nil Resources.Limit makes the relay unlimited,
relay.go:473). Operators of a relay-only mesh can then size it for
their workloads.
- Explicit error on relay reset: give the egress proxy an
ErrorHandler that recognises network.ErrReset on a limited (relayed)
connection and answers with a body that names the cause, e.g.
502 relayed connection to <peer> was reset by the relay (limit: 2m / 128KiB),
and logs it at warn level on the caller's node. The status can stay 502;
the point is a body and a log line that name the relay limit.
Out of scope: clients that avoid long-lived requests (A2A ReturnImmediately
plus polling GetTask) already survive the resets; that is a per-client
choice and does not remove the 128 KiB cap.
Problem
Nodes behind NAT reach each other through circuit-relay-v2 on the router. The
router sets up the relay with go-libp2p's default resources
(
relay.New(hostNode, relay.WithACL(...)),internal/router/router.go:410),so every relayed connection is reset after 2 minutes or 128 KiB per
direction, whichever comes first (go-libp2p v0.49.0,
p2p/protocol/circuitv2/relay/resources.go:63-66). Both limits count over thelifetime of the relayed connection, not per request.
Nothing in SAM exposes these limits, so on a mesh where peers only talk
through the relay (every node behind NAT, e.g. a
sam-oneon Cloud Run withlaptop, phone and VM nodes):
blocking A2A
SendMessageto an agent that runs an LLM and delegates toanother agent routinely exceeds this.
e.g. file artifacts or a long MCP session.
What the caller sees
The egress proxy (
createEgressProxy,internal/node/sidecar.go:690) is anhttputil.ReverseProxywith noErrorHandler. When the relay resets thestream, the transport returns
network.ErrResetand the caller gets a plain502 Bad Gatewaywith an empty body. The callee keeps working: the remoteagent carries on, delegates to a third agent and finishes, while the caller
has already reported failure. Nothing in the 502 points at the
relay, so the natural reading ("the remote service crashed") is wrong.
Repro
sam-one) with two nodes that cannot connect directly,so traffic between them is relayed.
curl -H 'X-Sam-Authentication: Bearer <token>' http://127.0.0.1:<port>/sam/<peer-B>/<type>/<name>/502, while B's backend is still running.Proposal
sam-routerandsam-one: a durationand a data limit for relayed connections, passed to
relay.WithResources. Keep the go-libp2p defaults when unset. Allow0to mean "no limit" (a nil
Resources.Limitmakes the relay unlimited,relay.go:473). Operators of a relay-only mesh can then size it fortheir workloads.
ErrorHandlerthat recognisesnetwork.ErrReseton a limited (relayed)connection and answers with a body that names the cause, e.g.
502 relayed connection to <peer> was reset by the relay (limit: 2m / 128KiB),and logs it at warn level on the caller's node. The status can stay 502;
the point is a body and a log line that name the relay limit.
Out of scope: clients that avoid long-lived requests (A2A
ReturnImmediatelyplus polling
GetTask) already survive the resets; that is a per-clientchoice and does not remove the 128 KiB cap.