feat(router): configurable relay circuit limits - #498
Conversation
The router relay used go-libp2p's defaults, cutting every relayed connection after 2 minutes or 128 KiB per direction. Add --relay-limit-duration (default 1h) and --relay-limit-data (default no limit, accepts sizes like 512MiB) to sam-router, and the matching --router-relay-limit-* tunables to sam-one. 0 means no limit. go.mod: go-humanize and golang.org/x/sys become direct dependencies (the latter was already imported by the mobile FFI). Part of google#483
There was a problem hiding this comment.
Code Review
This pull request introduces configuration options and command-line flags to cap relayed connections by duration and data limits across sam-router and sam-one, utilizing the go-humanize library to parse human-readable byte sizes. The review feedback highlights a critical issue where limits exceeding math.MaxUint32 will silently wrap around during go-libp2p serialization, potentially cutting connections immediately; the reviewer suggests capping these values to math.MaxUint32 and updating the test suite accordingly. Additionally, a mismatched protocol ID ('test' instead of '/test') was identified in the unit tests for the limited connection stream.
| func relayLimit(duration time.Duration, data int64) *relay.RelayLimit { | ||
| if duration <= 0 && data <= 0 { | ||
| return nil | ||
| } | ||
| limit := &relay.RelayLimit{Duration: duration, Data: data} | ||
| if duration <= 0 { | ||
| // The relay advertises seconds as uint32; anything larger wraps. | ||
| limit.Duration = math.MaxUint32 * time.Second | ||
| } | ||
| if data <= 0 { | ||
| limit.Data = math.MaxInt64 | ||
| } | ||
| return limit | ||
| } |
There was a problem hiding this comment.
In the libp2p circuit relay v2 protocol, both duration (in seconds) and data (in bytes) are represented as uint32 on the wire. If the configured limits exceed math.MaxUint32 (e.g., math.MaxInt64 or a user-configured value like 5GB), they will silently wrap around when serialized by go-libp2p. For example, a 5GB limit wraps around to 1GB, and a 4GB limit wraps to 0 (which cuts the connection immediately). To prevent silent wrap-around, we should explicitly cap both duration and data to math.MaxUint32 (or math.MaxUint32 * time.Second for duration).
| func relayLimit(duration time.Duration, data int64) *relay.RelayLimit { | |
| if duration <= 0 && data <= 0 { | |
| return nil | |
| } | |
| limit := &relay.RelayLimit{Duration: duration, Data: data} | |
| if duration <= 0 { | |
| // The relay advertises seconds as uint32; anything larger wraps. | |
| limit.Duration = math.MaxUint32 * time.Second | |
| } | |
| if data <= 0 { | |
| limit.Data = math.MaxInt64 | |
| } | |
| return limit | |
| } | |
| func relayLimit(duration time.Duration, data int64) *relay.RelayLimit { | |
| if duration <= 0 && data <= 0 { | |
| return nil | |
| } | |
| limit := &relay.RelayLimit{Duration: duration, Data: data} | |
| if duration <= 0 || duration > math.MaxUint32*time.Second { | |
| // The relay advertises seconds as uint32; anything larger wraps. | |
| limit.Duration = math.MaxUint32 * time.Second | |
| } | |
| if data <= 0 || data > math.MaxUint32 { | |
| limit.Data = math.MaxUint32 | |
| } | |
| return limit | |
| } |
There was a problem hiding this comment.
Only Duration is uint32 on the wire; Data is uint64 (go-libp2p v0.49.0 p2p/protocol/circuitv2/pb/circuit.pb.go:454), so byte limits don't wrap and capping them at 4 GiB would add a hidden limit. Also, the relay enforces its local rc.Limit (deadline + io.LimitReader, relay/relay.go:473-478); the wire Limit is only advertised to the client, so a wrapped value couldn't cut a connection. The only case that would advertise a wrong number is a duration above ~136 years. Leaving as is.
| func TestRelayLimit(t *testing.T) { | ||
| unlimitedDuration := math.MaxUint32 * time.Second | ||
| for _, tc := range []struct { | ||
| name string | ||
| duration time.Duration | ||
| data int64 | ||
| want *relay.RelayLimit | ||
| }{ | ||
| {"no limits", 0, 0, nil}, | ||
| {"duration only", time.Hour, 0, &relay.RelayLimit{Duration: time.Hour, Data: math.MaxInt64}}, | ||
| {"data only", 0, 4096, &relay.RelayLimit{Duration: unlimitedDuration, Data: 4096}}, | ||
| {"both", time.Minute, 4096, &relay.RelayLimit{Duration: time.Minute, Data: 4096}}, | ||
| } { | ||
| t.Run(tc.name, func(t *testing.T) { | ||
| if got := relayLimit(tc.duration, tc.data); !reflect.DeepEqual(got, tc.want) { | ||
| t.Errorf("relayLimit(%v, %d) = %+v, want %+v", tc.duration, tc.data, got, tc.want) | ||
| } | ||
| }) | ||
| } | ||
| } |
There was a problem hiding this comment.
Update the test cases to align with the new capping behavior in relayLimit (where unlimited or overflowing values are capped to math.MaxUint32 instead of math.MaxInt64), and add test cases to verify that overflow is correctly handled.
func TestRelayLimit(t *testing.T) {
unlimitedDuration := math.MaxUint32 * time.Second
for _, tc := range []struct {
name string
duration time.Duration
data int64
want *relay.RelayLimit
}{
{"no limits", 0, 0, nil},
{"duration only", time.Hour, 0, &relay.RelayLimit{Duration: time.Hour, Data: math.MaxUint32}},
{"data only", 0, 4096, &relay.RelayLimit{Duration: unlimitedDuration, Data: 4096}},
{"both", time.Minute, 4096, &relay.RelayLimit{Duration: time.Minute, Data: 4096}},
{"duration overflow", 200 * 365 * 24 * time.Hour, 0, &relay.RelayLimit{Duration: unlimitedDuration, Data: math.MaxUint32}},
{"data overflow", 0, 5 << 30, &relay.RelayLimit{Duration: unlimitedDuration, Data: math.MaxUint32}},
} {
t.Run(tc.name, func(t *testing.T) {
if got := relayLimit(tc.duration, tc.data); !reflect.DeepEqual(got, tc.want) {
t.Errorf("relayLimit(%v, %d) = %+v, want %+v", tc.duration, tc.data, got, tc.want)
}
})
}
}There was a problem hiding this comment.
Follows from the thread above: no capping change, so the expectations stay.
| t.Fatal(err) | ||
| } | ||
|
|
||
| s, err := src.NewStream(network.WithAllowLimitedConn(ctx, "test"), dst.ID(), "/test") |
There was a problem hiding this comment.
The protocol ID registered on the destination host is "/test" (line 97), and the stream is opened with "/test" (line 120). However, WithAllowLimitedConn is called with "test". Since protocol IDs must match exactly, this should be "/test" to correctly allow the protocol on the limited connection.
| s, err := src.NewStream(network.WithAllowLimitedConn(ctx, "test"), dst.ID(), "/test") | |
| s, err := src.NewStream(network.WithAllowLimitedConn(ctx, "/test"), dst.ID(), "/test") |
There was a problem hiding this comment.
The second argument of network.WithAllowLimitedConn is a free-form reason string, not a protocol ID (go-libp2p v0.49.0 core/network/context.go:100), so "test" is fine.
The router started circuit-relay-v2 with go-libp2p's default resources, so every relayed connection was reset after 2 minutes or 128 KiB per direction. On a mesh where nodes only reach each other through the relay (every node behind NAT, e.g.
sam-oneon Cloud Run), long blocking calls and large transfers died mid-flight.sam-router:--relay-limit-duration(default1h) and--relay-limit-data(default no limit; accepts sizes like512MiB).0means no limit.sam-one:--router-relay-limit-duration/--router-relay-limit-data;0keeps the component default, like the other--router-*tunables.relayLimitmaps a single0to the maximum value.Part of #483