Skip to content

Optimize for sending >8MB in many (10k) sessions - #25

Merged
mzagozen merged 2 commits into
mainfrom
large-writes
Nov 11, 2026
Merged

mzagozen merged 2 commits into
mainfrom
large-writes

Conversation

@mzagozen

@mzagozen mzagozen commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

channel_try_write handed ssh_channel_write() a queued chunk's whole remainder, but the length is a uint32_t. With max_write_buffer=0, a write of 2 GiB or more failed there, and at 4 GiB the length wrapped. Each call now gets at most 256 KiB, so a write of any size goes out in pieces.

The cap also bounds libssh's send buffer. One call encrypts as much as the peer's window allows (OpenSSH: 2 MB), and what the kernel's socket buffer does not take right away waits in libssh's buffer. The next write waits until that buffer is empty, so the cap bounds it per connection: about 300 KiB at 256 KiB, against about 2 MiB without a cap. The cap comes from the scp branch, where Antonio Prcela added it.

With writes of any size working, the second commit makes 0 (no limit) the default max_write_buffer for Client and Server. The 8 MiB default failed a channel on any write that would queue more. SshPipe creates its Clients without the option, so a NETCONF message over 8 MiB could not be sent at all. A value above 0 still fails a channel whose queued writes would exceed it.

256 KiB was selected empirically by running 10,000 concurrent uploads to an sshd over loopback on a 128-thread machine, medians of 2 runs:

cap 10,000 × 32 MiB 10,000 × 512 MiB memory per connection
64 KiB 11,953 MiB/s 13,963 MiB/s 75 KiB
256 KiB 14,141 MiB/s 14,999 MiB/s 295 KiB
1 MiB 12,205 MiB/s 15,127 MiB/s 1,063 KiB
no cap 7,702 MiB/s 14,555 MiB/s 2,087 KiB

64 KiB cost throughput and CPU. Larger caps used more memory per session and were slower with 32 MiB uploads.

channel_try_write handed ssh_channel_write() a queued chunk's whole
remainder, cast to uint32_t. With max_write_buffer=0, a write of 2 GiB or
more fails there, and at 4 GiB the length wraps. Each call now gets at
most 256 KiB, so a chunk of any size goes out in pieces.

One call encrypts as much as the peer's window allows (OpenSSH: 2 MB),
and what the kernel's socket buffer cannot take right away waits in
libssh's send buffer. The cap keeps that at about 350 KB per busy
connection instead of about 2.4 MB.

The cap comes from the scp branch, where Antonio Prcela added and
measured it.

A sweep of caps ran 10,000 concurrent uploads to an sshd over loopback
on a 128-thread machine. Medians of 2 runs, with CPU relative to no cap:

            10,000 x 32 MiB    10,000 x 512 MiB
  cap       MiB/s  CPU         MiB/s  CPU
  64 KiB    11953  +10.0%      13963  +3.0%
  256 KiB   14141   +7.4%      14999  -0.8%
  1 MiB     12205   +6.8%      15127  -0.1%
  no cap     7702    0         14555   0

Memory at 10,000 sessions, as the growth of the process's RSS outside
the GC heap per session, and the process's peak RSS:

  cap       per session   peak RSS
  64 KiB      75 KiB       1.3 to 1.8 GiB
  256 KiB    295 KiB       3.4 to 3.9 GiB
  1 MiB     1063 KiB      10.7 to 11.3 GiB
  no cap    2087 KiB      20.5 to 21.0 GiB

256 KiB was the fastest cap, or within 1% of it, in every sweep. Below
it, throughput drops and CPU rises: 64 KiB was 7 to 15% slower. Above
it, the memory per connection grows with the cap for at most 1% more
throughput, and short uploads got slower: 10,000 uploads of 32 MiB ran
14% slower at 1 MiB and 46% slower with no cap.
Client and Server defaulted to an 8 MiB max_write_buffer, which fails a
channel on any write that would queue more. SshPipe creates its Clients
without the option, so a NETCONF message over 8 MiB could not be sent at
all.

The default is now 0, no limit. A value above 0 still fails a channel
whose queued writes would exceed it. Since the previous commit, a write
of any size goes to libssh in pieces of at most 256 KiB.
@mzagozen
mzagozen merged commit e1cc337 into main Nov 11, 2026
1 check passed
@mzagozen
mzagozen deleted the large-writes branch November 11, 2026 09:29
@mzagozen mzagozen mentioned this pull request Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant