Repository navigation
Optimize for sending >8MB in many (10k) sessions - #25
Merged
Merged
Conversation
channel_try_write handed ssh_channel_write() a queued chunk's whole
remainder, cast to uint32_t. With max_write_buffer=0, a write of 2 GiB or
more fails there, and at 4 GiB the length wraps. Each call now gets at
most 256 KiB, so a chunk of any size goes out in pieces.
One call encrypts as much as the peer's window allows (OpenSSH: 2 MB),
and what the kernel's socket buffer cannot take right away waits in
libssh's send buffer. The cap keeps that at about 350 KB per busy
connection instead of about 2.4 MB.
The cap comes from the scp branch, where Antonio Prcela added and
measured it.
A sweep of caps ran 10,000 concurrent uploads to an sshd over loopback
on a 128-thread machine. Medians of 2 runs, with CPU relative to no cap:
10,000 x 32 MiB 10,000 x 512 MiB
cap MiB/s CPU MiB/s CPU
64 KiB 11953 +10.0% 13963 +3.0%
256 KiB 14141 +7.4% 14999 -0.8%
1 MiB 12205 +6.8% 15127 -0.1%
no cap 7702 0 14555 0
Memory at 10,000 sessions, as the growth of the process's RSS outside
the GC heap per session, and the process's peak RSS:
cap per session peak RSS
64 KiB 75 KiB 1.3 to 1.8 GiB
256 KiB 295 KiB 3.4 to 3.9 GiB
1 MiB 1063 KiB 10.7 to 11.3 GiB
no cap 2087 KiB 20.5 to 21.0 GiB
256 KiB was the fastest cap, or within 1% of it, in every sweep. Below
it, throughput drops and CPU rises: 64 KiB was 7 to 15% slower. Above
it, the memory per connection grows with the cap for at most 1% more
throughput, and short uploads got slower: 10,000 uploads of 32 MiB ran
14% slower at 1 MiB and 46% slower with no cap.
Client and Server defaulted to an 8 MiB max_write_buffer, which fails a channel on any write that would queue more. SshPipe creates its Clients without the option, so a NETCONF message over 8 MiB could not be sent at all. The default is now 0, no limit. A value above 0 still fails a channel whose queued writes would exceed it. Since the previous commit, a write of any size goes to libssh in pieces of at most 256 KiB.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
channel_try_write handed ssh_channel_write() a queued chunk's whole remainder, but the length is a uint32_t. With max_write_buffer=0, a write of 2 GiB or more failed there, and at 4 GiB the length wrapped. Each call now gets at most 256 KiB, so a write of any size goes out in pieces.
The cap also bounds libssh's send buffer. One call encrypts as much as the peer's window allows (OpenSSH: 2 MB), and what the kernel's socket buffer does not take right away waits in libssh's buffer. The next write waits until that buffer is empty, so the cap bounds it per connection: about 300 KiB at 256 KiB, against about 2 MiB without a cap. The cap comes from the scp branch, where Antonio Prcela added it.
With writes of any size working, the second commit makes 0 (no limit) the default max_write_buffer for Client and Server. The 8 MiB default failed a channel on any write that would queue more. SshPipe creates its Clients without the option, so a NETCONF message over 8 MiB could not be sent at all. A value above 0 still fails a channel whose queued writes would exceed it.
256 KiB was selected empirically by running 10,000 concurrent uploads to an sshd over loopback on a 128-thread machine, medians of 2 runs:
64 KiB cost throughput and CPU. Larger caps used more memory per session and were slower with 32 MiB uploads.