Skip to content

SCP implementation - #22

Merged
mzagozen merged 10 commits into
mainfrom
scp
Nov 16, 2026
Merged

mzagozen merged 10 commits into
mainfrom
scp

Conversation

@precla

@precla precla commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

ScpPut and ScpGet copy one file over SCP on an existing Client without blocking. They run the server's scp -t / scp -f on an exec channel and implement the protocol in Acton, because libssh's ssh_scp_* blocks.

The branch also adds:

  • example_scp_client
  • example_scp_multi (sends one file to many devices)
  • bench_scp
  • test_scp (tag openssh)

ScpPut writes the whole file at once, so the file must fit in max_write_buffer (0 for no limit). Pass the same bytes to every upload and the file is in RAM only once.

C changes

  • A queued write no longer busy-loops while the peer's window is shut.
  • Each ssh_channel_write() call writes at most 256 KiB, not up to the peer's window (OpenSSH: 2 MB). Whatever the kernel didn't take used to wait in libssh's send buffer. ACTON_SSH_WRITE_CALL_MAX=16777216 restores the old behavior.

benchmarks

bench_scp over a 1 Gbit/s LAN. The link is the limit (about 111 MiB/s), so any extra cost would show in CPU time.

retain: 5 uploads of 1000 MiB with timeout=600. The table shows the live GC heap after each upload. The pending timeout timer used to keep each upload's data alive.

After is with commit 6d2d5c8 and 37f9c00

After upload Before After
1 1000.8 MiB 0.8 MiB
2 2000.8 MiB 0.9 MiB
3 3000.9 MiB 0.9 MiB
4 4000.9 MiB 0.9 MiB
5 5000.9 MiB 0.9 MiB

get: 5 downloads of 1000 MiB. ScpGet used to copy every received chunk an extra time.

Average of 5 Before After
Time 9107 ms 9124 ms
CPU 5202 ms 5029 ms
GC allocated per byte received 2.02 1.02

fanout: 50 sessions upload the same 500 MiB buffer to /dev/null at the same time. "Old" sets ACTON_SSH_WRITE_CALL_MAX=16777216; "new" uses the 256 KiB default.

Old New
RSS outside the GC heap +104.7 MiB (2144 KiB per session) +17.5 MiB (358 KiB per session)
Peak RSS 622.2 MiB 532.0 MiB
Throughput 111.0 MiB/s 111.2 MiB/s
CPU 117.4 s 116.6 s
Fastest to slowest session 221.3 to 225.1 s 224.6 to 224.7 s

One 2000 MiB upload: the cap means 8 times as many write calls.

Old New
Time 18114 ms 18032 ms
CPU 8652 ms 8637 ms

To reproduce:

./out/bin/bench_scp retain HOST USER PASSWORD /dev/null 1000 5
./out/bin/bench_scp get HOST USER PASSWORD /tmp 1000 5
ACTON_SSH_WRITE_CALL_MAX=16777216 ./out/bin/bench_scp fanout HOST USER PASSWORD /dev/null 500 50
./out/bin/bench_scp fanout HOST USER PASSWORD /dev/null 500 50

Testing

  • The CI command (acton test --min-time 2000 --max-time 10000 --tag openssh): 61 of 62 tests pass.
  • The remaining test, InteropSshdShellNoPtyTester, failed in non-English locales, where tty prints a translated message. It now runs LC_ALL=C tty and passes.
  • test_scp covers:
    • a round trip larger than a channel window
    • an empty file
    • a directory path with quotes and spaces
    • errors from the remote scp
    • a file over max_write_buffer
    • two parallel uploads on one Client

Limitations

  • One regular file per transfer, and it must fit in RAM. No recursive copy and no -p.
  • At scale, limit how many uploads run at once. Each holds up to about 256 KiB in libssh, plus its socket buffer.
  • Also raise ulimit -n, and use IP addresses, because hostname lookup blocks.

Comment thread src/bench_scp.act Resolved
Signed-off-by: Antonio Prcela <antonio.prcela@sartura.hr>
precla and others added 9 commits November 16, 2026 12:36
ScpPut and ScpGet copied every received chunk with `_rx += chunk`, even
when _rx was empty, so every downloaded byte was allocated twice.

A pending timeout timer keeps its actor alive until it fires, which for
ScpPut includes its reference to data. RunCommand, ScpPut and ScpGet mark
their timeouts with a TODO to cancel the timer once Acton supports
cancelling timers. The new bench_scp shows what a finished ScpPut keeps:
it prints the live GC heap after each of a series of uploads.

Measured downloading 1000 MiB over 1 Gbit: GC allocation per byte
downloaded 2.02 -> 1.02.

Signed-off-by: Antonio Prcela <antonio.prcela@sartura.hr>
Signed-off-by: Antonio Prcela <antonio.prcela@sartura.hr>
channel_finalize cleared the context's actor pointer but kept its
callbacks, and the context stays on the client's retired_channels list
until the client finalizes. Each callback refers to the actor that uses
the channel, so every finished RunCommand, ScpPut or ScpGet stayed in
memory, with everything it refers to, for as long as its Client was
open. server_channel_finalize did the same with the session's
retired_channels and the actor behind a ServerChannel's callbacks.

Both now clear the callbacks after the last delivery. A queued message
holds its own reference to the callback it calls, and every reader of
these fields checks for NULL.

A finished consumer can still stay in memory: it and its Channel refer
to each other, and Channel's __cleanup__ registers a GC finalizer, so
the GC never collects the pair.
ScpGetScaleTester downloads a file of t.scale() MiB once per t.loop()
body, so acton test scale and acton test perf can measure how the time
and the GC allocation of a get grow with the file size. Ordinary testing
makes one 1 MiB get.

It is a scenario of ScpTester and uses its sshd. dd makes the file as a
hole that reads as zeros, so the remote scp reads no disk blocks, and
the client turns compression off, so the zeros cross the link at full
size. Once the loop runs, the scenario's 25 s timeout no longer applies:
a perf run can take longer, under the test runner's watchdog.

The loop binds next() to a variable before isinstance, because the
compiler calls next() twice in isinstance(next(it), just).

Up to 256 MiB on a laptop, a get took about 11 ms plus 2.3 ms per MiB
and allocated 1.03 bytes per byte received.
ScpFanoutScaleTester connects t.scale() Clients, at most 64 at a time,
because sshd's listen queue holds 128 connections and macOS resets the
ones past it. The test's sshd_config sets MaxStartups to 100, since the
default refuses some unauthenticated connections past 10. Each t.loop()
body then puts the same 8 MB on every Client at once and waits for all
of them, so acton test scale shows how the time, the GC allocation and
the peak RSS grow with the number of Clients. The peak RSS includes
libssh's send buffers, which are outside the GC heap.

Up to 64 Clients on a laptop, a body took about 12 to 15 ms per Client
and allocated under 0.7 MiB in all, so no Client copies the data. The
peak RSS grew by about 0.55 MiB per Client. With 1024 Clients a body
took 33 s and allocated 7.3 MB, and the peak RSS was 430 MB.

The scale scenarios now share loop_next, which starts the next t.loop()
body.

Co-authored-by: Antonio Prcela <antonio.prcela@sartura.hr>
The step runs ScpGetScaleTester under acton test perf at 50 MiB and
fails when the average GC allocation per loop body is above 1.1 bytes
per byte received. Locally it measured 1.02 to 1.03, also with 14 busy
loops beside it. Each download arrives in 32 KiB chunks, and each chunk
costs its copy plus about 1 KiB, so 1.1 fails when a chunk costs about
2 KiB more. Smaller files are noisier.

acton test perf does not fail on a change, so jq checks the allocation
in its JSON report. A skipped or failed test has no measurement, and the
step fails then too.
@mzagozen

Copy link
Copy Markdown
Collaborator

Excellent @precla! I merged the buffer changes separately in #25, and have now refactored the tests in this branch to fit in better with the existing Acton test infra (acton test perf and acton test scale).

@mzagozen
mzagozen merged commit 2f8a407 into main Nov 16, 2026
1 check passed
@mzagozen
mzagozen deleted the scp branch November 16, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants