Skip to content

perf(remotefs): send Windows file writes in 4 MiB rigrcp commands - #482

Merged
kke merged 1 commit into
mainfrom
perf/rigrcp-write-chunks
Sep 24, 2026
Merged

kke merged 1 commit into
mainfrom
perf/rigrcp-write-chunks

Conversation

@kke

@kke kke commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Uploads to Windows hosts went through winFile.CopyFrom, which used io.Copy. Upload wraps its source in an io.TeeReader, which hides any io.WriterTo, so io.Copy fell back to its 32 KiB buffer and every 32 KiB became a w command of its own: a reply to wait for before the payload, and since #480, a completion reply after it too. A 100 MiB upload made about 3,200 of those exchanges. CopyFrom now fills 4 MiB at a time, which brings that down to about 25. It does so for every source, including an *os.File, whose own WriteTo would also write 32 KiB at a time.

WriteFile had the opposite problem. bytes.Reader implements io.WriterTo, so its whole payload arrived in one Write and went out as a single w. The helper reads a command's whole payload into memory before writing it, and parses its length as a PowerShell [int]. So WriteFile held the entire file in the helper's memory and failed above 2 GiB. Write now splits a larger payload into commands of at most 4 MiB.

The idle timeout is unaffected. It is still measured per 32 KiB piece of payload handed to the pipe, not per command, so a larger command does not raise the transfer rate a slow link has to sustain.

Refs #475

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The test factory incorrectly captures the parent test context and must use the subtest context.

Review effort: Balanced
Findings: None

What changed in this PR

Improves Windows remote upload performance and memory usage by batching writes into 4 MiB rigrcp commands.

Changes:

  • Splits large writes into bounded commands.
  • Buffers CopyFrom operations in 4 MiB batches.
  • Adds batching and error-handling tests.
File Review
remotefs/​winfs_test.go Adds coverage, but source factories must receive the subtest’s t to avoid invalid parent FailNow calls.
remotefs/​winfile.go Implements bounded writes and buffered copying.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@kke
kke force-pushed the perf/rigrcp-write-chunks branch from 0a4104a to 097d369 Compare September 24, 2026 10:02
@kke
kke requested a balanced review from Copilot September 24, 2026 10:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The critical compilation failure and streaming deadlock risk must be resolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 High severity

Open (2)

Comment thread remotefs/winfile.go
Comment thread remotefs/winfs_test.go
Uploads to Windows hosts went through winFile.CopyFrom, which used
io.Copy. Upload wraps its source in an io.TeeReader, which hides any
io.WriterTo, so io.Copy fell back to its 32 KiB buffer and every 32 KiB
became a w command of its own: a reply to wait for before the payload,
and since #480, a completion reply after it too. A 100 MiB upload made
about 3,200 of those exchanges. CopyFrom now fills 4 MiB at a time,
which brings that down to about 25. It does so for every source,
including an *os.File, whose own WriteTo would also write 32 KiB at a
time.

WriteFile had the opposite problem. bytes.Reader implements
io.WriterTo, so its whole payload arrived in one Write and went out as a
single w. The helper reads a command's whole payload into memory before
writing it, and parses its length as a PowerShell [int]. So WriteFile
held the entire file in the helper's memory and failed above 2 GiB.
Write now splits a larger payload into commands of at most 4 MiB.

The idle timeout is unaffected. It is still measured per 32 KiB piece
of payload handed to the pipe, not per command, so a larger command
does not raise the transfer rate a slow link has to sustain.

Refs #475

Signed-off-by: Kimmo Lehto <klehto@mirantis.com>
@kke
kke force-pushed the perf/rigrcp-write-chunks branch from 097d369 to f14bc32 Compare September 24, 2026 10:12
@kke
kke requested a balanced review from Copilot September 24, 2026 10:12

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

All reviewed changes are covered with no unresolved issues.

Review effort: Balanced
Findings: None

Resolved since last review (2)

@kke
kke marked this pull request as ready for review September 24, 2026 10:30
@kke
kke merged commit be76bd0 into main Sep 24, 2026
18 checks passed
@kke
kke deleted the perf/rigrcp-write-chunks branch September 24, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants