Skip to content

feat(transfer): add multi-source planning and global controls#271

Draft
cxymds wants to merge 1 commit into
cxymds/issue-1385-command-stackfrom
cxymds/issue-1372-transfer-controls
Draft

feat(transfer): add multi-source planning and global controls#271
cxymds wants to merge 1 commit into
cxymds/issue-1385-command-stackfrom
cxymds/issue-1372-transfer-controls

Conversation

@cxymds

@cxymds cxymds commented Jul 21, 2026

Copy link
Copy Markdown
Member

Related issues

Closes rustfs/backlog#1372

Parent roadmap: rustfs/backlog#1361

Depends on #272. Review and merge the explicit-stack launcher prerequisite first.

Background and user impact

rc cp accepted only one source and recursive transfers used command-local sequencing, which made practical mc-style bulk workflows impossible to configure consistently. Filtering, retries, concurrency, failure aggregation, and summaries also had no shared contract that a future mirror implementation could reuse.

This PR adds deterministic multi-source planning and command-wide transfer controls while preserving the existing single-object path and output contract.

Solution

  • Add reusable storage-independent TransferPlan, TransferSelection, and TransferExecutor primitives in rc-core.
  • Accept multiple copy sources and reject ambiguous or colliding destinations before transfer execution.
  • Add include/exclude globs, strict UTC newer/older boundaries, inclusive rewind-style selection, and configurable empty-selection behavior.
  • Enforce one concurrency budget, aggregate rate pacing, classified transient retry budget, fail-fast cancellation, and continue-on-error behavior across the command.
  • Preserve deterministic per-item outcomes and aggregate planned, skipped, successful, failed, cancelled, and transferred-byte counters.
  • Add safe recursive source traversal and destination validation for planned downloads.
  • Keep failures visible and return the first deterministic non-success exit code even with --continue-on-error.

Scope and deferred work

  • Recursive remote-to-remote and multipart copy remain in rustfs/backlog#1371 scope.
  • Full mirror parity will consume the shared planner in the dependent mirror roadmap work.
  • Batch JSON summaries remain deferred until the versioned output contract in feat(rc-output): introduce output schema v3 #266 lands. Planned bulk JSON fails explicitly; legacy single-object JSON is unchanged.

BREAKING

This PR updates the protected docs/reference/rc/cp.md behavior contract. The CLI change is additive and backward-compatible: existing one-source syntax and legacy single-object JSON output remain valid. No configuration migration or output-schema version bump is required. Planned bulk JSON is explicitly rejected until a versioned batch schema is available.

Validation

  • CARGO_TARGET_DIR=/tmp/rc-target-1372-restack CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=1 cargo fmt --all --check
  • CARGO_TARGET_DIR=/tmp/rc-target-1372-restack CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=1 cargo clippy --workspace -- -D warnings
  • CARGO_TARGET_DIR=/tmp/rc-target-1372-restack CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=1 cargo test --workspace
  • CARGO_TARGET_DIR=/tmp/rc-target-1372-restack CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=1 cargo test -p rustfs-cli --features golden --test golden
  • CARGO_TARGET_DIR=/tmp/rc-target-1372-restack CARGO_INCREMENTAL=0 CARGO_BUILD_JOBS=1 cargo test -p rustfs-cli --test help_contract
  • Protected-file audit: no changes to output schema, exit codes, or configuration schema.
  • Exact-head CI run 29809102164 passed on 2ae14f8330a3a3f70fd914017bfce6dbaa4bf5db, including Windows build and tests.
  • Exact-head Integration Tests run 29809102148 passed on the same commit.

@cxymds
cxymds changed the base branch from main to cxymds/issue-1385-command-stack July 21, 2026 06:36
@cxymds
cxymds force-pushed the cxymds/issue-1372-transfer-controls branch from 000cf2f to 2ae14f8 Compare July 21, 2026 07:03
@cxymds
cxymds changed the base branch from cxymds/issue-1385-command-stack to main July 21, 2026 07:03
@cxymds cxymds closed this Jul 21, 2026
@cxymds cxymds reopened this Jul 21, 2026
@cxymds
cxymds changed the base branch from main to cxymds/issue-1385-command-stack July 21, 2026 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant