Support copying back MLC cached entry for mlc remote-run and mlc slurm-run

[Copilot]

Adds options to copy the MLC cache directory back to the local machine after mlc remote-run and mlc slurm-run complete.

remote_run.py

• --remote_copy_back_mlc_cache — appends the remote MLC cache dir to files_to_copy_back, which the remote,run,cmds,ssh script rsync's back. Handles both normal mode (~/MLC/repos/local/cache) and --remote_isolated (absolute temp-dir path, known at command construction time).
• --remote_copy_back_mlc_cache_path — overrides the local destination; takes precedence over any pre-set path_to_copy_back_files.

slurm_run.py

• --slurm_copy_back_mlc_cache — inserts an rsync -a command into the srun job after the script completes. For --slurm_isolated this is critical: the cache lives inside the temp dir cleaned up by the EXIT trap, so the copy must run inside srun before exit. For non-isolated + no explicit path, the nodes share the filesystem so the option is a no-op.
• --slurm_copy_back_mlc_cache_path — overrides the rsync destination (defaults to ~/MLC/repos/local/cache).

Usage examples:

# Remote run: copy cache back after run
mlcr my,script --remote_host=myserver --remote_copy_back_mlc_cache=yes

# Remote run: copy to custom path
mlcr my,script --remote_host=myserver --remote_copy_back_mlc_cache=yes \
  --remote_copy_back_mlc_cache_path=/data/mlc-cache

# SLURM isolated run: preserve cache before temp dir cleanup
mlcsr my,script --slurm_isolated --slurm_copy_back_mlc_cache=yes

✅ PR Checklist

✅ Testing & CI

• Have tested the changes in my local environment, else have properly conveyed in the PR description
• The change includes a GitHub Action to test the script(if it is possible to be added).
• No existing GitHub Actions are failing because of this change.

📚 Documentation

• README or help docs are updated for new features or changes.
• CLI help messages are meaningful and complete.

📁 File Hygiene & Output Handling

• No unintended files (e.g., logs, cache, temp files, pycache, output folders) are committed.

🛡️ Safety & Security

• No secrets or credentials are committed.
• Paths, shell commands, and environment handling are safe and portable.

🙌 Contribution Hygiene

• PR title and description are concise and clearly state the purpose of the change.
• Related issues (if any) are properly referenced using Fixes # or Closes #.
• All reviewer feedback has been addressed.

• Fixes Support copying back mlc cached entry for mlc remote-run and mlc slurm-run #310

[arjunsuresh]

@copilot can you add GitHub action tests for the new changes?

[Copilot]

@copilot can you add GitHub action tests for the new changes?

Added in commit 7a84d27:

Unit tests — tests/test_slurm_run.py (TestCopyBackMlcCache, 11 tests):

• remote_copy_back_mlc_cache: cache path appended, correct default destination, custom path overrides, isolated mode uses absolute tmp path, no-op when flag absent
• slurm_copy_back_mlc_cache: isolated mode adds rsync, custom path respected, non-isolated + no explicit path is a no-op, non-isolated + explicit path adds rsync

test-mlc-remote-run.yml additions:

• Unit test step (python -m pytest tests/test_slurm_run.py -v -k "copy_back") runs before the live SSH tests
• Test 4 — --remote_copy_back_mlc_cache=yes + --remote_copy_back_mlc_cache_path with real SSH-to-localhost, verifies cache is rsynced back
• Test 5 — same with --remote_isolated=yes (validates the isolated-mode absolute-path branch)

The test-mlc-slurm-core.yml workflow already runs the full test_slurm_run.py test file, so it picks up the new tests automatically.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅