Publish team Torch experiments to OSN by default - #859
Conversation
4da3160 to
c588cd3
Compare
c588cd3 to
2cc3a98
Compare
YuanYuan98
changed the title
|
@copilot may i have your review? |
I reviewed this PR. The recent Test GPU failures are infrastructure-related ( |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 34eac9557c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Addressed all three automated review findings in
Also added a fresh-run OSN destination collision check so publication cannot silently merge into an existing owner/run prefix. Validation: 28 archive tests passed, full CPU suite passed (437 passed, 2 skipped, 10 xfailed), and pre-commit passed. @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 80e765b1de
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Fixed the credential-bearing repository URL finding in Validation: 35 focused archive/security tests passed; full CPU suite passed (444 passed, 2 skipped, 10 xfailed); pre-commit and detect-secrets passed. @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c859d25d75
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| result = subprocess.run( | ||
| command, | ||
| check=True, | ||
| stdout=subprocess.PIPE, | ||
| ) |
There was a problem hiding this comment.
When two fresh jobs use the same ARCHIVE_OWNER and NAME but different local OUTPUT_BASE paths, both can observe an empty destination here before either reaches the initial publish in initialize_archive. Both jobs then train and periodically copy into the same archive prefix, merging checkpoints and lifecycle metadata from unrelated experiments. Replace this check-then-copy sequence with an atomic reservation or destination lock.
Useful? React with 👍 / 👎.
alxmrs
left a comment
alxmrs
left a comment
There was a problem hiding this comment.
Quick note, but will make a full review later today or tomorrow.
|
|
||
| """Publish Samudra experiment run directories to an rclone archive. | ||
|
|
||
| The CLI is deliberately dry-run by default. ``--apply`` is required before it |
There was a problem hiding this comment.
🐑 I prefer the opposite default, where you add --dry-run to test things out but by default it does what you intent.
|
I have a high level question about this PR: is it possible to directly write to the OSN pod from torch without writing to local disk first? Would that slow down the training process? Maybe we can meet a bit after our Monday weekly sync and discuss this PR / ideas that could help us implement a solution for the sync. |
Part 2 of #711. This PR can be reviewed and merged before the historical backfill work in #858.
Goal
Keep
/scratchas the live training filesystem while archiving team Torch experiment runs to the public OSN research archive by default. A user can explicitly opt out for a private run. Samudra generic configuration, SkyPilot, and third-party workflows are unchanged.What changed
PUBLISH_TO_OSN=1; set it to0to opt out<owner>/<run-name>; owner defaults to the Torch$USERand can be overridden withARCHIVE_OWNER$RUN_DIRevery 15 minutesarchive-status.jsonwith owner, lifecycle, and Slurm metadataThe default destination is
nyu-osn:m2lines-pubs/Samudra/experiments/<owner>/<run-name>(s3://m2lines-pubs/Samudra/experiments/<owner>/<run-name>). The archive uses copy/check operations only. It does not turn OSN intooutput_base, delete scratch data, publish the whole scratch tree, or migrate historical runs.Verification
uv run pytest tests/test_experiment_archive.py(21 passed)uv run mypy scripts/experiment_archive.pybash -n scripts/slurm_apptainer_train.sbatchgit diff --checkA credentialed Torch/OSN smoke run remains an operator step after review.