Skip to content

Add safe backfill for the public experiment archive - #858

Draft
YuanYuan98 wants to merge 1 commit into
u/yuanyuan/public-experiment-publishingfrom
u/yuanyuan/public-experiment-backfill
Draft

Add safe backfill for the public experiment archive#858
YuanYuan98 wants to merge 1 commit into
u/yuanyuan/public-experiment-publishingfrom
u/yuanyuan/public-experiment-backfill

Conversation

@YuanYuan98

@YuanYuan98 YuanYuan98 commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Historical backfill follow-up for #711. This PR is stacked on #859 and can remain draft until the team agrees on per-user source paths and permissions.

Goal

Copy existing successful and failed experiment run directories into the public archive after an authorized operator has reviewed them.

What changed

  • add a dry-run-by-default backfill command to the archive helper from Publish team Torch experiments to OSN by default #859
  • require an explicit source root and explicit --owner for each migration
  • store every run under <owner>/<run-name> so identical run names from different users cannot collide
  • scan only direct child run directories and support repeated --run selection
  • refuse filesystem root, all of /scratch, and symlinked roots/runs
  • use copy/check operations only; never sync or delete source or destination data
  • continue across individual failures and optionally write a JSON migration report containing the owner
  • document that migration runs once per authorized user root and requires source read permission

The default destination inherited from #859 is nyu-osn:m2lines-pubs/Samudra/experiments/<owner>/<run-name>.

Verification

  • uv run pytest tests/test_experiment_archive.py (24 passed in the stacked tree)
  • focused pre-commit hooks (all passed)
  • uv run mypy scripts/experiment_archive.py
  • Python 3.6 grammar check for the Torch host-side tool
  • git diff --check

No historical data is moved by this PR itself. Running the migration remains a separate operator action after paths and permissions are agreed.

@YuanYuan98
YuanYuan98 force-pushed the u/yuanyuan/public-experiment-backfill branch from 4462b97 to ad85599 Compare August 25, 2026 14:16
@YuanYuan98
YuanYuan98 changed the base branch from main to u/yuanyuan/public-experiment-publishing August 25, 2026 14:16
@YuanYuan98
YuanYuan98 force-pushed the u/yuanyuan/public-experiment-backfill branch 2 times, most recently from 7ea54c2 to 49b6ac1 Compare August 25, 2026 14:59
@YuanYuan98
YuanYuan98 force-pushed the u/yuanyuan/public-experiment-backfill branch from 49b6ac1 to 553b23a Compare August 25, 2026 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

1 participant