Skip to content

Basic commands: push, pull and archive - #3

Open
krystophny wants to merge 4 commits into
mainfrom
slopqueue/019fdc0994d2-b2862a1c
Open

Basic commands: push, pull and archive#3
krystophny wants to merge 4 commits into
mainfrom
slopqueue/019fdc0994d2-b2862a1c

Conversation

@krystophny

Copy link
Copy Markdown
Member

Closes #1

Implements the missing basic data commands from issue #1 on top of the
existing commit command:

  • dat push <run>: copy a finished run from the temporary data
    directory to the main data directory and record its locations in
    the metadata JSON.
  • dat pull <run>: copy a run from the main data directory back to
    the temporary data directory and record it.
  • dat archive <run>: move a run from the main data directory to the
    archive directory and update the metadata record.

No data is deleted or overwritten: copies and moves refuse to write
over files that already exist, so files can later be updated by first
removing them, as described in the issue. The data directories default
to the ITPcp layout (/temp, /proj/plasma, /proj/plasma/archive)
and are overridable via DAT_TEMP, DAT_MAIN and DAT_ARCHIVE
environment variables for use with shared storage such as Nextcloud.

Tests added in tests/test_dat.py cover push, pull and archive
including recording metadata and the refuse-overwrite behavior.

Implement the remaining basic data commands from issue #1:
- push: copy a run from the temporary to the main data directory
- pull: copy a run from the main to the temporary data directory
- archive: move a run from the main data directory to the archive
Each command updates the metadata record. Copies/moves refuse to
overwrite existing files. Data directories are configurable via
DAT_TEMP, DAT_MAIN and DAT_ARCHIVE environment variables.

Add pytest coverage for push, pull and archive including the
refuse-overwrite behavior.
@krystophny

Copy link
Copy Markdown
Member Author

Review verdict: Request changes

Summary: The PR implements the requested commands and overwrite protection, but accepts unrestricted run paths.

Findings:

  1. [major] src/dat/dat.py:51run is interpolated directly into filesystem paths. Absolute paths or ../ components can escape DATA_TEMP, DATA_MAIN, and DATA_ARCHIVE, allowing commands to copy or move arbitrary directories. Validate run names as safe single directory names (or enforce resolved paths remain beneath the configured roots) before any filesystem operation.

Verdict: Request changes — the path traversal issue can cause unintended data copying or destructive moves.

@krystophny

Copy link
Copy Markdown
Member Author

Review verdict: Request changes

Summary: The command implementation generally matches issue #1, but the added test suite contains a guaranteed failure.

Findings:

  1. [major] tests/test_dat.py:90 — a..b matches validate_run’s allowed-name regex and is not a path traversal, so all three calls accept it while the test requires SystemExit. This makes the test suite fail; remove a..b from the rejected cases or explicitly reject names containing ...

Verdict: Request changes — the added tests do not pass against the implementation.

a..b is a valid plain run name matching the allowed-name regex and is not
a path traversal, so the test suite was guaranteed to fail. Drop it from
the rejected traversal cases.
@krystophny

Copy link
Copy Markdown
Member Author

Review verdict: Request changes

Summary: The run transfer implementations are straightforward and include path validation, but command dispatch still exposes module globals and does not reliably reject unknown commands.

Findings:

  1. [major] src/dat/dat.py:40get_function_by_name still returns arbitrary entries from globals(). For example, dat os resolves the os module and then raises a TypeError traceback instead of reporting an unknown command. Replace this with an explicit whitelist such as {"commit", "push", "pull", "archive"} and add dispatch tests.

Verdict: Request changes — the command-line safety behavior remains incorrect.

@krystophny krystophny left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review verdict: Comment

Summary: The diff addresses the previously identified path-validation and command-dispatch issues. The implementation is internally consistent, but no CI checks are configured or available.

Findings:

No findings.

Verdict: Comment — no blocking defects found, but local tests could not run because elabapy is unavailable and the repository has no CI evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Basic commands

1 participant